Mitigating Task-Order Sensitivity and Forgetting via Hierarchical Second-Order Consolidation

We introduce $\mathbf{\text{Hierarchical}}\mathbf{\text{Taylor}}\mathbf{\text{Series-based}}\mathbf{\text{Continual}}\mathbf{\text{Learning}}\mathbf{\text{(HTCL)}}$, a framework that couples fast local adaptation with conservative, second-order global consolidation to address the high variance introduced by random task ordering. To address task-order effects, HTCL identifies the best intra-group task sequence and integrates the resulting local updates through a Hessian-regularized Taylor expansion, yielding a consolidation step with theoretical guarantees. The approach naturally extends to an $L$-level hierarchy, enabling multiscale knowledge integration in a manner not supported by conventional single-level CL systems. Across a wide range of datasets and replay and regularization baselines, HTCL acts as a model-agnostic consolidation layer that consistently enhances performance, yielding mean accuracy gains of $7\mathrm{\%}$ to $25\mathrm{\%}$ while reducing the standard deviation of final accuracy by up to $68\mathrm{\%}$ across random task permutations.