Continual Learning Mechanisms Compose for Long-Horizon Memorization

The paper investigates long-horizon memorization, where a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining prior examples or task IDs. The authors organize anti-forgetting mechanisms along two design dimensions: data, function, and weight anchors combined with low-rank allocation rules. Using factorial experiments across three 100-task datasets, the best method combining all three anchors with merged LoRA raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, representing a 28-fold improvement.

The method combining all three anchors with merged LoRA raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement.