SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization

The authors study long-context extension for Gated DeltaNet (GDN) models through the spectral perspective of transition matrix dynamics, identifying the need for a broad slow spectral band and preserved fast-decaying modes. They propose SpectralShift, a spectral reparameterization approach that modifies alpha projection initialization to reshape the decay spectrum and applies learning-rate scaling. Experiments demonstrate that SpectralShift consistently improves long-context capabilities during continual pretraining.

SpectralShift reparameterizes alpha projections initialization to reshape the decay spectrum for Gated DeltaNet.

The method introduces learning-rate scaling for alpha projections to facilitate long-context training.