CONSONANCE.for your information
Monday, 5 October 2026frenvi

Worth reading closely

01 — architecture 10 upvotes

Fractional State Space Transition for Long Sequence Modeling

QUESTION — How can exponential forgetting in state space models be replaced with fractional dynamics to improve long-context performance?

This paper introduces FRAC, a selective State Space Model (SSM) architecture derived from fractional dynamics that replaces exponential decay with power-law long memory to overcome the limitations of ODE-based SSMs. FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes, transforming fractional memory into an efficient recurrent module with parallel training and prefill, while maintaining bounded-state autoregressive decoding. Experiments on a 1.3B-parameter language model demonstrate that FRAC improves long-context performance over state-of-the-art SSM baselines.

FRAC replaces exponential decay with power-law long memory using fractional dynamics approximations.

FRAC provides an efficient recurrent module with parallel training and prefill while retaining bounded-state autoregressive decoding.

Experiments including a 1.3B-parameter language model demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines.

ghaddara · 28 Sept 2026 read the original ↗
↑