Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

This work proposes Neural Spectral Capacity (NSC), a closed-form scalar grounded in the singular-value spectrum of each weight matrix. Leveraging the Marchenko-Pastur law, NSC is computable directly from architectural specifications without model instantiation, data, or gradients. Its layer-wise additive structure enables NSC-DP, an exact dynamic-programming solver that returns the globally optimal architecture under resource constraints in seconds on a CPU. Empirically, NSC outperforms #Params and #FLOPs in ranking Transformer and CNN families.

NSC-DP discovers a Transformer-XL architecture on WikiText-103 that beats the human-designed baseline in 2 seconds.

It prunes LLaMA-7B to the best 5.7B model across eight commonsense reasoning tasks without any calibration data.

It operates about 5900x faster than the strongest training-free proxy baseline.