DepthBench: Measuring How Residual Connections Enable More Computational Depth

Depth is a natural way to increase computational capacity in Transformers, but deeper layers often yield diminishing returns. While enhanced normalization or residual connections exist, it remains unclear whether they truly translate architectural depth into effective computational depth. The authors introduce DepthBench, a controlled benchmark to study computational depth across architectures by varying the width-depth aspect ratio. Evaluating ten representative architectures, they find that standard normalization variants provide little benefit as models become deeper and narrower, whereas HC and Full AttnRes improve consistently, identifying residual connection design as a key determinant.

Standard Pre-LN and most of its norm- and scaling-based variants provide little benefit and can even degrade performance as models become deeper and narrower.

HC and Full AttnRes improve consistently even at extreme deep shapes.

The gains of HC and Full AttnRes are associated with more effective utilization of additional layers.