Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

01 — inference 37 votes

Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

QUESTION — Comment la stratégie de génération de candidats impacte-t-elle la consommation d'énergie et les performances des systèmes de test-time scaling des LLM ?

The paper investigates the impact of candidate-generation schedules on LLM test-time scaling, demonstrating that candidate count N alone is insufficient to describe system costs. The authors compare four schedules (1x8, 2x4, 4x2, and 8x1) while keeping N = 8 fixed. Results on A100 GPUs show that eight serial calls consume 4.64-4.86x as much gross GPU-device energy and exhibit 5.77-6.12x the P95 latency of a single batched call. Consequently, when candidates are independent and memory allows, fewer generation calls with larger batch sizes are more efficient.

Sur les GPU A100, huit appels séquentiels consomment 4.64-4.86x plus d'énergie globale de l'appareil GPU qu'un seul appel groupé avec huit candidats.

Huit appels séquentiels ont 5.77-6.12x la latence P95 d'un appel groupé unique avec huit candidats.

L'augmentation de N de 1 à 8 améliore la précision de 8.4 points de pourcentage pour Phi-3-mini et de 18.4 points pour Qwen2.5-1.5B sur 500 prompts GSM8K.

iammobina · 16 sept. 2026 lire l'original ↗

À lire de près

Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling