01 · agents
An Empirical Study of Harness Design for Coding Agents
QuestionHow do individual harness components such as context management, planning, and action space impact the long-horizon performance of coding agents?
The authors evaluate coding harness components by keeping the execution loop fixed while varying three elements: planning, action space, and context management. Across four models and 176 matched settings on SWE-Bench Verified and Terminal-Bench 2.1, they find that context management prevents context-overflow failures and is most valuable under tight budgets. Staging rule-based elision before LLM summarization provides top efficiency. Planning acts as a cost saver for stronger models rather than an accuracy booster. Furthermore, bash-capable models operating with a bash-only interface achieve substantially lower costs without sacrificing task performance.
Context management prevents context-overflow failures and becomes increasingly valuable as the context-window budget tightens.
Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models.
A bash-only interface allows bash-capable models to achieve substantially lower cost, especially on command-line-centric tasks.