BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence

The study introduces BI-Bench to evaluate LLM capabilities on end-to-end business intelligence (BI), finding frontier models perform poorly with under 50% accuracy. To address this, the authors design BI-Agent, a tool-augmented agent that decomposes BI workflows into structured subtasks like search, join, and transform. They also develop a post-training framework leveraging Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on synthesized trajectories from real projects. BI-Agent achieves accuracy gains of up to 40 percentage points with vanilla LLMs, and post-trained versions yield gains of up to 30 points.

BI-Agent achieves substantial accuracy gains of up to 40 percentage points with vanilla LLMs, and post-trained BI-Agent yields gains of up to 30 points.