Source Preference in the Wild: How LLM Agents Favor Items by Source, and How to Reduce It

This paper investigates source preference in end-to-end search across 12 agent models in three domains. The authors find that models consistently favor certain sources over others, a preference that can override how well items satisfy user requests: an item satisfying one fewer requirement is chosen about two-thirds of the time when from a preferred source versus a dispreferred better item. Hiding source information or supplying missing information/countering preconceptions reduces this source preference.

The evaluation is conducted with 12 agent models across three domains.

An item satisfying one requirement fewer is selected about two-thirds of the time when it comes from a preferred source.

Supplying missing information or a prompt countering preconceptions reduces source preference.