Content revision 4e771eaf0390cc87b6c7c6fd8469f3508f6ceeec7119472e6c1ccf17e2e1e5d2

## BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval

Research
Source ID research:bright
Original https://www.henryw.me/#/research#bright

Finding a useful document can depend on first understanding what a problem requires. We developed BRIGHT to evaluate this reasoning-intensive retrieval, using 1,384 real queries across domains including economics, psychology, mathematics and coding. I contributed biology and psychology data and reproduced programming examples to work through the questions and the material needed to answer them. Models that performed well on conventional retrieval benchmarks struggled on BRIGHT. Explicit reasoning about a query improved retrieval, and supplying relevant documents improved question answering. The benchmark makes it possible to study retrieval as part of solving a problem, including how reasoning helps a system recognise the information it needs.