The test travels into the search results
A June 3, 2026 preprint by Wang and colleagues examines search-time contamination: benchmark-related information appears in the material an agent retrieves while being tested. The concern differs from memorizing training data. Here, the evaluation itself allows a route to material that may reveal the answer. [1]
- Metadata: a test name or identifier
- Context: the test question appears
- Answer: question and answer appear together
A repository link is not an answer key
The authors separate benchmark metadata, question context and explicit answers. Their detection combines URL patterns, text overlap and an AI judge for answer exposure, with human checks on selected data. This distinction matters: finding a benchmark-related page alone does not show that the agent saw the answer. [1]
A result with a limited scope
The work focuses mainly on medical and clinical benchmarks. Its comparisons assume similar difficulty between some question subsets, and the authors have limited visibility into proprietary agents. These limits make broad rankings or causal performance claims inappropriate here. Their suggested safeguards include inspectable search traces and controlled access to test material. [1]
Go a little deeper
Optional reading · about 1 more minute
Evidence that matches the question
Hypothetical example: Suppose an agent is asked to infer an answer from several facts. One search returns a page containing the same question followed by the answer. That result might demonstrate successful retrieval, but it would not settle whether the agent could perform the intended inference without that page.
Ask what the score measures
Our interpretation: A useful evaluation report should explain the permitted information sources and how answer exposure was checked. Retrieval skill can be valuable; a reasoning test must still distinguish it from the capability the score is intended to represent.
Original sources
Attributed synthesis, not original reporting. Examples labeled hypothetical or illustrative are explanatory. Reviewing a source does not independently validate its findings.
- Wang and colleagues: search-time contamination ↗
June 3, 2026 v1 preprint; introduction, sections 3–4 and 5.6 read September 28. Author-affiliated research, no independent replication established. No model ranking or inflation percentage adopted.