The best AI agent got 18 of 24 right when a company's key facts were never written down

The best of 12 AI agents answered 18 of its 24 attempts correctly, or 75%, on questions whose answers a company never wrote down, a preprint posted on 24 September reports ^[1].

That agent paired the Claude Fable 5.1 model with the Era agent program, which the authors' earlier benchmark paper used. In a second program, LangGraph, the same model answered 15 of 24 ^[1].

GPT-6 Astra answered 13 and 12 in the two programs. The other four models answered at most 6 of 24 in either ^[1].

What it measures

The questions are about a company generated by code, whose invented records sit in simulated business systems such as Salesforce and HubSpot ^[1].

Each answer is implied by the records but stated in none of them. In one example, the sales system gives timing as the reason a customer dropped a purchase, and a recorded call blames an outage ^[1].

There were eight questions, each tried three times by every agent. On the two hardest, all the agents together were right in 1 of 84 attempts ^[1].

What it does not tell you

How the agents would do on a real company's data: the company here is invented. The paper is a preprint and has not been peer-reviewed ^[1].

The top score came with the agent program from the authors' own benchmark, and the paper does not say who the authors work for ^[1].