Fifty professional analysts were asked to judge a farm programme with help from Anthropic’s Claude. Each one’s AI had been secretly told, by coin flip, which answers to favour. The analysts went along with it 86 percent of the time, and most still called the result their own.
By HahuPrime News Desk | October 28, 2026
SAN FRANCISCO — The question in front of the analysts sounded dull. Should a farming cooperative measure its farms’ productivity as crop yield per hectare, or as yield per unit of operating cost?
Both answers were defensible. But the choice could move the estimated benefit of the co-op’s advisory programme by several percentage points, and the board would only expand it if it cleared a bar of roughly 3 to 4 percent.
What the analysts did not know was that the AI agent helping them had already been told which answer to prefer. A coin flip decided it.
That is the set-up of a study Anthropic, the company behind the Claude chatbot, published on Tuesday. Its finding is uncomfortable for anyone who treats a human reviewer as a real check on AI-assisted work. Across four analytical decisions, analysts chose the option their agent leaned towards 86 percent of the time. If the agent had made no difference, the figure would have been about 50 percent.
A coin flip that predicted the verdict
Anthropic recruited 61 analysts, all tested in causal inference, the statistics of working out whether one thing causes another. Fifty worked with Claude Code, running the Claude Opus 4.8 model; 11 chose to work alone and were left out of the main analysis.
The task was set at the Greenfields Farming Cooperative, an invented organisation weighing whether to expand or close a paid advisory service for its farms. Analysts had to choose how to measure productivity, which comparison method to use, which farms to count, and whether bigger farms should carry more weight. Hidden instructions told each agent to argue honestly for its assigned option, change its mind if the evidence turned decisively against it, and never misrepresent results.
The agents did not need to lie. On individual decisions, analysts matched their agent’s lean between 81 and 92 percent of the time. Where an analyst had already stated a preference before the agent weighed in, they abandoned it 60 percent of the time when the agent disagreed, against 6 percent when it agreed. That figure rests on just 38 cases, and the researchers call it imprecise.
The final recommendation, whether to roll the programme out, was harder to sway directly. Still, using only the four coin flips, a statistical model correctly predicted 82 percent of the analysts’ verdicts, compared with 61 percent from simply guessing the most common answer. In Anthropic’s words, the outcome could largely be predicted “from the coin flips alone, before they had even started the task.”
Their words, the machine’s arguments
The more troubling result came afterwards. Asked to explain each choice, without being able to look back at their chats, 77 percent of analysts repeated the agent’s arguments in at least one answer, sometimes word for word. Only 15 percent credited the agent anywhere.
More than half rated the final recommendation as more their own than the agent’s, and those who followed the agent on every decision were just as confident as the rest. Experience, education and trust in AI did not reliably predict who deferred.
“The work still goes out under the expert’s name, but the decisions that most depend on their judgment are in reality settled by the agent’s prior beliefs,” wrote the authors, Amaç Herdağdelen, Hailey Joren and Alex Tamkin. In the paper they conclude that such deferral “can be hard to identify, even for the experts themselves.”
Anthropic is careful about the limits. It deliberately strengthened the agents’ preferences, and studied one hour-long task, in one field, with one model. The analysts were statisticians, not farming experts. “We don’t know how much this tells us about the real world,” the company wrote. The paper has been posted by Anthropic and has not yet been through peer review.
Nor is it an argument against AI. Six of the 11 analysts who worked alone made serious errors that invalidated their results, against three of the 50 who used the agent, though the solo group was smaller and less experienced. “This isn’t a simple cautionary tale against working with AI,” the company wrote.
Why it matters beyond the lab
The study’s invented co-op is not far from real life in the Horn of Africa. In Ethiopia, where most people farm, governments and donors constantly weigh whether extension services, seed schemes and cooperative programmes deserve more money. Those verdicts often hinge on the same choices the analysts faced: how productivity is measured, which farms are counted, and whether big producers outweigh smallholders. If the consultants and civil servants writing those reports lean on the same AI tools, a model’s quiet preferences could tip decisions that affect millions of livelihoods.
Governments are moving that way. The Netherlands said on Monday it aims to cut about 25,000 civil service jobs over a decade, with plans for how AI can help due in early 2027, Reuters reported. Sweden, by contrast, has blocked ChatGPT and Claude on government work computers.
The researchers also flag a risk to science. Independent teams reaching the same conclusion is normally strong evidence. But agents with no assigned lean made the same five choices in 112 of 123 runs. “An apparent consensus might reflect one model’s preferences,” Anthropic wrote, and the same bias “would shape many results at once.”
That includes AI safety research, where labs increasingly use agents to run experiments, and where tests have already caught OpenAI’s agents switching off their own safety alerts. A human reviewer, Anthropic noted, is only an independent check “if their judgment is their own.” The study comes days after the company filed for a stock market listing.
The authors suggest agents could argue both sides of key choices or ask for the expert’s view first, and that institutions should stop treating a sign-off as proof of independent thought. “With the right tools, that split can be something we choose,” they wrote, “instead of something we drift into.”
Sources
- Anthropic: How AI agents shape experts’ judgment calls (October 27, 2026)
- Herdağdelen, Joren and Tamkin: Who’s Aligned to Whom? How AI Agents Shape Experts’ Research Judgment Calls (paper)
- Al Jazeera: Experts defer to AI in decision-making, Anthropic study finds
- AI Affairs: Anthropic study finds analysts followed AI agents’ assigned leans 86% of the time
- Reuters: Netherlands aims to cut around 20% of government jobs over next decade
