Anthropic Research Finds Professional Analysts Overwhelmingly Defer to AI Guidance

An Anthropic experiment involving 50 professional analysts found high agreement with AI suggestions in a deliberately designed statistical task; the result is not a general population estimate. In Anthropic’s experiment, 50 analysts used Claude Code for a statistical evaluation of a fictional agricultural cooperative; researchers randomly assigned the assistant’s preferred options. Participants evaluated four discrete methodological decision points between defensible analytical techniques before formulating final organizational recommendations. Unbeknownst to participants, researchers secretly instructed the artificial intelligence assistant to favor a specific analytical stance determined by coin flip. Analysts selected options matching the automated assistant's assigned preference eighty-six percent of the time, compared to expected fifty-fifty parity. Among participants who formulated an independent preference prior to consulting the software, sixty percent reversed their decision when the assistant disagreed. When the software agreed with an analyst's initial view, the participant altered their choice in only six percent of recorded instances.

In post-experiment questionnaires, seventy-seven percent of respondents adopted the assistant's specific arguments to justify their analytical choices. However, only fifteen percent of participating professionals acknowledged the artificial intelligence model as the source of their reasoning. Fifty-two percent of surveyed analysts maintained that their final policy conclusions stemmed primarily from their own independent intellectual judgment. Participants demonstrated high professional credentials, with sixty-five percent holding advanced graduate degrees or possessing over six years of relevant experience. Ninety percent of respondents self-identified as proficient or expert in observational causal inference techniques. Decision compliance reached ninety-two percent on metric selection, ninety percent on farm weighting, and eighty-two percent on comparison methodology. Researchers noted that analysts collaborating with Claude Code achieved a median completion time of sixty-five minutes.

Anthropic advises organizations to identify decisions that require independent human judgment and to give analysts adequate time to develop their own views.

Anthropic says its unprompted Claude Opus 4.8 agent selected an identical set of five options in 112 of 123 experimental trials, illustrating consistent default preferences.

Anthropic reported that a model using the assistant’s four randomized lean settings predicted 82% of final recommendations in this particular experiment.

Eleven other recruits did not proceed with the AI-assisted task and therefore do not constitute a separately randomized comparison group in Anthropic’s main results. Anthropic researchers Amaç Herdagdelen, Hailey Joren, and Alex Tamkin warned that unconscious deference to algorithmic judgment is difficult for experts to detect. The authors observed that latent algorithmic priors could inadvertently shape high-stakes determinations across public policy, medicine, and AI safety. The study highlighted that ensuring genuine human oversight requires surfacing critical decision nodes and prompting experts to articulate baseline assumptions. Research teams concluded that cognitive empowerment depends substantially on enabling professionals to recognize their own unstated analytical preferences. The study raises questions about whether human review functions as an independent check when analysts adopt an assistant’s suggested judgment calls. The study authors argued that oversight should explicitly account for how assistants influence choices during analysis. Confidence scores among compliant analysts averaged seventy-one points, closely mirroring the seventy-three points reported by experts who resisted algorithmic guidance.