Stanford HAI Study of 250k Claude Chats Reveals Humans Use AI as Dynamic Collaborators, Not Shortcuts

Analyzing thousands of human-AI conversations, a new study shows that people are turning to chatbots for increasingly critical tasks – and that productive outcomes depend on how the collaboration unfolds.

What actually happens when humans and AI work together? Challenging the assumption that people deploy AI as a shortcut or quick fix, a new study supported by Stanford’s Institute for Human-Centered AI (HAI) shows how real-world users rely on the tool as a dynamic collaborator to elevate their work output and be more productive.

Analyzing a quarter-million anonymized conversations between humans and Anthropic’s Claude chatbot, a team of Stanford researchers found that most people engage with AI in meaningful, substantial ways.

“Counter to many dire predictions, the vast majority of users of AI are not treating it as a ‘free lunch’ but rather as a tool that presents new ideas and ways of thinking, pushes back on the user’s knowledge and reasoning, and collaborates in ways that humans find useful in improving their own work,” said lead author Diyi Yang, an assistant professor of computer science at Stanford University and a senior fellow at HAI. The paper has been accepted by Nature Human Behavior.

Key Questions

Using Insights, Anthropic’s analytical tool, the researchers set out to answer three key questions: Is the work people bring to AI meaningful? What degree of agency do humans take in the conversations? And where does the human-AI partnership experience “friction” that can be productive when positive or break down when negative? What they found challenges conventional wisdom and, in many ways, upends common narratives that AI leads to intellectual surrender.

“We find that, yes, users sometimes bring trivial tasks to AI, like drafting emails, but much more often it is high-stakes, hard-to-reverse, and consequential work. So it’s really important to ensure that the humans do not just simply offload to AI and take its answers wholesale without scrutiny,” noted co-first author Yijia Shao, a computer science graduate student in Yang’s lab. Well over half (56%) of all collaborations were deemed to be of “consequential” or greater level.

“What’s more, as the stakes rise, users begin to engage more actively with the AI, and conversation length nearly doubles from those addressing more mundane tasks,” added co-first author Dora Zhao, a fellow computer science graduate student in the lab.

Beneficial Friction

The researchers found that people rarely use AI to passively offload their work. Instead, they say that most users stay actively engaged with the chatbot, creating a dynamic interplay that improves the work product. Roughly half of the conversations involve what the authors call “friction” – a sort of creative tension that drives learning by humans. Friction is natural and essential in any collaboration and fundamental to growth, they say.

“What we know now is that friction can be beneficial,” Zhao says. “Speaking up and being able to advocate for yourself in the workplace is often a sign of having a healthy team culture. It seems to be the same with human agency in collaboration with AI.”

Productive friction leads to clarification, refinement, and learning. The researchers identified a distinct user type that digs deeper, prompting AI to elucidate its ideas rather than simply generating content. In two-thirds of the interactions, the researchers found a dynamic arising where humans ask AI to explain – to teach – rather than merely produce content for them.

But the friction can turn negative, and the collaborations begin to break down. When they do, users must learn how to work with AI that pushes back – to recover, as the researchers put it. In almost 80 percent of collaborations experiencing negative friction, the user attempted to recover. Whether the outcome of the negative friction is productive, however, depends on how humans use the AI. AI teaching was most beneficial in high-consequence domains like law, software engineering, home improvement, and health.

Recommendations

What is apparent from the study is that people are using AI for important and consequential conversations about legal, medical, professional, and financial matters – in addition to more personal mental health and relationship advice. The study also revealed that the work brought to AI is notably consequential – not mundane – especially in high-stakes fields like law and medicine. In fact, the findings directly contradict earlier research suggesting that people reserve AI for low-accountability tasks. In other words, the subject matter matters.

Taking their research about how humans and AI are collaborating a step further, the researchers then suggested ways to make the collaboration even more productive. One recommendation the authors offer to further improve AI-human collaboration is to raise human literacy about AI.

“What I’m really passionate about is how we can grow the population that benefits from AI,” Shao said, pointing to future avenues of research. “This means we need to get more laypeople to use AI responsibly and maintain or even enhance their skills. This study shows a way forward where humans understand AI’s strengths and weaknesses and make the most of their growing collaboration.”

One area of note for the authors is a growing concern that AI’s beneficial effects may become concentrated in the higher-stakes knowledge sectors, deepening inequalities for those outside them – or worse, creating a sort of intellectual capitulation that many fear will lead to a “de-skilled” workforce.

“We could see a widening gap between those who recognize AI as a tool for skill development and career advancement, while others accept it as a substitute for their own good judgment and reasoning,” Yang noted.

In terms of next steps, Shao is developing tools to help users know when AI output requires greater scrutiny, when to welcome friction as part of a collaboration and exert greater agency, and how to get back on track when friction turns negative. Zhao, meanwhile, is looking into personalization and how to design AI systems that support longer-term benefits, such as increased cognitive engagement, for users. Both researchers underscored the value of ongoing academia–industry partnerships, such as their collaboration with Anthropic, in producing transparent, illuminating research at scale.

“This work is a call to action for more partnerships between academia and industry to advance similar kinds of analyses that show how humans and AI can work together most effectively,” Zhao said.