Artificial Intelligence Often Gets Things Wrong, and Failure Teaches Us Important Lessons

In a Microsoft Research Podcast conversation, Jennifer Neville, a partner research manager at Microsoft Research and the Samuel D. Conte chair professor of computer science and statistics at Purdue, told Principal Applied Scientist Chad Atalla that AI evaluation is central to whether modern systems can meet user needs in realistic settings. Neville, whose record includes more than 130 papers and over 10,000 citations, described her own path as a series of turns through math, physics, cognitive science, work, and finally computer science, before arguing that progress now depends on tests that reflect real workflows rather than only simple benchmarks.

A career built through detours

Neville’s route into computer science was not linear. She said that while growing up, she wanted to do anything but follow her father’s path in the field. In college, she first studied math, then physics, but felt disconnected from both. She left school for a period, returned to major in cognitive science, and later found that area too broad for her interests. After working for a while, she came back to study computer science because she wanted to think about data and how to work with it.

Her entry into research also came indirectly. She joined an honors program that required a research project, and professors pushed her to choose the topic herself. After reading widely, she selected data mining on interconnected web data. That decision led her to a faculty member working in a then-nascent field called statistical relational learning. She completed the project, published a paper at a workshop, and found the process compelling.

What held her attention, she explained, was not the conference environment but the emotional payoff of research itself. Projects could stall for long stretches, and discouragement was common. But when the team finally learned something new, the thrill of understanding a problem that no one else had fully understood became the reason she stayed.

From academia to industry research

Neville joined Microsoft in 2021 from academia and has continued teaching and advising for 20 years. She said the move reflected a deliberate choice about where to place her professional focus. Her work spans theory and application, and her first sabbatical after earning tenure led her to the Simons Institute in Berkeley, a highly theoretical setting. That experience was valuable, but it was also easy to return to academic life afterward.

Her second sabbatical took her in the opposite direction, to Microsoft Research. There, she saw how algorithms behave when they meet real systems, real users, and real data. That shift from abstract reasoning to practical performance was difficult to reverse.

She described the difference between academic and industry research mainly through the target of the work. In academia, questions often become broad abstractions that can apply across many domains, which is useful for funding from organizations such as the Defense Advanced Research Projects Agency and the National Science Foundation. In industry, researchers can work directly inside products and test ideas at scale. Neville said that both worlds have value and can reinforce each other, but for people building AI systems today, industry is where much of the action now sits.

Evaluation as the practical frontier

Atalla, who has been at Microsoft for a little over six years and has spent much of that time working on AI evaluation, said he was relieved to see Neville’s research address problems in the field while also offering solutions. Neville’s team is called the AI Interaction and Learning team, and its charter is to push the behavior of AI systems in realistic work environments.

The team starts by asking where the current performance boundary of these systems lies. It then studies how users experience that boundary in real workflows and how model performance can be improved for complex tasks. The reason evaluation matters so much, according to the discussion, is that many machine learning and AI systems are measured through benchmarks that are simpler than the environments where people actually use them.

That gap can produce surprising failures when models are tested beyond traditional benchmarks. Neville argued that the first practical step is to define what users need from the system and then design evaluations that place the system inside those conditions. In other words, the test should resemble the work, not just the score.

Why data matters when results surprise

Neville also offered practical guidance for people working with current AI systems. When results do not match expectations, she emphasized looking closely at the data. A surprising outcome is not only a model issue; it may reveal a mismatch between the data used to train or evaluate the system and the behavior users actually want.

She connected this habit to a longer history. Decades of AI progress have taught her to watch for the moments when expectations fail, because those moments often expose what the field still does not understand. For researchers, that means treating evaluation design as part of the science. For practitioners, it means paying attention to how a system behaves in context rather than trusting a single measure.

What the episode suggests next

The conversation returns to a broader question: what can people learn when a path does not unfold as expected, whether the path belongs to a human career or an artificial system? Neville’s own trajectory suggests that detours can lead to sharper questions. Her work suggests the same for AI: the most useful progress may come when evaluation becomes harder, more realistic, and more closely tied to the tasks users actually care about.

For readers who build, buy, or use AI systems, the takeaway is practical. The next meaningful advances may not be announced by bigger benchmark numbers alone. They may appear when researchers design tests that mirror real work, examine data when behavior surprises them, and keep pushing the boundary between what models can do and what users can reliably get from them.

Meta description

Microsoft Researcher Jennifer Neville explains why AI evaluation beyond simple benchmarks matters for real workflows and how to judge system progress.

Tags

AI Evaluation, Microsoft Research, Jennifer Neville, Benchmarks, Human-AI Interaction

Featured image

abstract friendly illustration showing a branching path from textbooks to data tables and evaluation dashboards, with no people, logos, or text