Tool Reviews

The value of medical AI assistance depends on users’ level of expertise

MIT found AI skin diagnosis helps all users, but explanations benefit non-experts through deference, while clinicians prefer bare predictions.

The value of medical AI assistance depends on users’ level of expertise

Key Takeaways

  • Explainable AI improved skin-diagnosis accuracy for both non-experts and clinicians, but the benefits depended strongly on users’ medical expertise.
  • Non-experts gained mostly by deferring to the AI, especially when using LLM explanations, and could become more confident even when the model was wrong.
  • Clinicians performed best with AI predictions alone and were less influenced by explanations, often using AI as a check rather than a primary source of judgment.
  • The study highlights automation bias: explanation style, timing, and perceived fluency can influence trust as much as medical correctness, so AI tools should support independent thinking rather than replace it.

Meta description: MIT study finds explainable AI skin diagnosis improves non-experts through deference, while clinicians do best with model predictions only, no explanation.

The value of medical AI assistance depends on users’ level of expertise

Tags: explainable AI, dermatology, skin cancer, clinical decision support, automation bias

Featured image: abstract, friendly illustration of a magnifying glass over a simple skin mole icon with soft heat-map dots, no real people, no logos, no text.

A study by researchers at MIT, Columbia University, Stanford University and other institutions, published in Nature Medicine, tested whether explainable AI in skin disease diagnosis helps different users. It found AI assistance generally improved accuracy for non-experts and clinicians, but the effect of explanations depended strongly on medical knowledge. The result matters for patients, primary care providers and developers building tools that can point to image regions or explain a prediction in plain language.

The study compared four ways to show AI reasoning

Explainable AI methods are meant to help users decide when to trust a model by describing or validating how it reached a decision. In skin diagnosis, one example is a heat map that marks image regions most important to the prediction. Another is a large language model, or LLM, that writes a plain-language rationale.

The research team placed these techniques against real user tasks. Non-experts looked at images of skin moles and decided whether a mole was cancerous. Primary care providers faced a harder task: providing a differential diagnosis for dermatological disease, meaning they weighed several possible conditions rather than making a single yes-or-no judgment.

Participants received a medical image plus an AI prediction of skin disease, then were tested under four conditions: prediction with confidence but no explanation; a method showing similar images to reinforce the prediction; a heat-map approach highlighting important regions; and an LLM explaining the model’s reasoning.

This setup reflected two growing paths in skin care. Some FDA-approved AI interfaces already help clinicians identify skin conditions in medical images, often by adding explanation to a disease prediction. At the same time, non-experts can run digital diagnosis on their own through AI-powered search engines that respond to user prompts, frequently using LLMs to make predictions easier to understand.

The value of medical AI assistance depends on users’ level of expertise

Non-experts gained accuracy, but often by deferring to the model

The clearest finding was not simply that AI helped. It helped different people in different ways. For non-experts, every explainable AI approach improved diagnostic accuracy. The improvement came mostly because the tools helped users identify non-cancerous moles.

However, much of that gain came from deference rather than better independent judgment. Non-experts trusted LLM-based explanations even when the model was wrong. They also found explanations more convincing when those explanations were vague or generic. In other words, fluency and plausibility could substitute for medical correctness.

The researchers also tested a fairness-constrained model designed to combat bias against darker skin tones. That model significantly improved accuracy and reduced diagnostic disparities based on skin tone. But Marzyeh Ghassemi, an associate professor in MIT’s Department of Electrical Engineering and Computer Science, a member of the Institute for Medical Engineering and Science, and a principal investigator at the Laboratory for Information and Decision Systems and the Abdul Latif Jameel Clinic for Machine Learning in Health, noted that non-experts were better mainly because they relied more on the models. When the model was wrong, it hurt performance more than correct AI helped performance. The deference effect was largest with LLM explanations, and users became more confident about wrong answers when an LLM supported them.

Ghassemi framed the problem as automation bias: an AI system can anchor a human’s judgment, causing them to follow the machine even when it makes an error. That anchoring is especially important when the users are not experts.

Clinicians did best when the AI stayed quiet

Clinicians showed a different pattern. They were not led astray by incorrect AI assistance, and they performed best when they received only the model’s prediction, with no explanation attached. Among the explainability methods, LLM explanations gave clinicians the smallest boost.

The reason was how each group used the explanation. A clinician often enters a case with a diagnosis already forming in mind and uses the AI as a check. A weak or wrong explanation is more likely to be caught against that clinician’s own training. A non-expert, by contrast, may use the same explanation to create an opinion in the first place. A confident-sounding rationale can then pull them toward the wrong answer.

Orson Xu, the study’s lead author and an assistant professor in the Department of Biomedical Informatics at Columbia University, said the same tool can be an asset for one user and a liability for another. How a recommendation is presented can matter as much as whether the recommendation is correct.

The value of medical AI assistance depends on users’ level of expertise

The next step is designing AI that does not replace thinking

The study also found behavioral clues about when deference occurs. Users who were most deferential to AI assistance were the worst performers on the task when no AI was present. The timing of the explanation mattered too. If users saw an explanation before making their own diagnosis, they tended to rely more on the model.

Performance also depended on the image itself. AI systems outperformed humans when disease presentation was subtle. Humans did better when images contained atypical symptoms or unrelated features. That pattern suggests AI can help fill gaps in visual detection, but it should not automatically override human judgment when cases are messy or unusual.

Roxana Daneshjou, a co-author and assistant professor of biomedical data science and dermatology at Stanford University, said the findings matter because patients increasingly turn to AI for health care. Those with the least medical knowledge are most likely to be led astray when explainable AI models produce an erroneous output.

For developers, the implication is not to add more detailed LLM explanations by default. The researchers suggest that a better design may require users to give a diagnostic hypothesis first, then let the AI highlight additional conditions worth considering. That approach preserves the user’s independent judgment and uses the model to expand possibilities rather than settle them.

The paper was authored by Ghassemi, Xu, and Daneshjou, along with many other authors, including MIT graduate student Haoran Zhang, undergraduate Reina Wang, and Luis Soenksen PhD ’20, a research affiliate at the Jameel Clinic, along with clinicians and researchers. The work was funded in part by the National Science Foundation, Schmidt Sciences, the National Bureau of Economic Research, and Columbia University.

Who should use it / who should skip it

For patients and non-experts, use AI skin tools only as a way to narrow questions, not as a diagnosis. If an explanation feels fluent, vivid, or confident, treat that as a reason to seek a clinician, not to settle the case.

Clinicians may skip verbose LLM explanations for image-based tasks and rely more on predictions, but should still watch for automation bias when cases are subtle. Developers should skip one-size-fits-all explainability and design separate paths for beginners and experts, because the same explanation can support one user and mislead another.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.