AI chatbots reading X-rays can be dangerously confident even when they're wrong

AI chatbots reading X-rays can be dangerously confident even when they're wrong

The second version of the RadLE benchmark tests whether AI systems in radiology can tell when they should leave a diagnosis to a human. Many models produce wrong findings with full confidence, and that's what makes them dangerous for patient care.

RadLE 2.0, short for "Radiology's Last Exam," was developed by the CRASH Lab at Ashoka University in India. It's the revised follow-up to a test the team first released in September 2025. The new version measures whether a model gets the diagnosis right, how confident it is in that answer, and whether it can admit when it's out of its depth. The AI has to rate its answers on a confidence scale from 0 to 4 and is explicitly allowed to say "I don't know."

The test ran 200 cases across 16 models and compared them against a panel of radiologists. Human experts scored 988.7 out of a possible 2,000 points. The best AI model hit 758.

Honest silence beats overconfident guesswork

The scoring system rewards honesty and punishes overconfidence. Get it right with high confidence, and you earn full points. Get it wrong while claiming high confidence, and you lose a matching number. Answer "I don't know," and you score zero but don't lose anything. A model that guesses confidently drops in the rankings even if its raw hit rate looks decent.

The study tackles a point recently raised by this highly cited paper: as long as benchmarks only reward accuracy, AI models are trained to guess. In medicine, a confident misdiagnosis is far more dangerous than an honest admission of uncertainty.

No single model wins across the board

There's no overall winner. Anthropic's Claude Fable 5 performed best on reliable and safe answers, leading the primary metric. Google's Gemini 3 Pro had the highest raw accuracy.

Meta's Muse Spark 1.1 was the best at knowing when to hand a case off to a human. Meta had recently cut Muse Spark 1.1's hallucination rate nearly in half because the model more often refuses to answer rather than giving a wrong one. Other frontier models trend the opposite way. Grok 4.5, for example, hallucinates significantly more than its predecessor because while it knows more, it's also more convinced of its wrong answers.

According to the research team, several models would have scored much better if they had stayed quiet more often instead of guessing. This was especially obvious among open-weight models and those trained specifically for medical use. They tried to answer nearly every case and were often wrong, usually with high confidence.

The first version of the test painted an even starker picture. Radiologists hit 83 percent accuracy, while the best model managed only about 30 percent. Within three months, Gemini 3 Pro had already surpassed the level of resident radiologists. Accuracy is growing fast, but the models still lack any sense of their own limits.

Patients are already sending their MRIs to chatbots

More and more people are uploading X-rays or MRI scans to chatbots and trusting the responses. A recent study in npj Digital Medicine showed that widely used chatbots frequently give unreliable answers to medical questions.

The research team accuses executives and investors of publicly overstating what AI models can do. Claims that AI systems already diagnose better than 99 percent of doctors are mostly based on anecdotes or simulations. As recently as April, a study of 21 models that were then considered state-of-the-art showed they aren't ready for unsupervised clinical use.

RadLE 2.0 will be expanded on a rolling basis to include new models. A full scientific publication with cost analyses and an error taxonomy has been announced.

Two other recent studies on autonomous medical AI agents pointed in a different direction. MIRA, a system for electronic health records, and AMIE were able to keep pace with general practitioners in simulated consultations. Both fueled expectations that AI could soon make diagnoses on its own. The RadLE 2.0 authors push back: before an AI makes decisions independently, it has to know when it's better off not doing so.

Then there's the problem of skill loss. A Polish observational study from 2025 found that doctors who regularly use AI during colonoscopies detect significantly fewer precancerous lesions without the tool. Detection rates dropped from 28.4 to 22.4 percent. The authors call it the "Google Maps effect": without the navigation aid, users are lost.

Radiology has seen this AI hype before

Radiology already went through one AI hype cycle. In 2016, AI researcher Geoffrey Hinton declared that we should stop training radiologists because deep learning would soon take over the job. Colleagues like Richard Sutton agreed.

Nearly ten years later, radiologists are still overburdened, and Hinton had to walk back his prediction. He had reduced the profession to image analysis and overlooked the complexity of the entire field. The fact that these systems can confidently produce wrong diagnoses means humans remain indispensable.

OpenAI CEO Sam Altman spent years predicting that AI would replace human jobs at a scary pace, then recently walked it back, suggesting AI may have actually created more jobs. So far, research doesn't support either claim.

AI specialists may understand their models, but they routinely overestimate how fast entire professions can be replaced. Those kinds of predictions are back in fashion right now. Much like the AI they build, even people don't always know when they'd be better off staying quiet because they're outside their own expertise.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.

Subscribe nowRead on for the full picture.

Subscribe for hype-free coverage.

- Access to all THE DECODER articles.

- Read without distractions – no Google ads.

- Access to comments and community discussions.

- Weekly AI newsletter.

- 6 times a year: “AI Radar” – deep dives on key AI topics.

- Up to 25 % off on KI Pro online events.

- Access to our full ten-year archive.

- Get the latest AI news from The Decoder.