IIIT-Hyderabad Study Cautions Doctors Against Blindly Trusting Medical AI Models
Researchers at the International Institute of Information Technology, Hyderabad have cautioned clinicians against relying entirely on artificial intelligence for medical diagnoses. A study conducted by the Language Technologies Research Centre at IIIT-Hyderabad evaluated four prominent vision-language models against thousands of publicly available chest X-rays, revealing significant discrepancies between the regions highlighted by AI heatmaps and those identified by human radiologists.
The audit evaluating medical vision-language models was accepted for presentation at the MICCAI 2026 international medical imaging conference in Strasbourg.
Researchers benchmarked four models including MAIRA-2, MedGemma-4B, LLaVA-Med-1.5, and LLaVA-1.5 across thousands of chest X-rays and conducted a human reader study with two radiologists.
The findings, accepted for presentation at MICCAI 2026 in Strasbourg, highlight a critical vulnerability in how automated diagnostic tools operate. While medical imaging AI is frequently deployed as a second pair of eyes to outline suspected abnormalities, the audit demonstrated that model attention overlays do not always correspond to actual pathological sites.
Auditing Vision-Language Models on Chest X-Rays
The research team, led by Prof. Parameswari Krishnamurthy with Dr. Syed Faizan serving as principal investigator, tested four systems: MAIRA-2, MedGemma-4B, LLaVA-Med-1.5, and LLaVA-1.5. Investigators observed that an AI model may highlight a correct area while utilizing flawed reasoning, often arriving at a diagnosis first and then reverse-engineering its heatmap placement.
To test this reverse-working hypothesis, researchers removed diagnostic text information from the testing pipeline. The models' localization performance subsequently dropped, indicating that apparent image-based reasoning was heavily influenced by anatomical expectations learned from diagnostic labels rather than pure visual feature extraction.
Researchers found that removing diagnostic information from the testing pipeline directly reduced AI localization accuracy, indicating the models' heatmaps were driven by text labels rather than pure visual analysis.
Radiologist Calibration and Clinical Utility
To establish a human baseline, the study incorporated a two-radiologist reader evaluation where specialists reviewed anonymized model overlays. Interestingly, human evaluators rated MedGemma higher in practical utility than MAIRA-2, even though automated audits initially ranked MAIRA-2 higher in strict heatmap pixel overlap.
Dr. Faizan explained that strict mathematical overlap does not always equate to clinical helpfulness. While a model might pinpoint the exact focal spot of a disorder, radiologists often require a broader surrounding anatomical view to assess disease spread and context, an attribute where MedGemma performed more effectively in human-in-the-loop evaluations.
Human radiologists rated MedGemma-4B higher than MAIRA-2 because it generates broader contextual heatmaps encompassing surrounding anatomical tissue needed to assess disease extent.
Natural Language Vulnerabilities in Healthcare AI
Beyond visual heatmaps, LTRC researchers are investigating vulnerabilities in medical natural language processing. Clinicians frequently vary their terminology when querying AI systems, substituting technical descriptors for colloquial expressions or altering sentence structure.
In collaboration with a radiologist from Christian Medical College Vellore, the research team developed categories of question rephrasing to test model robustness. Accepted at EMNLP 2026, this companion study examines whether vision-language models genuinely comprehend clinical questions or remain overly dependent on precise syntactic formatting.
Institute leadership emphasizes that the ultimate objective of healthcare NLP research is not to replace clinicians but to automate administrative burdens like documentation, report writing, and communication, leaving core medical decisions firmly in human hands.
A medical AI answer is not a medical decision
A language model can produce a confident explanation while omitting a contraindication, misreading a test or failing to ask for essential patient history. Evaluating such systems requires more than measuring factual question-answer scores: clinicians need to see how a model handles uncertainty, conflicting evidence and unsafe recommendations. The IIIT-Hyderabad findings are a reminder to use these tools as support for qualified professionals rather than independent diagnosticians. In real care, the relevant standard remains a traceable assessment using the patient's records, clinical guidelines and professional judgement.