Multimodal LLMs show limited real-world accuracy in clinical dermatology

A new study evaluated the real-world performance of multimodal large language models (MLLMs) in clinical dermatology, finding a significant gap between benchmark results and actual clinical utility. While models like GPT-4.1 showed promise on public datasets, their diagnostic accuracy dropped considerably when applied to a real-world cohort of 5,811 cases. Incorporating clinical context improved performance, but outputs remained sensitive to data inaccuracies, suggesting current MLLMs are not yet reliable for clinical deployment. AI

IMPACT Current multimodal LLMs show a significant performance drop in real-world clinical dermatology compared to benchmarks, indicating they are not yet ready for deployment.

RANK_REASON This is a research paper evaluating the performance of existing multimodal LLMs on a specific clinical task. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• arXiv
• GPT-4.1
• InternVL-Chat v1.5
• LLaVA-Med v1.5
• MedGemma-4B-Instruct
• SkinGPT4

• paper
• safety
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →