How to Stop AI From Missing Rare Diseases It's Supposed to Catch
A new framework exposes why AI accuracy ratings mislead when hunting for uncommon medical events—and how hospitals and pharma companies should actually evaluate these tools. The stakes are high: faulty rare-disease detection in drug safety systems could delay crucial warnings about medication dangers.
Originaltitel: Critical Appraisal of Artificial Intelligence for Rare-Event Recognition: Principles and Pharmacovigilance Case Studies
Many high-stakes artificial intelligence (AI) applications target low-prevalence events, where apparent accuracy can conceal limited real-world value. Relevant AI models range from expert-defined rules and traditional machine learning to generative large language models (LLMs) constrained for classification. As the effort and expertise required to develop modern AI decrease, there is a risk that organizations devote too little time to understanding their limitations and sources of error. We outline key dimensions for critical appraisal of AI in rare-event recognition, including problem framing and test set design, prevalence-aware statistical evaluation, robustness assessment, and integration into human workflows. In addition, we propose an approach to structured case-level examination (SCLE), to complement statistical performance evaluation, and a set of considerations to guide procurement or development of AI models for rare-event recognition. We instantiate the framework in pharmacovigilance, drawing on three studies: rule-based retrieval of pregnancy-related reports, duplicate detection combining machine learning with probabilistic record linkage, and automated redaction of person names using an LLM. We highlight pitfalls specific to the rare-event setting including optimism from unrealistic class balance and lack of difficult positive controls in test sets-and show how cost-sensitive targets align model performance with operational value. While grounded in pharmacovigilance practice, the principles generalize to domains where positives are scarce, and error costs may be asymmetric.