This study evaluates the diagnostic performance of a multimodal artificial intelligence (AI) system (AIMD.1) using de-identified medical images and semi-synthetic patient simulations. The study combines retrospective analysis of existing publicly available image datasets with prospective data collection from licensed clinicians who complete diagnostic evaluation tasks.
In the One-Shot Vision Differential Evaluation (OSVDE) stage, clinicians review individual de-identified medical images and generate a ranked list of potential diagnoses based solely on visual features. In the Multi-Step Conversational Non-Inferiority Evaluation (MSCNE) stage, clinicians complete diagnostic assessments using semi-synthetic patient simulations derived from de-identified medical images. Clinician performance will be compared with the AI system on the same diagnostic tasks.
Human participants consist solely of licensed clinicians who provide diagnostic responses. Medical images and simulated cases are study materials and are not considered study participants. No identifiable patient data are used, and the AI system is evaluated in an offline research environment and is not used for clinical decision-making or patient care.
Active license in Dermatology, Internal Medicine, Otolaryngology, Gynecology, Orthopedics, Pediatrics, Geriatrics, Emergency Medicine, Ophthalmology, Psychiatry, Endocrinology, Family Medicine, or a closely related specialty
Age 18 years or older
Ability to complete diagnostic evaluation sessions remotely using a computer or tablet with reliable internet access
Loss of active license in an eligible specialty
Inability to complete the evaluation session remotely
The study population consists of licensed physicians across multiple medical specialties who participate in diagnostic evaluation tasks using de-identified medical images and semi-synthetic patient simulations. Participants are recruited through professional networks and medical associations. No patients are enrolled in this study.
Case-only
Prospective
AI Diagnostic System (AIMD.1)
Diagnostic testAIMD.1 (also known as NollaMD agent) is a multimodal artificial intelligence (AI) diagnostic system designed to generate differential diagnoses based on analysis of medical images and structured clinical information. In this study, the system is evaluated using de-identified medical images and semi-synthetic patient simulations under controlled research conditions. The AI system generates ranked diagnostic outputs and associated confidence scores, which are compared with reference diagnoses and clinician performance metrics. The system is evaluated in an offline research environment. AI outputs are not used for clinical decision-making, patient management, or real-world medical care.
Top-1 Diagnostic Accuracy
Proportion of evaluated cases in which the primary diagnosis generated by the AI Diagnostic System matches the reference (ground truth) diagnosis. Accuracy will be calculated across de-identified medical image cases and semi-synthetic patient simulation cases and compared with clinician performance.
Top-5 Diagnostic Accuracy
Proportion of evaluated cases in which the correct reference diagnosis appears within the top five ranked diagnoses generated by the AI Diagnostic System.
Diagnostic Accuracy of Clinician Participants
Proportion of evaluated cases in which the clinician participant's primary diagnosis matches the reference diagnosis, calculated across assigned image and simulation cases.
Non-Inferiority of AI Diagnostic Accuracy Compared to Clinicians
Difference in Top-1 diagnostic accuracy between the AI Diagnostic System and clinician participants. Non-inferiority will be assessed using a predefined margin of 5%.
Calibration of AI Diagnostic Confidence
Calibration performance of AI-generated diagnostic confidence scores assessed using Expected Calibration Error (ECE).
Click on the lead sponsor to view all of their trials.
