This prospective observational diagnostic accuracy study evaluates whether large language models (LLMs) - GPT-4o (OpenAI, gpt-4o-2024-11-20) and Claude (Anthropic, claude-sonnet-4-6) - can accurately calculate HEART scores from unstructured Turkish clinical notes and predict 30-day major adverse cardiac events (MACE) in emergency department patients presenting with non-traumatic chest pain.
The study will enroll 600 consecutive adult patients. For each patient, the same anonymized data (free-text anamnesis, ECG report text, troponin value, and age) will be independently processed by both LLMs via separate API calls with deterministic settings (temperature=0, JSON format). A three-expert consensus HEART score - derived through blinded independent scoring by three emergency medicine physicians with majority-vote adjudication - serves as the reference standard for agreement analysis. Actual 30-day MACE (all-cause death, AMI Type 1/2/4b, unplanned revascularization) determined via national health database and telephone follow-up serves as the outcome for diagnostic accuracy analysis.
A secondary documentation-quality sub-study will quantify how spontaneously Turkish emergency anamnesis notes capture HEART score parameters.
Age >=18 years
Chief complaint of non-traumatic chest pain at the emergency department
Written informed consent obtained from the patient or legally authorized representative
Availability for 30-day follow-up (reachable by telephone and/or actively registered in the e-Nabiz national health database)
Traumatic chest pain etiology
ST-elevation myocardial infarction (STEMI) at presentation requiring immediate reperfusion protocol
Refusal or subsequent withdrawal of informed consent
Inability to complete the mandatory 30-day follow-up period
The study population consists of consecutive adult patients presenting with a chief complaint of non-traumatic chest pain to the emergency department of Marmara University Pendik Training and Research Hospital, a tertiary care academic medical center in Istanbul, Turkey. This target population comprises real-world emergency medicine admissions that require acute coronary syndrome risk stratification and evaluation with the HEART score. It excludes individuals presenting with traumatic pain etiologies or acute ST-elevation myocardial infarction (STEMI) requiring immediate, time-critical reperfusion pathways.
Cohort
Prospective
GPT-4o HEART Score Calculator
Other interventionOpenAI GPT-4o (model: gpt-4o-2024-11-20, temperature=0, max\_tokens=500, response\_format=JSON). Each patient's anonymized anamnesis text, ECG report text, troponin value, and age are submitted via a separate API call with no conversation history. Output: HEART score components (0-2 each), total score (0-10), risk group, and indeterminate status.
Claude HEART Score Calculator
Other interventionAnthropic Claude (model: claude-sonnet-4-6, temperature=0, max\_tokens=500, response\_format=JSON). Identical system prompt and input format as GPT-4o. Processed independently with no cross-contamination between models. Output: same JSON schema as GPT-4o.
Three-Expert Consensus HEART Score
Other interventionThree emergency medicine physicians (\>=3 years experience, HEART-score trained) independently score each anonymized record. Majority vote (2/3) determines component scores; a 4th adjudicator resolves ties. Experts are blinded to LLM scores, each other's scores, and MACE outcomes.
Area Under the ROC Curve (AUC) of GPT-4o and Claude HEART Score for 30-Day MACE Prediction
AUC calculated separately for GPT-4o and Claude using the Hanley-McNeil method. MACE is defined as a composite of all-cause death, acute myocardial infarction (Type 1/2/4b), and unplanned revascularization within 30 days. HEART score range is 0-10; a higher score indicates a higher risk of MACE. Analysis will be performed on complete cases only (0 indeterminate components).
Sensitivity and Specificity of GPT-4o and Claude HEART Score at Prespecified Thresholds
Diagnostic sensitivity and specificity calculated at two threshold types: (a) total score \>=4 (binary high-risk cutoff) and (b) trimodal cutoffs (0-3 low risk, 4-6 intermediate risk, 7-10 high risk). Metrics will be reported with 95% Wilson confidence intervals separately for each LLM.
Component-Level and Total-Score Agreement (Cohen's Kappa) Between LLMs and Expert Consensus
Inter-rater agreement will be computed using quadratic weighted Cohen's Kappa for the ordinal total HEART score (range 0-10) and linear weighted Kappa for the individual components (range 0-2). Calculated separately for GPT-4o vs. expert consensus and Claude vs. expert consensus. Values will be interpreted using the Landis \& Koch scale (\<0.20 poor, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 good, \>0.80 excellent).
Comparative AUC Difference Between GPT-4o and Claude (DeLong Test)
Statistical comparison of paired ROC curves between GPT-4o and Claude using the DeLong et al. (1988) method. The formal hypothesis is non-inferiority with an expected delta AUC \<= 0.05. The correlation coefficient between the paired LLM measurements is estimated as rho \>= 0.70.
Proportion of Indeterminate Results for GPT-4o and Claude
The proportion of cases classified into predefined missing data tiers: Complete (0 indeterminate components), Partial indeterminate (exactly 1 missing component preventing definitive score calculation), and Full indeterminate (\>=2 missing components). Reported separately for each LLM and statistically compared between the two models.
Click on the lead sponsor to view all of their trials.
