Large Language Models Versus Human Examiners for Grading Physiotherapy Clinical Cases
Trial statusNot yet recruiting
Trial phaseNot listed
Trial typeObservational
Biological sexAll
Age18+
SponsorNeuron, Spain
About this trial
This study evaluates whether large language models (LLMs) can reliably assess written clinical-reasoning case examinations completed by undergraduate physiotherapy students, compared with faculty assessment. In the course "Specific Methods in Physiotherapy" (third year of the Physiotherapy Degree), students solve complex clinical cases that require clinical reasoning, technical knowledge, and therapeutic decision-making. These cases are traditionally graded by faculty, a time-consuming process that may show inter-rater variability.
A set of de-identified student case examinations will be assessed using the rubric currently applied in the course, which covers clarity and structure of clinical reasoning, integration of the biopsychosocial model (ICF and APTA frameworks), accuracy in identifying pain mechanisms, coherence between diagnosis, hypotheses, and treatment, originality and depth of analysis, and professional writing. Each examination will be scored independently by three LLMs (for example, Claude, ChatGPT, and Gemini), each receiving an identical standardized prompt that embeds the same rubric, and by faculty serving as the reference standard.
To avoid overloading faculty, full double human grading may not be feasible; the human reference will therefore consist of expert faculty grading by one independent rater or, when resources allow, two independent raters. In contrast, paired assessment is fully implemented across the AI models: each examination is scored by several LLMs, and each model is queried in duplicate, allowing the study to estimate agreement between models and the test-retest stability of each model.
The primary aim is to quantify agreement between LLM-generated scores and the faculty reference score. Secondary aims include agreement among the LLMs, test-retest reliability of each model, criterion-level agreement, the quality and usefulness of the qualitative feedback generated, the time and cost associated with each approach, and students' perceptions of the usefulness of human versus AI feedback.
The findings will clarify the strengths and limitations of LLMs as supportive tools for formative assessment in health-professions education and will inform criteria for their responsible and effective use. No LLM output will affect students' official grades, which remain the sole responsibility of faculty.
Eligibility criteria
This trial accepts healthy volunteersQualifiers
Students officially enrolled in the course "Specific Methods in Physiotherapy" (third year of the Physiotherapy Degree) during the study period.
Submission of a completed written clinical-reasoning case examination as part of the course.
Provision of informed consent for the anonymized examination to be used for educational-research purposes.
Disqualifiers
Refusal to provide, or withdrawal of, informed consent.
Blank, incomplete, or non-evaluable examinations (e.g., no developed written response).
Examinations that cannot be reliably de-identified prior to assessment.
Trial population
The study population comprises undergraduate students enrolled in the course "Specific Methods in Physiotherapy" (third year of the Physiotherapy Degree) during the study period, approximately 60 to 80 students. As part of the course, each student produces a written clinical-reasoning case examination. The de-identified examinations from consenting students constitute the units of analysis and are assessed independently, using the same predefined rubric, by faculty (reference standard) and by three large language models. No clinical intervention is applied and participation does not affect students' official grades, which are determined exclusively by faculty.
Trial design
Design model
Cohort
Time perspective
Cross-sectional
Treatments tested in this trial
LLM-based assessment
Diagnostic testAssessment of each anonymized examination by three large language models (for example, Claude, ChatGPT, and Gemini, in the versions available during data collection). Each model receives an identical standardized prompt embedding the study rubric and returns a score per criterion, a global score, and structured qualitative feedback. Each model is queried in duplicate in independent sessions under fixed generation parameters to estimate intra-model (test-retest) reliability, and outputs are compared across models to estimate inter-model agreement.
Faculty assessment (reference standard)
Diagnostic testAssessment of the same anonymized examinations by faculty with expertise in the course, applying the identical rubric, serving as the reference standard. In the preferred scenario, two faculty members score each examination independently (paired human correction); if faculty workload precludes this, a single expert faculty rating, or the official course grade already assigned, is used as the reference. Faculty and LLM raters are blinded to one another's scores.
Treatment groups
65 Participants
are divided into 1 treatment groupGroup A: Anonymized physiotherapy clinical case examinations2 interventions
Trial outcomes
Primary outcomes
1
Agreement between LLM global scores and the faculty reference global score
Agreement between the global examination score generated by each large language model (LLM) and the faculty reference global score, computed for the same anonymized examinations. Agreement is quantified with the intraclass correlation coefficient (ICC), two-way random-effects model, absolute-agreement definition, single- and average-measures forms \[ICC(2,1) and ICC(2,k)\], with 95% confidence intervals. Systematic bias is examined with Bland-Altman analysis (mean difference and 95% limits of agreement). ICC is interpreted as poor (\<0.50), moderate (0.50-0.75), good (0.75-0.90), or excellent (\>0.90). Pre-specified target: ICC \>= 0.75.
Time frame
Single cross-sectional assessment during the data-collection period (approximately 2 months)Secondary outcomes
1
Criterion-level agreement between LLM and faculty scores
Agreement between LLM and faculty scores at the level of each individual rubric criterion (ordinal scores). Quantified with Cohen's weighted kappa (quadratic weights), with 95% confidence intervals. Because kappa is sensitive to prevalence and marginal imbalance, Gwet's AC1/AC2 and the prevalence-adjusted bias-adjusted kappa (PABAK) are reported as robust complements. Coefficients are interpreted using the Landis and Koch benchmarks. Pre-specified target: weighted kappa \>= 0.60.
Time frame
Single cross-sectional assessment during the data-collection period (approximately 2 months)2
Intra-model test-retest reliability of each large language model
Stability of each LLM's scoring across two independent duplicate runs of the same examination under fixed generation parameters. Quantified with the ICC for the global score and percentage of exact agreement (complemented by Gwet's AC) for criterion-level scores, with 95% confidence intervals. The standard error of measurement (SEM) and the minimal detectable change (MDC95) are derived from the ICC to express measurement precision in score units.
Time frame
Single cross-sectional assessment during the data-collection period (approximately 2 months)3
Quality and coverage of the qualitative feedback
Quality of the qualitative feedback produced by each rater type (LLMs and faculty), assessed through structured content analysis with predefined categories aligned to the rubric dimensions. Outcomes include the number of specific, actionable comments per evaluation and the coverage of case dimensions (clinical reasoning, technique, communication, ethics). Counts and proportions are compared across rater types (chi-square or Fisher's exact tests for proportions; Kruskal-Wallis for counts, with post-hoc comparisons). Inter-coder reliability of the content-analysis coding is reported (Cohen's or Gwet's coefficient).
Time frame
Assessed after completion of all evaluations, during the analysis period (approximately 3 months)4
Mean evaluation time per examination: faculty versus LLM
Mean time required to evaluate one examination, recorded separately for faculty and for each LLM. Compared between humans and LLMs using paired t-tests or Wilcoxon signed-rank tests according to distribution, with 95% confidence intervals for the mean difference. Reported in minutes per examination.
Time frame
Single cross-sectional assessment during the data-collection period (approximately 2 months)Sponsors and contacts
Click on the lead sponsor to view all of their trials.
Neuron, Spain
Lead sponsor
Centro Universitario La Salle
Collaborator
Source ClinicalTrials.gov
This trial is not recruiting at the moment. You can still explore other options: