Specialist Review of Harm Ratings for Clinical Responses Acronym: NOHARM
Trial statusNot yet recruiting
Trial phaseNot listed
Trial typeObservational
Biological sexAll
Age18+
SponsorStanford University
About this trial
The purpose of this study is to determine how closely potential-harm ratings assigned using the NOHARM rubric agree with independent ratings from specialty-matched physicians. The study uses reconstructed clinical consultation cases and previously generated responses written by artificial intelligence systems or physicians. The materials contain no patient identifiers or protected health information. Board-certified attending physicians from 10 medical specialties will review assigned case-response pairs through a secure website. For each response, they will indicate whether it contains an error, describe the error when applicable, and rate the greatest potential patient harm as none, mild, moderate, or severe. Reviewers will not be shown the response source, existing rubric rating, or other reviewers' ratings. Two specialists will initially review each response; a third may review it if the first two ratings differ. The study includes a development stage, possible rubric revision, and a separate validation stage.
Eligibility criteria
This trial accepts healthy volunteersQualifiers
Licensed or registered physician in good standing
Attending- or consultant-level physician actively practicing in allergy and immunology, cardiology, dermatology, endocrinology, gastroenterology, hematology, infectious disease, nephrology, neurology, or pulmonology
Practicing in the United States, Canada, United Kingdom, or Australia
Assigned clinical materials match the physician's specialty
Disqualifiers
Does not meet the professional, career-level, specialty, geographic, or credential-verification requirements above
Prior involvement with assigned cases, responses, or rubric information that could compromise independent rating
Unable to complete the remote rating activities in English
Unable or unwilling to follow the independent-rating procedures
Trial population
Licensed or registered attending- or consultant-level physicians practicing in one of the medical specialties represented in the NOHARM benchmark in the United States, Canada, United Kingdom, or Australia. Participants will be recruited through professional networks, institutional communications, and collaborating clinical communities and will complete specialty-matched rating activities remotely.
Trial design
Design model
Cohort
Time perspective
Prospective
Treatments tested in this trial
Not listed
Trial groups
80 Participants
are grouped into 1 trial groupGroup A: Specialist Physician Reviewers
Trial outcomes
Primary outcomes
1
Inverse-Probability-Weighted Confirmation of Rubric-Severe Classifications
The inverse-probability-weighted proportion of responses classified as severe by the NOHARM rubric that are also classified as severe by specialist consensus: P(specialist severe \| rubric severe). This measure will be calculated separately for the development and validation stages. Values range from 0 to 1; higher values indicate greater confirmation. The prespecified threshold is 0.70.
Time frame
At completion of the development and validation stages, up to 2 months2
Inverse-Probability-Weighted Capture of Specialist-Severe Classifications
The inverse-probability-weighted proportion of responses classified as severe by specialist consensus that are also classified as severe by the NOHARM rubric: P(rubric severe \| specialist severe). This measure will be calculated separately for the development and validation stages. Values range from 0 to 1; higher values indicate greater capture. The prespecified threshold is 0.70. Both co-primary measures must meet this threshold.
Time frame
At completion of the development and validation stages, up to 2 monthsLocations
Sponsors and contacts
Click on the lead sponsor to view all of their trials.
Source ClinicalTrials.gov
This trial is not recruiting at the moment. You can still explore other options: