This observational study will develop and validate a large language model-assisted workflow for imaging cTNM staging annotation and uncertainty recognition in prostate cancer using Chinese PSMA PET/CT report texts generated during routine clinical care. The study will use de-identified report texts and necessary baseline clinical information only. No additional imaging examination, blood test, treatment, or follow-up visit will be assigned for this study.
The main objective is to evaluate whether a locally or institutionally controlled large language model can identify report-derived imaging cT, cN, and cM categories, extract supporting evidence from the original report, and recognize uncertainty expressions. Model performance will be assessed using an internal independent validation set, external validation reports from two collaborating hospitals, and a prospective validation set of 100 consecutive routine PSMA PET/CT reports. A human-AI comparison will also be performed using physicians from urology and imaging-related specialties with different seniority levels.
Male patients aged 18 years or older.
Patients with clinically diagnosed, pathologically diagnosed, or clinically suspected prostate cancer.
Patients who underwent PSMA PET/CT for initial staging, recurrence assessment, treatment response evaluation, metastatic assessment, or other clinical purposes during routine care.
Complete or basically complete Chinese PSMA PET/CT report text is available, including imaging findings and/or diagnostic impression.
PSMA PET/CT reports unrelated to prostate cancer, or reports clearly irrelevant to the research task.
Reports with severely missing, unreadable, or unavailable main text, imaging findings, or diagnostic impression.
Reports that cannot be adequately de-identified or contain residual direct personal identifiers that cannot be safely removed.
Duplicate records, repeated exports of the same examination, or records for which the unique report version cannot be confirmed.
The study population consists of male patients aged 18 years or older with clinically diagnosed, pathologically diagnosed, or clinically suspected prostate cancer who underwent PSMA PET/CT as part of routine clinical care. The study will include de-identified Chinese PSMA PET/CT report texts from the First Affiliated Hospital of Wenzhou Medical University, the First Affiliated Hospital of Ningbo University, and Liuzhou People's Hospital, as well as a prospective set of 100 consecutive routine PSMA PET/CT reports from the First Affiliated Hospital of Wenzhou Medical University.
Cohort
Other
Large Language Model-Assisted Report Annotation
Other interventionA locally or institutionally controlled large language model workflow will analyze de-identified Chinese PSMA PET/CT report texts and generate structured outputs for report-derived imaging cTNM staging annotation, uncertainty recognition, and supporting evidence extraction. This workflow is used only for research evaluation and methodological analysis. It will not assign any examination, treatment, medication, procedure, or follow-up to participants, and it will not guide clinical diagnosis or treatment decisions.
Accuracy of LLM-Assisted Imaging cTNM Staging Annotation
The primary outcome is the accuracy of the large language model in identifying report-derived imaging cT, cN, and cM categories from de-identified Chinese PSMA PET/CT report texts. The LLM-generated cT\_report, cN\_report, and cM\_report will be compared with the expert consensus reference standard. Accuracy, precision, recall, F1-score, macro-F1, micro-F1, complete cTNM triplet matching rate, and confusion matrices will be calculated in the internal 300-report validation set, external validation sets, and prospective 100-report validation set.
Component-Level Accuracy of LLM-Based Uncertainty Recognition
This outcome measures the component-level accuracy of the large language model in recognizing uncertainty labels for report-derived imaging cTNM staging. The LLM-generated cT\_uncertain, cN\_uncertain, and cM\_uncertain labels will be compared with the expert consensus reference standard. Accuracy will be calculated as the number of correctly classified uncertainty labels divided by the total number of component-level uncertainty labels across all reports. The three uncertainty components will be aggregated into one percentage value. 中文对应
Complete cTNM Triplet Matching Rate for Human Evaluators and the LLM
This outcome measures the proportion of reports for which the complete report-derived imaging cTNM triplet assigned by human evaluators and by the large language model exactly matches the expert consensus reference standard. A report will be counted as correct only when all three components, cT\_report, cN\_report, and cM\_report, are correct. The result will be reported as the percentage of reports with complete cTNM triplet agreement. Results will be summarized separately for pre-training human annotation, post-training human annotation, and LLM batch inference.
Annotation Time per Report for Human Evaluators and the LLM
This outcome measures the mean time required to complete report-derived imaging cTNM annotation per report. For human evaluators, annotation time will be recorded during the annotation rounds and divided by the number of annotated reports. For the LLM workflow, batch inference time will be divided by the number of processed reports. Results will be reported separately for pre-training human annotation, post-training human annotation, and LLM batch inference.
Click on the lead sponsor to view all of their trials.
