Objective The College of American Pathologists (CAP) Category 1 quality measures, tumor stage, Gleason score, and surgical margin status, are used by physicians and cancer registrars to categorize patients into groups for clinical trials and treatment planning. This study was conducted to evaluate the effectiveness of an application designed to automatically extract these quality measures from the post-operative pathology reports of patients having undergone prostatectomies for treatment of prostate cancer.

Design An application was developed with the Clinical Outcomes Assessment Toolkit that employs an "information pipeline" of regular expressions and support vector machines to extract CAP Category 1 quality measures. System performance was evaluated against a gold standard of 676 pathology reports from the UCLA Medical Center and Brigham and Women's Hospital. To evaluate the feasibility of clinical implementation, all pathology reports were gathered using administrative codes with no manual preprocessing of the data performed.

Measurements The sensitivity, specificity, and overall accuracy of system performance were measured for all three quality measures. Performance at both hospitals was compared and a detailed failure analysis was conducted to identify errors due to poor data quality versus system shortcomings.

Results Accuracies for Gleason score was 99.7%, tumor stage was 99.1%, and margin status was 97.2% for an overall accuracy of 98.67%. System performance on data from both hospitals was comparable. Poor clinical data quality led to a decrease in overall accuracy of only 0.3% but accounted for 25.9% of the total error.

Conclusion Despite differences in document format and pathologists' reporting styles, strong system performance indicates the potential of employing a combination of regular expressions and support vector machines to automatically extract CAP Category 1 quality measures from post-operative prostate cancer pathology reports.