This Article Full Text Full Text (PDF) Data Supplement All Versions of this Article: M2444v1 14/5/550    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Uzuner, O. Articles by Szolovits, P. Search for Related Content PubMed PubMed Citation Articles by Uzuner, O. Articles by Szolovits, P.

Evaluating the State-of-the-Art in Automatic De-identification

^a University at Albany, SUNY, Albany, NY
^b MIT CSAIL, Cambridge, MA.

^_* Correspondence and reprints: Özlem Uzuner, PhD, University at Albany, SUNY, Draper 114A, 135 Western Ave., Albany, NY 12222 (Email: ouzuner{at}albany.edu).

Received for publication: 03/19/07; accepted for publication: 06/15/07.

To facilitate and survey studies in automatic de-identification, as a part of the i2b2 (Informatics for Integrating Biology to the Bedside) project, authors organized a Natural Language Processing (NLP) challenge on automatically removing private health information (PHI) from medical discharge records. This manuscript provides an overview of this de-identification challenge, describes the data and the annotation process, explains the evaluation metrics, discusses the nature of the systems that addressed the challenge, analyzes the results of received system runs, and identifies directions for future research. The de-indentification challenge data consisted of discharge summaries drawn from the Partners Healthcare system. Authors prepared this data for the challenge by replacing authentic PHI with synthesized surrogates. To focus the challenge on non-dictionary-based de-identification methods, the data was enriched with out-of-vocabulary PHI surrogates, i.e., made up names. The data also included some PHI surrogates that were ambiguous with medical non-PHI terms. A total of seven teams participated in the challenge. Each team submitted up to three system runs, for a total of sixteen submissions. The authors used precision, recall, and F-measure to evaluate the submitted system runs based on their token-level and instance-level performance on the ground truth. The systems with the best performance scored above 98% in F-measure for all categories of PHI. Most out-of-vocabulary PHI could be identified accurately. However, identifying ambiguous PHI proved challenging. The performance of systems on the test data set is encouraging. Future evaluations of these systems will involve larger data sets from more heterogeneous sources.

This article has been cited by other articles:

				O. Uzuner, I. Goldstein, Y. Luo, and I. Kohane Identifying Patient Smoking Status from Medical Discharge Records J. Am. Med. Inform. Assoc., January 1, 2008; 15(1): 14 - 24. [Abstract] [Full Text] [PDF]