This Article Full Text Full Text (PDF) All Versions of this Article: M1733v1 12/3/296    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Hripcsak, G. Articles by Rothschild, A. S. Search for Related Content PubMed PubMed Citation Articles by Hripcsak, G. Articles by Rothschild, A. S.

Agreement, the F-Measure, and Reliability in Information Retrieval

Affiliation of the authors: Department of Biomedical Informatics, Columbia University, New York, NY.

Correspondence and reprints: George Hripcsak, MD, MS, Department of Medical Informatics, Columbia University, 622 West 168th Street, VC5, New York, NY 10032; e-mail: <hripcsak{at}columbia.edu>.

Received for publication: 11/03/04; accepted for publication: 01/04/05.

Information retrieval studies that involve searching the Internet or marking phrases usually lack a well-defined number of negative cases. This prevents the use of traditional interrater reliability metrics like the statistic to assess the quality of expert-generated gold standards. Such studies often quantify system performance as precision, recall, and F-measure, or as agreement. It can be shown that the average F-measure among pairs of experts is numerically identical to the average positive specific agreement among experts and that approaches these measures as the number of negative cases grows large. Positive specific agreement—or the equivalent F-measure—may be an appropriate way to quantify interrater reliability and therefore to assess the reliability of a gold standard in these studies.

This article has been cited by other articles:

				O. Uzuner, I. Goldstein, Y. Luo, and I. Kohane Identifying Patient Smoking Status from Medical Discharge Records J. Am. Med. Inform. Assoc., January 1, 2008; 15(1): 14 - 24. [Abstract] [Full Text] [PDF]

				O. Uzuner, Y. Luo, and P. Szolovits Evaluating the State-of-the-Art in Automatic De-identification J. Am. Med. Inform. Assoc., September 1, 2007; 14(5): 550 - 563. [Abstract] [Full Text] [PDF]