help button home button JAMIA Bigger figures
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH
First published January 31, 2005 as JAMIA PrePrint; doi:10.1197/jamia.M1733
Journal of the American Medical Informatics Association 2005;12(3):296-298
© 2005 American Medical Informatics Association

A more recent version of this article appeared on May 1, 2005
This Article
Right arrow Full Text (PDF)
Right arrow All Versions of this Article:
M1733v1
12/3/296    most recent
Right arrow Submit a response
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Hripcsak, G.
Right arrow Articles by Rothschild, A. S.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Hripcsak, G.
Right arrow Articles by Rothschild, A. S.

Submitted on November 3, 2004
Accepted on January 4, 2005

Agreement, the F-measure, and reliability in information retrieval

George Hripcsak MD, MS1* and Adam S. Rothschild MD1

Affiliation of the authors: 1 Deparment of Biomedical Informatics, Columbia University, New York, NY

* To whom correspondence should be addressed.

Information retrieval studies that involve searching the Internet or marking phrases usually lack a well-defined number of negative cases. This prevents the use of traditional interrater reliability metrics like the kappa statistic to assess the quality of expert-generated gold standards. Such studies often quantify system performance as precision, recall, and F-measure or as agreement. It can be shown that the average F-measure among pairs of experts is numerically identical to the average positive specific agreement among experts, and that kappa approaches these measures as the number of negative cases grows large. Positive specific agreement-or the equivalent F-measure-may be an appropriate way to quantify interrater reliability and therefore to assess the reliability of a gold standard in these studies.




This article has been cited by other articles:


Home page
 J. Am. Med. Inform. Assoc.Home page
O. Uzuner, I. Goldstein, Y. Luo, and I. Kohane
Identifying Patient Smoking Status from Medical Discharge Records
J. Am. Med. Inform. Assoc., January 1, 2008; 15(1): 14 - 24.
[Abstract] [Full Text] [PDF]

Home page
 J. Am. Med. Inform. Assoc.Home page
O. Uzuner, Y. Luo, and P. Szolovits
Evaluating the State-of-the-Art in Automatic De-identification
J. Am. Med. Inform. Assoc., September 1, 2007; 14(5): 550 - 563.
[Abstract] [Full Text] [PDF]


HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH
Copyright © 1994 by the American Medical Informatics Association.