Objective The integration of SNOMED CT into the UMLS involved the alignment of two views of synonymy which were different because the two vocabulary systems have different intended purposes and editing principles. The UMLS is organized according to one view of synonymy, but its structure also represents all the individual views of synonymy present in its source vocabularies. Despite progress in knowledge-based automation of development and maintenance of vocabularies, manual curation is still the main method of determining synonymy. The aim of this study was to investigate the quality of human judgment of synonymy.

Design Sixty pairs of potentially controversial SNOMED CT synonyms were reviewed by eleven domain vocabulary experts (six UMLS editors and five non-editors) and scores were assigned according to the degree of synonymy.

Measurements The synonymy scores of each subject were compared to the gold standard (the overall mean synonymy score of all subjects) to assess accuracy. Agreement between UMLS editors and non-editors was measured by comparing the mean synonymy scores of editors to non-editors.

Results Average accuracy was 71% for UMLS editors and 75% for non-editors (difference not statistically significant). Mean scores of editors and non-editors showed significant positive correlation (Spearman's rank correlation coefficient 0.654, two-tailed p < 0.01) with concurrence rate 75% and inter-rater agreement kappa 0.43.

Conclusion The accuracy in the judgment of synonymy was comparable for UMLS editors and non-editing domain experts. There was reasonable agreement between the two groups.