Nancy Chinchor
ABSTRACT
The MUC-4 evaluation metrics measure the performance of the message understanding systems. This paper describes the scoring algorithms used to arrive at the metrics as well as the improvements that were made to the MUC-3 methods. MUC-4 evaluation metrics were stricter than those used in MUC-3. Given the differences in scoring between MUC-3 and MUC-4, the MUC-4 systems' scores represent a larger improvement over MUC-3 performance than the numbers themselves suggest.
- Frakes, W. B. and R. Baeza-Yates (eds.) (1992) Information Retrieval: Data Structures & Algorithms.< Englewood Cliffs: Prentice Hall. Google Scholar
Digital Library
- Van Rijsbergen, C. J. (1979) Information Retrieval. London: Butterworths. Google ScholarDigital Library
- Nierstrasz, Oscar (1989) "A Survey of Object-Oriented Concepts" in W. Kim and F. H. Lochovsky (Eds.) Object-Oriented Concepts, Databases, and Applications. New York: Addison-Wesley. Google ScholarDigital Library
Recommendations
MUC-5 evaluation metrics
MUC5 '93: Proceedings of the 5th conference on Message understandingThe metrics used for the Fifth Message Understanding Conference (MUC-5) evaluation are a major update to those used for MUC-4 in 1992. The official MUC-5 metrics express error rates while the official MUC-4 metrics express performance in terms of recall ...
MUC-3 evaluation metrics
MUC3 '91: Proceedings of the 3rd conference on Message understandingThe MUC-3 evaluation metrics are measures of performance for the MUC-3 template fill task. Obtaining summary measures of performance necessitates the loss of information about many details of performance. The utility of summary measures for comparison ...
Morphemes and POS tags for n-gram based evaluation metrics
WMT '11: Proceedings of the Sixth Workshop on Statistical Machine TranslationWe propose the use of morphemes for automatic evaluation of machine translation output, and systematically investigate a set of F score and bleu score based metrics calculated on words, morphemes and pos tags along with all corresponding combinations. ...
Comments