This paper reports the experiences we had in evaluating the ACCeSS system using the EAGLES evaluation metrics both at the input/output (black box evaluation) and component levels (glass box evaluation). Our primary motivation for using the EAGLES metrics was to apply a set of metrics that allow objective measurement of system performance. These measures are used both for diagnostic evaluation of the current version and for progress evaluation of successive system versions. The EAGLES metrics provided very valuable guidelines for our evaluation, but not all the metrics were unproblematic. Especially for diagnostics during development, the addition of subtask success rate proved very helpful.
Cite as: Hanrieder, G., Heisterkamp, P., Brey, T. (1998) Fly with the EAGLES: evaluation of the "ACCeSS" spoken language dialogue system. Proc. 5th International Conference on Spoken Language Processing (ICSLP 1998), paper 0274, doi: 10.21437/ICSLP.1998-75
@inproceedings{hanrieder98_icslp,
author={Gerhard Hanrieder and Paul Heisterkamp and Thomas Brey},
title={{Fly with the EAGLES: evaluation of the "ACCeSS" spoken language dialogue system}},
year=1998,
booktitle={Proc. 5th International Conference on Spoken Language Processing (ICSLP 1998)},
pages={paper 0274},
doi={10.21437/ICSLP.1998-75}
}