DARPA communicator: cross-system results for the 2001 evaluation

Marilyn A. Walker, Alexander I. Rudnicky, Rashmi Prasad, John Aberdeen, Elizabeth Owen Bratt, John S. Garofolo, Helen Hastie, Audrey N. Le, Bryan Pellom, Alex Potamianos, Rebecca Passonneau, Salim Roukos, Gregory A. Sanders, Stephanie Seneff, David Stallard

This paper describes the evaluation methodology and results of the 2001 DARPA Communicator evaluation. The experiment spanned 6 months of 2001 and involved eight DARPA Communicator systems in the travel planning domain. It resulted in a corpus of 1242 dialogs which include many more dialogues for complex tasks than the 2000 evaluation. We describe the experimental design, the approach to data collection, and the results. We compare the results by the type of travel plan and by system. The results demonstrate some large differences across sites and show that the complex trips are clearly more difficult.

doi: 10.21437/ICSLP.2002-132

Cite as: Walker, M.A., Rudnicky, A.I., Prasad, R., Aberdeen, J., Bratt, E.O., Garofolo, J.S., Hastie, H., Le, A.N., Pellom, B., Potamianos, A., Passonneau, R., Roukos, S., Sanders, G.A., Seneff, S., Stallard, D. (2002) DARPA communicator: cross-system results for the 2001 evaluation. Proc. 7th International Conference on Spoken Language Processing (ICSLP 2002), 269-272, doi: 10.21437/ICSLP.2002-132

@inproceedings{walker02_icslp,
  author={Marilyn A. Walker and Alexander I. Rudnicky and Rashmi Prasad and John Aberdeen and Elizabeth Owen Bratt and John S. Garofolo and Helen Hastie and Audrey N. Le and Bryan Pellom and Alex Potamianos and Rebecca Passonneau and Salim Roukos and Gregory A. Sanders and Stephanie Seneff and David Stallard},
  title={{DARPA communicator: cross-system results for the 2001 evaluation}},
  year=2002,
  booktitle={Proc. 7th International Conference on Spoken Language Processing (ICSLP 2002)},
  pages={269--272},
  doi={10.21437/ICSLP.2002-132}
}