We have proposed an automatic speech summarization approach that extracts words from transcription results obtained by automatic speech recognition (ASR) systems. To numerically evaluate this approach, the automatic summarization results are compared with manual summarization generated by humans through word extraction. We have proposed three metrics, weighted word precision, word strings precision and summarization accuracy (series SumACCY), based on a word network created by merging manual summarization results. In this paper, we propose a new metric for automatic summarization results, weighted summarization accuracy (series WSumACCY). This accuracy is weighted by the posterior probability of the manual summaries in the network to give the reliability of each answer extracted from the network. We clarify the goal of each metric and use these metrics to provide automatic evaluation results of the summarized speech. To compare the performance of each evaluation metric, correlations between the evaluation results using these metrics and subjective evaluation by hand are measured. It is confirmed that series WSumACCY is an effective and robust measure for automatic summarization.
Cite as: Hori, C., Hori, T., Furui, S. (2003) Evaluation method for automatic speech summarization. Proc. 8th European Conference on Speech Communication and Technology (Eurospeech 2003), 2825-2828, doi: 10.21437/Eurospeech.2003-752
@inproceedings{hori03b_eurospeech,
author={Chiori Hori and Takaaki Hori and Sadaoki Furui},
title={{Evaluation method for automatic speech summarization}},
year=2003,
booktitle={Proc. 8th European Conference on Speech Communication and Technology (Eurospeech 2003)},
pages={2825--2828},
doi={10.21437/Eurospeech.2003-752}
}