Automatic Fluency Assessment Method for Spontaneous Speech without Reference Text

Abstract

The automatic fluency assessment of spontaneous speech without reference text is a challenging task that heavily depends on the accuracy of automatic speech recognition (ASR). Considering this scenario, it is necessary to explore an assessment method that combines ASR. This is mainly due to the fact that in addition to acoustic features being essential for assessment, the text features output by ASR may also contain potentially fluency information. However, most existing studies on automatic fluency assessment of spontaneous speech are based solely on audio features, without utilizing textual information, which may lead to a limited understanding of fluency features. To address this, we propose a multimodal automatic speech fluency assessment method that combines ASR output. Specifically, we first explore the relevance of the fluency assessment task to the ASR task and fine-tune the Wav2Vec2.0 model using multi-task learning to jointly optimize the ASR task and fluency assessment task, resulting in both the fluency assessment results and the ASR output. Then, the text features and audio features obtained from the fine-tuned model are fed into the multimodal fluency assessment model, using attention mechanisms to obtain more reliable assessment results. Finally, experiments on the PSCPSF and Speechocean762 dataset suggest that our proposed method performs well in different assessment scenarios.

1. Introduction

Speech fluency is an essential criterion of speakers’ verbal ability and is one of the scoring requirements for many language proficiency tests. For example, the TOEFL iBT speaking test in English and the Mandarin Proficiency Test (Putonghua Shuiping Ceshi, PSC) in Chinese assess speaking skills in areas such as pronunciation, grammar, vocabulary, fluency and coherence. Both the language proficiency tests require the speakers to speak spontaneously within a specified time frame without reference text. Factors affecting speech fluency include speech rate, time to fill speech versus silence, hesitation, and repair phenomena [1]. The assessment of speech fluency is more subjective than the common criteria used in tests, such as pronunciation, vocabulary, and grammar. As spontaneous oral test requires a multi-dimensional examination of speakers, manual scoring is still used. However, manual scoring is susceptible to cost, time, and space, resulting in unreliable manual scoring results. Therefore, automatic fluency assessment models are needed to address the above mentioned issues. Unlike assessment models with reference texts, the two main challenges of spontaneous speech fluency assessment are as follows: (a) Pattern matching or forced alignment methods cannot be used as there is no reference text. (b) Spontaneous speech fluency assessment tasks need to be run on transcriptions of ASR, which can contain errors. Therefore, to solve the task of assessing speech fluency without reference text, an automatic fluency assessment method precisely for spontaneous speech needs to be designed.

Automatic speech fluency assessment methods are generally studied for specific tasks and datasets [2]. Different modeling methods exist for specific tasks. Common methods for speech fluency assessment can be classified into two categories: traditional machine learning and deep learning approaches. Traditional research in automatic speech assessment methods has focused on speech signal processing to extract acoustic features and then using machine learning models to map the features to scores. The deep learning model eliminates the need for tedious feature extraction and obtains fluency features from the raw audio, resulting in improved performance of the speech assessment model.

However, there are some limitations to the existing studies, which tend to assess only audio-based aspects and do not combine textual information. However, we found that some disfluent textual features, such as “um”, “un” and repetition, are also frequently present in spontaneous speech. For assessment scenarios with no textual reference, textual features can be obtained using ASR and combined with acoustic features to assess the oral expression of the speakers. This paper proposes a multimodal automatic speech fluency assessment method combining ASR output. Specifically, multi-task learning is first applied to fine-tune the Wav2Vec2.0 model so that the fluency assessment task is jointly optimized with the ASR task. Then we combined the last hidden layer’s output of the fine-tuned Wav2Vec2.0 model with the ASR output and fed them into a fluency assessment model based on multimodality. After fully extracting the unimodal internal features, feature fusion and enhancement are successively achieved using cross-modal attention and self-attention to obtain the final fluency assessment results.

The main contributions of this paper can be summarized as follows:

• A 9.26-h PSC propositional speaking fluency assessment dataset PSCPSF is constructed to address the scarcity of spontaneous speech data resources in Chinese.
• To validate the relevance of the speech assessment task to the ASR task, the multi-task learning framework is used to fine-tune the Wav2Vec2.0 model on the specific dataset so that the model is jointly optimized by using the speech fluency assessment task as the main task and the ASR task as the auxiliary task, thus improving the accuracy of the model in speech fluency assessment.
• A multimodal automatic fluency assessment method combining ASR output is proposed, which combines the final layer of the hidden output of the fine-tuned Wav2Vec2.0 model with the decoded text output, followed by a cross-modal attention mechanism to fuse acoustic and textual features, resulting in more useful fluency features for achieving adequate speech fluency assessment.
• Experiments on the PSCPSF and Speechocean762 datasets for spontaneous speech and read-aloud scenarios demonstrate that our method performs well in the different assessment scenarios.

The structure of the paper is as follows. Section 2 presents work related to the method, and Section 3 describes the proposed model, including the fine-tuning Wav2Vec2.0 model, and the multimodel model. Section 4 then describes the data set and the experimental setup. Section 5 is devoted to the analysis of the experimental results. Section 6 concludes the paper.

2. Related Work

2.1. Automatic Fluency Assessment

Automatic speech fluency assessment models typically involve ASR to generate time-aligned word sequences for the input speech, as well as fluency feature extraction and scoring models. Fluency features generally include long silence, words per second, phone duration, etc. [2,3]. Scoring models are typically processed for classification or regression tasks. Researchers in [4,5,6,7] used support vector machines to classify speech into different fluency levels. In [8], the authors implemented fluency assessment through a Gaussian model. Authors in [9,10,11,12] used manual features as input and multiple linear or ordered regressions to predict fluency scores. With the widespread use of deep learning, deep neural network (DNN)-based approaches were used to improve the performance of spoken English fluency scores by using DNN-based acoustic models and confidence features [13,14]. Chung et al. [15] proposed a convolutional neural network (CNN)-based approach to learn fluency features directly from a corpus of raw data. In addition, the authors in [16,17] used long and short-term memory (LSTM) and bidirectional long and short-term memory (BLSTM) to better capture the dynamic changes in phone-level fluency features.

For most studies on speech, fluency assessment is conducted in read-aloud scenarios. The scenarios of asking speakers to read a given text are relatively easy to assess. In the read-aloud scenario, speech interruptions, speech rate, articulatory quality features, and goodness of pronunciation (GOP) are frequently used in fluency assessment tasks [7,9,17,18]. In contrast to the assessment of speech fluency in the read-aloud scenario, spontaneous speech without reference text is more complex, as the speakers are asked to express themselves freely according to the topic. Unlike the read-aloud scenarios, spontaneous speech requires ASR to obtain transcriptions. Afterward, the transcriptions are forcibly aligned with the audio to get phone, word, and utterance boundaries, after which predefined linguistic rules are applied to extract prosodic and lexical features, and finally regression or classification models are used to obtain assessment results [2,3,5,6,19].

Currently, automatic fluency assessment tasks achieve satisfactory results in both read-aloud scenarios and spontaneous speech scenarios. However, there is still a need for more generalized speech fluency assessment methods. Traditional assessment methods rely heavily on the validity of hand-crafted fluency features that are only applicable to specific datasets and may not apply to other datasets. Furthermore, we find that text features transcribed by ASR are rarely used for fluency assessment. Few studies have combined fluency with text features for comprehensive assessment. Words containing fluency features are present in both English and Chinese, and these text-related fluency features can also be used for fluency assessment. In this work, we drew inspiration from ASR models, speech emotion classification models, music classification models, and multi-layer frameworks [20,21,22,23,24,25,26,27] to propose a multimodal model that focuses on learning useful features from raw speech data for fluency assessment.

2.2. Multi-Task Learning

Multi-task learning (MTL) refers to combining multiple single tasks with relevance to learn from each other, allowing these tasks to share information during the learning process and using the valuable information implicit in various related tasks to help improve performance and generalization on each task [28]. MTL is widely used in deep neural models in the speech domain. In end-to-end ASR tasks, researchers in [29] used a joint connectionist temporal classification (CTC) [30] attention model within the multi-task learning framework. For the speech emotion recognition task, the authors in [31,32,33] used a multi-task learning approach to improve emotion recognition accuracy through shared representation. This paper explores the correlation between speech fluency assessment and ASR tasks and uses MTL when fine-tuning the Wav2Vec2.0 model. Based on the idea of MTL, the speech fluency assessment task is used as the main task and the ASR task as a secondary task, making the speech fluency assessment task learn further useful information from the ASR task to compensate for the shortcomings of the single-task model that does not learn enough useful information.

2.3. Wav2Vec2.0

Wav2Vec2.0 [34] is a framework for the self-supervised learning of speech representations [35], which has been extensively utilized for multi-task speech emotion recognition [24] and various speech activated tasks, such as speaker verification, keyword spotting, voice activity detection, etc. [36]. This paper uses MTL to fine-tune the Wav2Vec2.0 model for different downstream tasks in the speech fluency assessment task.

3. Proposed Method

In the fluency assessment scenario without reference text, the lack of access to the exact content of the speaker’s expression results in low accuracy of the ASR. We integrate the ASR and fluency assessment models to address this problem. This section proposes a multimodal automatic speech fluency assessment method combining ASR outputs while using the combined features from audio and text modalities to improve the stability of the speech fluency assessment model.

The structure of the proposed model is illustrated in Figure 1, which includes a fine-tuned Wav2Vec2.0 model and a multimodal fusion network. The implementation of the whole model can be done step by step. We first fine-tune the Wav2Vec2.0 model using multi-task learning so that the ASR task is jointly optimized with the fluency assessment task to obtain the fluency assessment results and the ASR output. Following this, we extract the hidden states from the fine-tuned Wav2Vec2.0 model as audio feature input and the text decoded by CTC as text feature input. The audio and text features are then processed through CNN to capture contextual features, which are then fused by cross-modal attention. Afterward, the fused text-audio and audio-text features are enhanced by the self-attentive mechanism, respectively. Finally, after connecting these two output vectors from the self-attentive mechanism, we obtain the predicted fluency levels through the fully connected layer. We will provide a detailed explanation of the two components of our model.

3.1. Fine-Tuning Wav2Vec2.0 Model

In recent years, the speech self-supervised pre-trained model has attracted much attention for its excellent performance. The combination of pre-trained and fine-tuning has been shown to meet the challenges posed by insufficient resources for annotated data, enabling models’ performance benefits. Using speech self-supervised pre-trained models such as Wav2Vec2.0 instead of traditional audio feature extraction modules such as MFCC and Fbank can effectively extract disfluency features and build a more suitable speech fluency assessment model.

This section starts with a pre-trained Wav2Vec2.0 model that uses two different tasks to fine-tune specific downstream tasks. The architecture of the model is depicted in Figure 2 and can be divided into two components: a feature encoder based on CNN and a contextual encoder based on Transformer. To prevent interference and corruption of the CNN layers, we fixed all parameters of these CNN blocks and only fine-tuned the parameters of the Transformer.

We propose to fine-tune the Wav2Vec2.0 model with multi-task learning, input the original audio waveform, and produce two output paths (purple and green in Figure 2). We denote the input waveform as X∈$R^L$, where L is the length. We obtain the output features from the last hidden layer of Wav2Vec2.0:

$$Z=f_{Wav2Vec2.0}\left(X\right)$$

(1)

where X∈$R^L$ represents the original speech signal, L denotes the length, and $f_{Wav2Vec2.0}$ is the pre-trained Wav2Vec2.0 model.

As shown in Figure 2, the ASR and fluency assessment tasks use Z as input. The purple task is fluency assessment (FA): The FA model takes speech as input and produces fluency labels as output. The ASR task, represented by the green module, takes speech as input and outputs text. The purple task is the main task in our task, the green task plays a supporting role in the training, and the specific downstream tasks will be described in detail.

3.1.1. ASR Task

For the ASR task, after obtaining the features, we use a fully connected layer (the FC block in the figure) to map the features Z into Logits, after which we apply a softmax operator to convert the character predictions expressed in Logits to the probability vector y. The formula is as follows, where W and b represent the weight vector and bias:

$$y=softmax\left(WZ+b\right)$$

(2)

This task involves the use of connectionist temporal classification (CTC) loss to encode the provided transcription. CTC is a technique used to map input signals to output targets in situations where they have varying lengths and no alignment information is available. Given that the length of the speech signal is usually much longer than that of the transcription, CTC is employed as a loss function to quantify the degree of difference between the input sequence data and the actual output generated by the neural network. We compute the CTC loss as

$${\mathcal{L}}_{CTC}=CTC\left(y,transcription\right)$$

(3)

3.1.2. FA Task

For the FA task, after obtaining the features, a sequence of length L is transformed into vectors by summarizing the length L through a pooling layer. Afterward, we also use a fully connected layer to map the processed features $\widehat{Z}$ into Logits, after which we apply a softmax operator to convert the fluency predictions represented in Logits to the probability vector f:

$$f=softmax\left(W\widehat{Z}+b\right)$$

(4)

In this task, we compute the cross-entropy loss between the probability vector and the fluency label:

$${\mathcal{L}}_{CrossEntroy}=CrossEntroy\left(f,label\right)$$

(5)

3.1.3. Multi-Task

The model can be fine-tuned on the specific dataset to adjust to the new environment, and we adopt multi-task learning to optimize the joint loss:

$$Min\mathcal{L}={\mathcal{L}}_{CrossEntroy}+\alpha {\mathcal{L}}_{CTC}$$

(6)

We use the hyperparameter $\alpha$ to combine the ${\mathcal{L}}_{CTC}$ and ${\mathcal{L}}_{CrossEntroy}$ into one loss. When $\alpha$ is close to 0, the model will pay more attention to FA.

3.2. Modal Based on Multimodality

This section presents the network architecture in Figure 3. First, CNN is used to encode audio features and text features. Then one modality representation is fused into another using the cross-attention module respectively. Based on this, the feature representation is enhanced by utilizing self-attention modules, and the fluency assessment results are obtained from the fully connected layer. Then, we will describe our proposed model in detail.

3.2.1. Modality Encoding

The data consist of audio and text modalities, which are denoted by $X_a\in R^{T_a\times d_a}$ and $X_t\in R^{T_t\times d_t}$, where $T_m$ and $d_m$ denote sequence length and feature vector size of each modality. For text and audio modalities, convolution layers are used to extract feature representation. Specifically, we pass the input sequence through a 1D time convolution layer. It is worth noting that here the temporal convolution projects features of different modalities into the same dimension:

$$Z_{\{a,t\}}=Conv1\mathrm{D}\left(X_{\{a,t\}}\right)$$

(7)

3.2.2. Multimodel Fusion

This subsection will focus on the fusion of different modalities inspired by Tsai [22]. Audio and text modalities play an important role in multimodal fluency assessment tasks. Attention is widely used in multimodal learning, which exploits the human mind to obtain more valuable information and ignores useless information. Cross-attention captures connections between modalities and enables modalities’ fusion. Self-attention is utilized to capture the internal relevance of modality. Therefore, this module combines the strengths of cross-modal attention and self-attention to propose a multi-level cross-modal feature fusion method.

Figure 4 shows the architecture of the cross-attention module. The following formulas related to the cross-attention module are referenced from papers [22,37]. To pass audio (a) to text (t), denoted as “a → t”, we utilize attention modules that require three inputs: query matrix, key matrix, and value matrix. The modality fusion process can be expressed as follows, where W represents the weight:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Q_t& =Z_t{W}_{Q_t}\hfill \\ \hfill K_a& =Z_a{W}_{K_a}\hfill \\ \hfill V_a& =Z_a{W}_{V_a}\hfill \end{array}\end{array}$$

(8)

The cross-attention $Y_t$, which represents the attention from audio (a) to text (t), can be expressed as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Y_t& =C{M}_{a\to t}\left(Z_t,Z_a\right)\hfill \\ \hfill & =softmax\left(\frac{Q_t{K}_a^{\top }}{\sqrt{d_k}}\right)V_a\hfill \\ \hfill & =softmax\left(\frac{Z_t{W}_{Q_t}{W}_{K_a}^{\top }{Z}_a^{\top }}{\sqrt{d_k}}\right)Z_a{W}_{V_a}\hfill \end{array}\end{array}$$

(9)

Following the cross-attention module, a residual connection and layer normalization are implemented to incorporate the original modality from the other modality:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill z_{a\to t}=LN\left(Y_t+Z_t\right)\end{array}\end{array}$$

(10)

After applying layer normalization (LN), a feed-forward layer is utilized to combine the feature representations:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Z_{a\to t}=LN\left(z_{a\to t}+FFN\left(z_{a\to t}\right)\right)\end{array}\end{array}$$

(11)

where FFN means fully connected feed-forward network. Similarly, we can get to the final embedding representation vector $Z_{t\to a}$ from t to a.

The self-attention is better at capturing the internal relevance of data. The difference between self-attention and cross-modal attention is that its query, key, and value are mapped from the same matrix. In the multimodel model, we put the $Z_{a\to t}$ and the $Z_{t\to a}$ through self-attention to enhance the information. Then, we concatenate the outputs and obtain the fluency assessment result by a layer of the fully connected network.

4. Experiments

4.1. Dataset

To validate the proposed method for spontaneous speech fluency assessment, we use the self-built PSCPSF dataset. Additionally, to demonstrate the applicability of the proposed method in other scenarios, such as read-aloud tasks, we conduct similar experiments using the publicly available Speechocean762 dataset. These experiments highlight the versatility of our proposed method for application in different assessment scenarios.

4.1.1. PSCPSF Dataset

PSCPSF is a self-built PSC propositional speaking fluency assessment dataset consisting of 3368 Chinese utterances and their corresponding scoring files. The PSC propositional speaking requires speakers to choose one of two given topics and speak freely for three minutes without reference text. All the topics in the dataset are selected from the “Topics for PSC” created by the National Putonghua Training and Testing Center. In the subsequent section, we will provide a detailed introduction to the PSCPSF dataset from three perspectives: dataset recording and processing, dataset scoring details, and dataset statistics.

During the data recording, the speakers are asked to simulate a real PSC exam scenario and complete a free speech on a relevant topic within three minutes. Some data are recorded in a silent office environment using a desktop computer and Walkers K815 headphones. Some are recorded in a designated App using a mobile phone as the recording device. The audio files are all saved in mono WAV format at 16,000 Hz. As there is no pre-existing reference text for PSC propositional speaking, we must rely on ASR transcriptions, which may contain errors. Therefore, Praat [38], an open source speech analysis software, is used for the alignment of audio and text files.

The manual scoring part of the dataset is evaluated using the National Putonghua Proficiency Test Propositional Speaking Scoring Criteria in seven dimensions: speech standardization, lexical and grammatical standardization, natural fluency, lack of time, relevance to the topic, reading aloud, and invalid discourse. This paper primarily focuses on the assessment of fluency, which employs three levels of fluency as presented in

Table 1. Manual scoring metrics of fluency on the PSCPSF.

Fluency	Description
level1	Natural and fluent speech
level2	Generally fluent in language but poor in oral expression
level3	Incoherent speech with a stiff tone

.

We invited three experienced experts to rate our self-built dataset. As fluency assessment is highly subjective, the results among experts may differ. Therefore, we calculated the average score of the three experts as the primary basis for scoring, using the formula shown in Equation (12). In addition, we calculated the correlation coefficient between each expert’s score and the expert average score, as shown in

Table 2. Correlation coefficients for scoring the fluency between experts on the PSCPSF.

Pearson’s Correlation Coefficient	Expert1	Expert2	Expert3
Expert average score	76.34%	86.25%	64.14%

. The data in the table indicate that the expert average score is reliable and can be used as the label for training models.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathrm{Expert}\mathrm{average}\mathrm{score}=\frac{\mathrm{Expert}1+\mathrm{Expert}2+\mathrm{Expert}3}{3}\end{array}\end{array}$$

(12)

Information on the specific distribution of the data is shown in

Table 3. Distribution of the data on the PSCPSF.

Dateset	Amount	Total Unique Words	Duration (Hours)	Speakers	Gender Proportion
Train	2168	2111	5.95	63	2:1
Test	1200	1777	3.31	45	2:1

below. The test set consists of 1200 utterances from 70 propositional speaking files and 1777 unique words. The training set consists of 2168 utterances from 120 propositional speaking files and 2111 unique words. Figure 5 shows the distribution of fluency scores on the train and test sets. The figure shows that the amount of fluency is consistent across the data set.

4.1.2. Speechocean762 Dataset

The Speechocean762 dataset is a collection of 5000 English utterances from 250 non-native speakers aged 6 to 43, aimed at assessing pronunciation. This dataset undergoes manual scoring by five experts at the word, phoneme, and sentence levels, covering multiple dimensions. The focus of this paper is on evaluating fluency at the sentence level, and the dataset provides a fluency score for each sentence with a scale of 10 points. However, this wealth of annotation needs to be fully exploited, and little research is dedicated to fluency assessment. Therefore, in the Speechocean762 dataset, the sentence-level fluency scores are extracted as research data.

Table 4. Manual scoring metrics of sentence-level fluency on Speechocean762.

Fluency	Description
0–3	The speaker either cannot read the entire sentence, or there is no sound being produced
4–5	The speech exhibits incoherence, characterized by frequent pauses, repetitions, and stammering
6–7	Overall, the speech is coherent with some occasional pauses, repetitions, and stammering
8–10	The speech is coherent and does not exhibit noticeable pauses, repetition, or stammering

shows a detailed description of the scoring metrics for manual annotation regarding sentence fluency on the Speechocean762.

The sentence-level training set consisted of 2500 sentences and 15,849 words; the test set consisted of 2500 utterances and 15,967 words, and the sampling rate of all speech data is 16,000 Hz.

Table 5. Distribution of the data on Speechocean762.

Dateset	Amount	Word	Duration (Hours)	Speakers	Gender Proportion
Train	2500	15,849	2.88	125	1:1
Test	2500	15,967	2.69	125	1:1

provides specific information on the distribution of the data, while Figure 6 illustrates the distribution of fluency scores in both the training and testing sets. We find that the range of fluency scores is mainly concentrated in the range of 7–9.

5. Results and Analysis

5.1. Experimental Settings

Our experiments use PyTorch [39], a flexible and efficient deep learning framework, on NVIDIA TESLA V100 GPU. For fine-tuning tasks, Pearson correlation (Corr), character error rate (Cer), and word error rate (Wer) are used in experiments to assess model performance. We conduct fine-tuned experiments for PSCPSF and Speechocean762 datasets using the pre-trained model: wav2vec-2.0-large and wav2vec-2.0-base, which are fine-tuned jonatasgrosman/wav2vec2-large-xlsr-53-chinese-zh-cn and fine-tuned facebook/wav2vec2-base-960h. The implementations are based on the Huggingface transformers repository [40]. To extract features from the text modality, we utilize BERT [35], generating 1024-dimensional features on the PSCPSF dataset and 768-dimensional features on the Speechocean762 dataset.

Table 6. The hyperparameters used for fine-tuning.

Hyperparameters	Setting
training epochs	100
optimizer	AdamW
$\alpha$	0, 0.001, 0.01, 0.1, 1
learning rate	10−5 if $\alpha$ = 0, 5 × 10−5 otherwise
batch size	8

and

Table 7. The hyperparameters used for multimodal model.

Hyperparameters	Setting
training epochs	20
optimizer	Adam
learning rate	10−3
batch size	16

list the settings of hyperparameters that are used in fine-tuning and multimodal experiments. It is worth noting that the learning rates in the Table 6 are different when $\alpha$ = 0. Since only fluency assessment tasks are performed at this time, it is necessary to reduce the learning rate to prevent significant oscillation after several epochs.

5.2. The Results of Fine-Tuned Wav2Vec2.0

To verify whether multi-task learning contributes to the FA task, we control the strength of the CTC loss by adjusting the hyper-parameter $\alpha$ to obtain different Corr. We adjust $\alpha$ from 0 to 1. When $\alpha$ = 0, the model is trained exclusively on fluency labels. As the value of $\alpha$ increases, the importance of CTC loss also increases. To show the effect of CTC loss on the ASR component, we present the Cer and Wer obtained during the experiment for the PSCPSF and the Speechocean762 datasets.

Table 8. The impact of the CTC loss on PSCPSF dataset.

The Value of Hyperparameter	Corr	Cer
$\alpha$ = 0	0.765	3.68
$\alpha$ = 0.001	0.783	0.176
$\alpha$ = 0.01	0.805	0.160
$\alpha$ = 0.1	0.818	0.151
$\alpha$ = 1	0.824	0.150

and

Table 9. The impact of the CTC loss on Speechocean762 dataset.

The Value of Hyperparameter	Corr	Wer
$\alpha$ = 0	0.750	0.999
$\alpha$ = 0.001	0.755	0.274
$\alpha$ = 0.01	0.763	0.273
$\alpha$ = 0.1	0.763	0.301
$\alpha$ = 1	0.765	0.273

report Corr and Cer, and Wer for the five $\alpha$ choices on the two datasets. As seen in Table 8 and Table 9, when $\alpha$ is set to 0, training only the FA task results in lower accuracy for ASR and a lower Corr compared to other multi-task training results. As we increase $\alpha$, we observe promising results for ASR, as well as an increase in Corr for the FA task. This validates the effectiveness of the multi-task learning mechanism, where the auxiliary task ASR can effectively improve the performance of speech fluency assessment. By considering the performance of Corr along with Cer and Wer on both datasets, we select $\alpha$ = 1 for subsequent experiments.

5.3. The Results of Multimodal Model

This section analyzes the results of the multimodal model experiments from three perspectives. First, we compare the audio-based and text-based models to ascertain the significance of audio and text modality in fluency assessment. Second, we compare the effectiveness of different multimodal models in the evaluation task to verify the effectiveness of the model proposed in this paper. Furthermore, we compare the effects of text generated by ASR output and manually proofread text in multimodal experiments. Lastly, through ablation experiments, we verify the impact of different components of the model on the experimental results.

Our proposed model demonstrates superior performance on the PSCPSF and Speechocean762 datasets, as evidenced by the results presented in both

Table 10. The performance on the PSCPSF dataset.

Model	Corr
Audio-based	0.878
Text-based	0.784
Text-based (Manual transcripts)	0.681
LF-DNN	0.868
LF-DNN (Manual transcripts)	0.863
LMF	0.869
LMF(Manual transcripts)	0.872
MFN	0.872
MFN (Manual transcripts)	0.874
Ours	0.879
Ours (Manual transcripts)	0.879

and

Table 11. The performance on Speechocean762.

Model	Corr
Audio-based	0.781
Text-based	0.261
Text-based (Manual transcripts)	0.585
LF-DNN	0.776
LF-DNN (Manual transcripts)	0.773
LMF	0.770
LMF (Manual transcripts)	0.774
MFN	0.770
MFN (Manual transcripts)	0.771
Ours	0.789
Ours (Manual transcripts)	0.790

. Firstly, we conduct experiments on audio-based and text-based models. The experimental results show that the audio-based model outperforms the text-based model in speech fluency assessment, indicating that speech features are more important for fluency assessment. On the other hand, the text-based model does not perform as well as the audio-based model in the fluency assessment task. In the PSCPSF dataset, the text features after ASR assessed in the text-based model resulted in the Corr that is 0.103 higher than that of the manual translation. The text features in the Speechocean762 dataset after ASR only showed the Corr of 0.261 by the text-based model, while the manually translated text features showed the Corr of 0.585 by the text-based model. This indicates that text features influence fluency assessment, and the text results after different ASR have different effects. The text features in Chinese may contain more disfluent features, resulting in the text-based model outperforming the PSCPSF on the Chinese dataset than the English dataset.

For the fusion of text features with audio features, other classical multimodal analysis models are chosen as comparison models in this paper. The later fusion DNN (LF-DNN) combines unimodal features by concatenating them prior to classification. The lowrank multimodal fusion (LMF) [41] employs modality-specific low-rank factors for effective multimodal fusion. The memory fusion network (MFN) proposes a multi-view continuous learning approach that uses delta-memory attention and multi-view gated memory to capture the interactions between time and patterns [42]. To investigate the performance of text features in different multimodal models, this study input fine-tuned Wav2Vec2.0 decoded text features and manually translated text features into various multimodal modalities. Figure 7 and Figure 8 present a more intuitive display of the differences in fluency assessment results for different text features in various multimodal models. These results show that our proposed model outperforms most multimodal models mentioned in this paper. The text features obtained from human translation are better in the multimodal model than after ASR transcription. Experiments show that more accurate ASR is better in multimodal speech fluency assessment tasks. It is worth noting that since the Speechocean762 dataset has been widely used in pronunciation assessment models, we compare our fluency model with the four pronunciation assessment models on fluency assessment, including LSTM, GOPT [43], an SSL-based method proposed by Kim [44], and a multi-task learning method proposed by Wong [45]. The results are shown in Figure 9. Compared to the pronunciation detection models mentioned in this paper, which have excellent performance in the Speechocean762 dataset, our model performs well in terms of the fluency assessment.

To investigate the impact of distinct components on model performance, we conduct separate ablation experiments on both the PSCPSF and Speechocean762 datasets.

Table 12. Ablation study on PSCPSF.

Model	Corr
Wav2Vec2.0 (Without fine-tuning)	0.800
self-attention (w/o)	0.874
cross-attention (w/o)	0.879
CNN (w/o)	0.876
Ours	0.879

and

Table 13. Ablation study on Speechocean762.

Model	Corr
Wav2Vec2.0 (Without fine-tuning)	0.738
self-attention (w/o)	0.784
cross-attention (w/o)	0.788
CNN (w/o)	0.782
Ours	0.789

present the findings of these experiments. First, we assess the impact of fine-tuning the Wav2Vec2.0 model on the experiments. We generate audio, and text features using the Wav2Vec2.0 model without fine-tuning and obtain the fluency results. The results show that the Corr of the features without fine-tuning decreased by 0.079 and 0.051 for the two datasets, respectively, indicating the effectiveness of the fine-tuning of the Wav2Vec2.0 model. Secondly, self-attention and cross-attention are removed separately. Self-attention had a more significant impact on the results than cross-attention, indicating that applying self-attention to encode the modal representation is necessary. The impact of the cross-attention module on fluency assessment results has not met expectations, possibly due to the influence of textual modality. However, the module’s effectiveness may improve as the dataset expands and experiments are conducted to evaluate longer texts with more non-fluent markers. In addition, we removed the CNN to demonstrate the importance of feature extraction. The ablation study demonstrates that every component is essential and cannot be omitted.

6. Conclusions

Assessing speech fluency is a critical evaluation criterion for spontaneous speech. However, automatic fluency assessment of spontaneous speech is a challenging task that heavily relies on the accuracy of ASR. In this paper, we propose a multimodal method for the automatic speech fluency assessment of spontaneous speech that combines ASR output. We first create a dataset called PSCPSF for propositional speaking fluency assessment, consisting of 9.26 h of speech. To validate the relevance of the speech assessment task to the ASR task, the multi-task learning framework is used to fine-tune the Wav2Vec2.0 model on the specific datasets. By jointly optimizing the model using the speech fluency assessment task as the main task and the ASR task as the auxiliary task, we are able to improve the accuracy of the model in speech fluency assessment. We then propose a multimodal automatic fluency assessment method that combines ASR output. Specifically, we combine the final layer of the fine-tuned Wav2Vec2.0 model’s hidden output with the decoded text output, followed by a cross-modal attention mechanism to fuse audio and text features. This results in more effective fluency features, allowing for more accurate speech fluency assessment. Experiments on the PSCPSF and Speechocean762 datasets for spontaneous speech and read-aloud scenarios demonstrate that our method performs well in the different assessment scenarios. In the future, our focus will be on three main areas. First, we plan to expand the size of our self-built dataset, which will enable us to obtain more reliable results for the fine-tuning of Wav2Vec2.0 and the evaluation of multimodal methods using larger datasets. Second, we will address the issue that the impact of text features on the experimental results is not as significant as that of speech features. To address this, we will annotate non-fluent text in spontaneous speech, such as affective words and repetitions that affect fluency. Finally, we aim to explore more efficient multimodal evaluation methods to continuously improve the accuracy of fluency assessment and ASR.