Recognition of English speech – using a deep learning algorithm

2.1 Traditional English speech recognition method

The traditional English speech recognition process is shown in Figure 1. The first step is the acquisition of speech samples, followed by pre-processing of speech samples and feature extraction [8]. After the speech pre-processing, features are extracted from every frame of speech signal using the Mel-frequency cepstral coefficient method [9]. Then, an acoustic model converts the speech features into the most probable phoneme. The acoustic model is divided into two parts: one is to convert the speech features into the most probable state prediction value using the GMM [10], and the other is to convert the state prediction value into the most probable phoneme using the HMM. Finally, the phonemes are converted into words to form a sentence with the highest likelihood using a language model.

Figure 1

Traditional English speech recognition process.

2.2 English speech recognition based on deep learning

As mentioned above, the traditional English speech recognition method divides the recognition process into several stages, and the models used in different stages are independent of each other [11]. A complex decoder is required to decode the acoustic and language models in the process of actual use, which is tedious. In addition, the training of acoustic models in the traditional English speech recognition method also requires the forced alignment of features and states.

Compared with the traditional English speech algorithm, the deep learning-based English speech recognition algorithm integrates the acoustic and language models into a deep learning neural network [12], that is, integrates the parameters of the acoustic and language models into the parameters of the neural network. The parameters of the recognition model are optimized by training the neural network so that the speech features can be directly converted into words through the neural network. In the process of recognizing continuous speech, the audio signal of speech is first divided into frames before recognition; that is, the characters represented by every audio frame are recognized, and finally, the characters of every frame are combined together. However, even when the same person speaks the same sentence, the length of speech may vary; because of this, a single phoneme may be split into multiple frames, making the frame sequence length of the audio much longer than the actual phoneme sequence; that is, the audio frame sequence cannot correspond to the actual phoneme sequence. Therefore, CTC is used to solve the problem of forced alignment between input and output sequences in traditional speech recognition algorithms [13]. The basic principle of the CTC algorithm for forced alignment is as follows. A blank character “-” is added to the candidate phonemes (characters) of the recognition model to make the recognition model get the candidate phoneme sequence frame by frame. The blank character is deleted after merging the consecutive repeated phonemes.

An RNN that can use historical information is used to recognize English speech [14]. Unlike a backpropagation neural network and a CNN, the hidden nodes in an RNN are computed by taking into account the input of the input layer at the current moment and the influence of the state of the hidden nodes at the last moment, in line with the temporal characteristics of the speech signal.

Figure 2

Training and use process of the deep learning-based English speech recognition algorithm.

Figure 2 shows the training and use process of the deep learning-based English speech recognition algorithm. The detailed steps are as follows.

• ① Like the traditional speech recognition algorithms, the input speech samples are processed by pre-emphasis, windowing [15], and feature extraction.

• ② The input layer nodes of the RNN are input with the speech feature vector of one frame at one moment, and the number of input layer nodes depends on the dimension of the speech feature vector of one frame.

• ③ The calculation is performed in the hidden layer nodes of the RNN:

  (1) h t = f ( w h x x t + w h h h t − 1 ) y t = g ( w h y h t ) ,

  where h t is the state vector of the nodes in the hidden layer at time t , x t is the speech feature vector input into the input layer at time t , h t − 1 is the state vector of the nodes in the hidden layer at the time t − 1 , y t is the output vector of the output layer at time t (the output vector of the whole output layer is the label probability distribution) [16], w h x is the weight between the input and hidden nodes, w h h is the weight between the hidden nodes, w h h is the weight between the hidden and output nodes, f ( ⋅ ) is the activation function of the hidden layer, and g ( ⋅ ) is the activation function of the output layer.

• ④ After obtaining the label probability distribution in the output layer, it is determined whether it is currently in the training phase. If it is not in the training phase, the beam search algorithm [17] is used to decode the label distribution probabilities arranged in chronological order and output by the output layer to obtain the word sequence corresponding to speech.

• ⑤ If it is still in the training phase, the negative log-likelihood of the output label sequence is calculated in the CTC layer based on the corresponding label sequences in the training samples and the label distribution probability sequences given in the output layer and used as the loss in the training process. The calculation formula is as follows:

  (2) L CTC = − ln ∑ u α t u β t u ,

  where L CTC is the training loss, α t u is the sum of the forward probabilities of label u in the training sample corresponding to the label sequence at time t , and β t u is the sum of the backward probabilities of label u in the training sample corresponding to the label sequence at time t [18].

• ⑥ It is determined whether the training is finished. If the training loss converges to stable or the number of iterations reaches the preset number, the training is finished; if the training is not finished, the weight parameters are reversely adjusted in the RNN according to the training loss, and then, it returns to step ③.

3.1 Experimental environment

MATLAB software [19] in the laboratory server was used to conduct simulation experiments on the deep learning-based English speech recognition algorithm.

3.2 Experimental setup

The speech dataset used to conduct the simulation experiments was the publicly available English speech recognition dataset TIMIT [20], which has sampling parameters of 16 kHz and 16 bits. There are 630 participants, and it includes 6,300 sentences. The phoneme level of every sentence was manually divided and labeled.

In the RNN-based English speech recognition algorithm, the relevant parameters of the RNN network are shown below. The number of nodes in the input layer was set as 39 according to the feature dimension obtained by the Mel-frequency cepstral coefficient method. The number of nodes in the output layer was set as 50 according to the number of phoneme tags and the blank and termination characters in the speech data set. The number of nodes in the hidden layer was set as 200. The activation function in the hidden layer was the sigmoid function, and the activation function in the output layer was set as the softmax function [21].

In order to further verify the recognition accuracy of the RNN-based English speech algorithm, this study also used GMM-HMM-based and CNN-based speech recognition algorithms for comparison. The parameters of the GMM-HMM-based speech recognition algorithm are shown below. The number of states of speech samples in the HMM model was set as 6, and the number of possible observations of every state was set as 4. The structural parameters of the CNN algorithm in the CNN-based speech recognition algorithm are shown below. The input specification of the input layer was set as 39 × 1. The number of nodes in the output layer was set as 50. The number of convolutional layers was set as 2, every convolutional layer had 32 convolutional kernels in a size of 2 × 1, and the sigmoid function was used in convolution calculation. The number of pooling layers was set as 1, the pooling box of every pooling layer was set as 2 × 1, the step length of the pooling box was 2 when sliding on the feature map, and mean-pooling was used in the pooling box.

The aforementioned algorithm parameters were obtained through orthogonal experiments.

3.3 Experimental items

① The English speech data set was divided into a training set and a test set; 500, 1,000, 1,500, 2,000, and 2,500 sentences were set in the training set, and 2,000 sentences were set in the test set, respectively, to test the recognition performance of three speech recognition algorithms for English speech under different numbers of samples in the training set. The training and the test time were also recorded.

② The English speech data set was divided into a training set and a test set; 2,500 sentences were set in the training set, and 500, 1,000, 1,500, 2,000, and 2,500 sentences were set in the test set. The English speech recognition performance of the three speech recognition algorithms was tested under different numbers of sentences in the test set. The training and the test time were also recorded.

3.4 Performance evaluation criteria

The word error rate [15] was used to evaluate the results after recognition by the speech recognition algorithm, and the calculation formula is as follows:

where X is the number of substituted words, Y is the number of missing words, Z is the number of inserted words, and P is the total number of words.

3.5 Experimental results

The recognition performance of the three speech recognition algorithms trained with different numbers of training samples after testing with the same number of samples in the test set is shown in Figure 3. Table 2 shows the training time of the three speech recognition algorithms under different numbers of samples in the training set and the test time. It was seen from Figure 3 that the recognition error rate of the three speech recognition algorithms under the same number of test samples gradually decreased as the number of training samples in the training set increased, while the recognition error rate of the GMM-HMM algorithm was the highest, the CNN-CTC algorithm was the second, and the RNN-CTC algorithm was the lowest when the number of training samples was the same.

Figure 3

Recognition performance of three speech recognition algorithms trained on training sets with different numbers of samples.

As could be seen from Table 1, the time spent by the three speech recognition algorithms in the training phase increased as the number of training samples in the training set increased; under the same number of training samples, the GMM-HMM algorithm took the most time to train, the CNN-CTC algorithm the second, and the RNN-CTC algorithm the least, but the test time was not increased when they were tested by the test set. In addition, the GMM-HMM algorithm spent the longest time in testing, the CNN-CTC algorithm the second, and the RNN-CTC algorithm the least.

After training on a training set containing 2,000 training samples, the three speech recognition algorithms were tested for speech recognition on a test set with different number of samples. The variation of their word error rates with the number of test samples is shown in Figure 4. Table 2 shows the corresponding training and testing time. It was seen from Figure 4 that the word error rates of the three speech recognition algorithms increased as the number of test samples increased, and the word error rate of the GMM-HMM algorithm was the highest, the CNN-CTC algorithm was the second, and the RNN-CTC algorithm was the lowest under the same number of test samples.

Figure 4

Recognition performance of three speech recognition algorithms for test sets with different numbers of samples.

As could be seen from Table 2, since the number of samples in the training set was constant at 2,000 sentences, the training time of the algorithms before testing on the test set with different numbers of test samples was nearly the same, and the training time of the GMM-HMM algorithm was the longest, followed by CNN-CTC and RNN-CTC algorithms. In terms of test time, the time consumed by all three speech recognition algorithms increased as the number of test samples increased; the test time consumed by the GMM-HMM algorithm was the longest, the CNN-CTC algorithm was the second, and the RNN-CTC algorithm was the least under the same number of test samples.