Formant Tracking Using Quasi-Closed Phase Forward-Backward Linear Prediction Analysis and Deep Neural Networks

Formant tracking is investigated in this study by using trackers based on dynamic programming (DP) and deep neural nets (DNNs). Using the DP approach, six formant estimation methods were first compared. The six methods include linear prediction (LP) algorithms, weighted LP algorithms and the recently developed quasi-closed phase forward-backward (QCP-FB) method. QCP-FB gave the best performance in the comparison. Therefore, a novel formant tracking approach, which combines benefits of deep learning and signal processing based on QCP-FB, was proposed. In this approach, the formants predicted by a DNN-based tracker from a speech frame are refined using the peaks of the all-pole spectrum computed by QCP-FB from the same frame. Results show that the proposed DNN-based tracker performed better both in detection rate and estimation error for the lowest three formants compared to reference formant trackers. Compared to the popular Wavesurfer, for example, the proposed tracker gave a reduction of 29%, 48% and 35% in the estimation error for the lowest three formants, respectively.


I. INTRODUCTION
Estimation and tracking of formant frequencies is an important research topic in several areas of speech science and technology [1]- [6]. During the past few decades, many techniques have been proposed for formant tracking [7]- [10]. These algorithms typically consist of two parts, the estimation stage and the tracking stage. In the former, initial estimates of the vocal tract resonances (VTRs) are computed in short frames (e.g., 25 ms) using spectral estimation methods such as linear prediction (LP). In the latter, the formants estimated from individual frames are expressed using contours which cover a longer unit (e.g., word or sentence) [7], [8]. In addition, estimation and tracking can be done simultaneously using an initial representation of the vocal tract system [9], [10]. In both approaches, accurate estimation of VTRs is an important and necessary computational block.
LP is the most widely used technique to estimate VTRs from speech [11] and therefore many variants of LP have been proposed (e.g. [12], [13]). In formant estimation and tracking, the most popular variants are the autocorrelation and covariance methods [7], [8]. The closed phase (CP) analysis is known to improve VTR estimates by avoiding the contribution of the speech samples in the open phase of the glottal cycle thereby decoupling the effect of the trachea more effectively [14]. CP analysis, however, works better for low-pitched voices which typically have a larger number of samples in the closed phase of the glottal cycle compared to high-pitched voices which might have just a few samples in the closed phase. To reduce problems caused by having a small number of closed phase samples, LP can be computed over multiple neighboring cycles [14].
Weighted linear prediction (WLP) is an all-pole modeling method based on temporally weighting the prediction error [13], [15]- [20]. Temporal weighting of the prediction error has been shown to be beneficial in computing vocal tract models which are robust with respect to noise and the selection of analysis window as well as the biasing effect of high fundamental frequency. Formant estimation of highpitched vowels was studied using WLP in [18] by developing a simple weighting function, called the attenuated main excitation (AME) function, to downgrade the strong effect of the glottal source in the computation of the vocal tract model. Based on [18], the quasi-closed phase (QCP) method was proposed for glottal inverse filtering (GIF) in [19]. In QCP, a more generalized AME-type of weighting function is used. Recently, a new formant estimation method based on QCP, called quasi-closed phase forward-backward (QCP-FB) LP analysis, was proposed in [21]. QCP-FB combines two approaches: (1) QCP analysis in which the residual is temporally weighted, and (2) forward-backward (FB) analysis in which the number of samples is increased in LP by using two prediction directions simultaneously. In addition, WLP methods have been proposed recently based on using stochastic approaches in the computation of the weighting function [22].
In this article, formant tracking is studied by investigating different all-pole modeling methods in formant estimation. The all-pole formant estimation methods are used with two formant tracking approaches, a dynamic programming (DP) -based approach and a deep neural net (DNN) -based approach. As the first part of the study, six different LP-based and WLP-based formant estimation methods are compared in formant tracking using a DP-based tracker. The novelty of this part is in studying how the potential new method, QCP-FB, which was investigated solely in formant estimation in [21], works in formant tracking. In the second part of the study, two most potential all-pole modeling methods from the first part are used with a modern DNN-based tracker by proposing a novel formant tracking approach, which combines benefits of the data-driven deep learning approach and benefits of the model-driven all-pole modeling approach. In this novel tracking approach, the formants, which are predicted by the DNN from a given speech frame, are refined using the spectral peaks, which are indicated by the spectrum, which is computed from the same frame with a model-based parametric all-pole spectral estimation method. Altogether five known formant trackers (Wavesurfer [8], PRAAT [7], MUST [23], KARMA [10], and Deep Formants [24]) are used as reference methods in this study.
The contributions of the study are as follows: • The potential new formant estimation method, QCP-FB, is evaluated in formant tracking and its performance is compared with existing LP-based and WLP-based formant estimation methods using a DP-based tracker. • A novel formant tracking technique is proposed by combining the data-driven DNN-based approach and the model-driven all-pole approach. In this technique, the formants predicted from a speech frame by a DNN are refined using the spectral peaks that are extracted from an all-pole model, which is computed from the same frame. • A systematic investigation is carried out by comparing the novel formant tracking method described above with five reference formant trackers (Wavesurfer [8], PRAAT [7], MUST [23], KARMA [10], and Deep Formants [24]). The paper is organised as follows. The QCP-FB method, which was introduced as a new formant estimation method recently in [21], is first described in section II. The other formant estimation methods and the formant trackers used in the study are described in section III. The results of the formant tracking experiments are reported in section IV. Finally, conclusions are drawn in section V.

II. QUASI-CLOSED PHASE FORWARD-BACKWARD ANALYSIS
The traditional formulation of LP is based on forward prediction in which the current speech sample is predicted from the past p samples. It is, however, also possible to use backward prediction in which the current sample is predicted from the future p samples. The filter coefficients computed using forward and backward predictions are inter-convertible, and therefore they do not carry any additional information when computed separately. However, by simultaneously using both backward and forward prediction, a prediction model different from that of traditional LP is obtained by using forwardbackward (FB) analysis, where the current sample is predicted based on past and future samples using a common set of p coefficients. The combined error to be minimized is given by where and denote the forward and backward errors, respectively, x n denotes the current speech sample, and a k denotes the prediction coefficients. The prediction coefficients can be computed by minimizing the combined error (∂E/∂a i = 0, 1 ≤ i ≤ p) which results in the following normal equations where Previous studies have shown that FB analysis reduces the dependency of spectral estimates on the initial sinusoidal phase, shifting of frequency estimates due to additive noise and the so called line-splitting problem (see [21] for a review). The line-splitting problem refers to obtaining spectral models which show a single sinusoidal component incorrectly as two distinct peaks. By taking advantage of FB analysis, two benefits are achieved: (1) the estimated spectral peak locations are less sensitive to the window position, and (2) the combination of the two prediction directions gives more samples to compute correlations for the given frame.
Quasi-closed phase forward-backward (QCP-FB) analysis involves the use of FB analysis within the framework of QCP in order to combine the benefits of both techniques. The resulting method imposes the temporal QCP weighting function w n , defined by [19], on the forward and backward errors individually. The combined error to be minimized is given by where and are the weighted forward and backward errors, respectively. The resulting normal equations are given by Appropriate choice of range for the variable n results in the autocorrelation or covariance methods for QCP-FB. QCP-FB is used in formant tracking in the current study and it is expected to show improved performance compared to existing formant tracking methods due to the following two main reasons. First, FB analysis helps to improve the formant estimation by providing more samples for prediction, and by reducing the problems of window positioning and line splitting. Second, QCP analysis exploits the WLP framework of sample selective prediction by designing a temporal weighting function that gives more emphasis on closed phase regions and deemphasizes the open phase as well as the region immediately after the main excitation. This results in more accurate closed phase estimates of the vocal tract system with a reduced influence from the glottal source.

III. FORMANT TRACKERS
Several formant tracking algorithms have been proposed in the literature [7]- [10], [24]. It is worth emphasising that a formant tracking algorithm will most likely show varying performance when combined with different formant estimation methods and this makes it difficult to compare different tracking algorithms. In principle, most of the tracking algorithms can be combined with any formant estimation method. Therefore, formant tracking is studied in this paper using trackers which are based on both DP and DNN.

A. DP-BASED FORMANT TRACKERS
Using the DP-based tracking algorithm proposed in [8], formant tracking performance was investigated by comparing six different formant estimation methods that all use all-pole modeling. These methods, listed in Table 1, are as follows: (1) conventional LP based on the autocorrelation method (LP-ACOR), (2) conventional LP based on the covariance method (LP-COV), (3) LP based on forward-backward prediction and the covariance method (LP-FBCOV), (4) QCP analysis based on the autocorrelation method (QCP-ACOR), (5) QCP analysis based on the covariance method (QCP-COV) and (6) QCP analysis based on forward-backward prediction and the covariance method (QCP-FBCOV). All these methods were computed using a frame length of 25 ms, a frame shift of 10 ms and an all-pole model order p=12. Speech signals, sampled using 8 kHz, were pre-emphasised using an FIR filter (P (z) = 1 − 0.5z −1 ). In the autocorrelation methods, the Hamming window was used. In the covariance methods, the rectangular window was used. The peaks in the spectrum were detected by convolving the spectrum using a Gaussian derivative window with a width of 100 Hz and picking the negative zero-crossings. Five most energetic peaks of the spectrum were selected as the formant candidates. A verbatim MATLAB implementation of the tracking algorithm [8] was used to track the best four contours from the underlying formant candidates estimated by the all-pole methods.

B. DNN-BASED FORMANT TRACKERS
In order to study the possible limitations of the DP-based tracker, a deep neural network (DNN) -based formant tracker was developed as an alternative. A simple four-layer feedforward DNN was used to capture the nonlinear mapping between the spectrum and the formant frequencies. The DNN had 300 units with tangent-hyperbolic activation in each of the three hidden layers [25]. The input dimension of 143 units corresponded to 13 RASTA-PLP [26] cepstral coefficients with an 11-frame neighborhood, and the three linear output units corresponded to the first (F 1 ), second (F 2 ) and third (F 3 ) formant to be predicted.
A common input feature was deliberately used to have a common baseline performance, and to study the incremental improvement provided by different spectrum estimation methods when used for refinement. 300 utterances from the train subset of the VTR-TIMIT database [27] were used to train the models. Mean square error between the estimated and actual formant values was used as the objective function. All parameters of the network were initialized randomly.
The stochastic gradient descent algorithm with standard backpropagation of error was used to learn the network parameters. The dropout regularization method was used to prevent overfitting the network. Input values were normalized to the range of [0.1, 0.9] and output values were normalized to have zero mean and unit variance. The DNN -based tracker was used in three modes: (1) by predicting the lowest three formants directly, (2) by refining the formants predicted by the DNN by replacing them with the frequencies of the corresponding nearest peaks in the LP-FBCOV spectrum, (3) by refining the formants predicted by the DNN by replacing them with the frequencies of corresponding nearest peaks in the QCP-FBCOV spectrum. (Note that with the model order p=12, the LP-FBCOV spectrum and the QCP-FBCOV spectrum can show maximally six peaks). These three trackers will be referred to as DNN, DNN-LP-FBCOV and DNN-QCP-FBCOV, respectively. It is worth emphasizing that the latter two modes combine a data-driven approach and a model-driven approach in formant tracking in a novel way: formants F 1 -F 3 are first predicted using a data-driven deep learning approach from a given frame with the DNN after which the predicted formants are refined using a modeldriven signal processing approach using the all-pole spectrum extracted from the frame.

C. REFERENCE FORMANT TRACKERS
The DP-based and the DNN-based formant tracking algorithms were compared to known formant trackers. These reference trackers include algorithms used in two popular speech analysis tools (Wavesurfer [8] and PRAAT [7]), the adaptive filter bank (AFB) -based formant tracking algorithm (denoted as MUST) [23], KARMA (based on Kalman filtering) [10], and Deep Formants (based on DNNs) [24], [28]. Both Wavesurfer and PRAAT use LP analysis followed by DP-based tracking. Wavesurfer was used in two forms corresponding to autocorrelation LP and stabilized covariance LP which are referred to as WSURF-0 and WSURF-1, respectively [8]. The PRAAT algorithm uses the BURG method in LP analysis [7]. All the algorithms tracked four formants from the top five formant candidates derived from the underlying spectrum at a frame rate of 100 Hz.

A. DATABASE AND PERFORMANCE METRICS
The formant tracking performance was evaluated using the VTR database, which is one of the most widely used speech databases in the areas of formant estimation and tracking [27]. The test data of the database was used for the evaluation. This data consists of 192 utterances (produced by 8 female and 16 male speakers, each pronouncing 8 utterances). The duration of each utterance varies between 2 and 5 s. The ground truth (i.e., formant frequencies) have been derived using a semi-supervised LP-based method [29]. The values of F 1 -F 3 have been corrected manually using spectrograms. The ground truth values for formants are provided for every 10 ms interval. The formant tracking performance was evaluated using two known metrics: the formant detection rate (FDR) and the formant estimation error (FEE) as defined in [21]. During the performance evaluation, the reference ground truth for each of the lowest three formants was associated with the nearest formant candidate lying within a specified relative (τ p ) and absolute (τ a ) deviations.
The FDR is computed in terms of the percentage of frames for which a hypothesized formant occurs within a specified deviation from the ground truth formant. The FDR for the i th formant over M analysis frames is computed as where I(.) denotes a binary formant detector function. ∆F i,n = |F i,n −F i,n | is the absolute deviation of the hypothesized formant frequency (F i,n ) from the reference ground truth (F i,n ) at the n th frame for the i th formant. The FEE is computed in terms of the average absolute deviation of the hypothesized formant from the ground truth formant. The FEE for the i th formant over M analysis frames is computed as The FEE values in conjunction with FDR values give a better sense of the performance of a formant tracker.

B. DP-BASED FORMANT TRACKING
The FDRs (within τ p = 30% and τ a = 300 Hz deviation) and FEEs for the different formant estimation methods are given in Table 1. In this table, the two metrics are computed by associating the three hypothesized formant tracks with the lowest three reference tracks. The scores in the parentheses, however, denote the best scores which were obtained by identifying each formant as the spectral peak (among the detected five candidates) that was closest to the corresponding reference formant. The scores in the parentheses describe the performance of the underlying formant estimation method when used with an ideal formant tracker. It can be seen from the results that the DP-based tracker gave scores inferior to the detection potential of the underlying spectral estimates.
It can be seen from Table 1 that the QCP-based methods performed consistently better than all of their LP-based counterparts. The covariance method performed better than the autocorrelation method for both LP and QCP. However, LP-FBCOV showed no improvement over LP-COV, and QCP-FBCOV seems to be inferior to QCP-COV, despite the detection potential being highest (the scores in parentheses) for  QCP-FBCOV. This behavior can be attributed to the inherent limitations of the DP-based tracker with a possibility of tracking a spurious candidate instead of the best candidate (which is otherwise not known without the ground truth).

C. DNN-BASED FORMANT TRACKING
A comparison of the performance of the DNN-based formant tracker using LP-FBCOV and QCP-FBCOV for refinement is given in Table 2 along with the performance of the DNN predictor. Three reference trackers (Wavesurfer (WSURF-1), KARMA, and Deep Formants) were chosen for comparison based on their performance shown in Table 1. It can be seen that the DNN-QCP-FBCOV tracker performed best, almost realizing the full potential of the QCP-FBCOV method (the scores in parentheses in Table 1). The improvement given by DNN-QCP-FBCOV compared particularly to the popular Wavesurfer tracker is large showing a reduction of 29%, 48% and 35% in the estimation error for the lowest three formants, respectively. These results demonstrate that the QCP-FBCOV method can be a good replacement for the popularly used LP-COV analysis in formant estimation and VOLUME 9, 2021 tracking tools and applications. A detailed comparison in the formant tracking performance of KARMA, Deep Formants, DNN and DNN-QCP-FBCOV is given in Table 3 for different phonetic categories (vowels, dipthongs and semovowels). It can be seen that the proposed DNN-QCP-FBCOV method performed clearly better for all the phonetic categories in both FDR and FEE. Formant tracking performance of different methods analyzed separately for male and female speakers is given in Table 4. From the table it can be observed that the performance of Deep Formants is better for male speakers (except in δF 1 , where the DNN-QCP-FBCOV method is better) but for female speakers the DNN-QCP-FBCOV method is better.
Formant tracking performance for different methods using speech degraded with white and babble noise at signal-tonoise ratio (SNR) levels of 10 dB and 5 dB are given in Table 5. From the table it can be observed that the proposed DNN-QCP-FBCOV method performed better in the case of speech degraded with white noise. In the case of speech degraded with babble noise, Deep Formants and DNN methods seems to perform better.
An illustration of formant tracking by KARMA and DNN-QCP-FBCOV for an utterance produced by a male speaker is shown in Fig. ??. It can be seen from the figure that the formants tracked by the DNN-QCP-FBCOV method match closely the ground truth in voiced segments. Furthermore, it can be clearly seen that DNN-QCP-FBCOV is better than KARMA in tracking all the formants.

V. CONCLUSIONS
Formant tracking was studied in this paper based on the widely used two-stage approach consisting of the estimation stage and the tracking stage. In the former, six different all-pole modeling methods were first compared with a DPbased tracker. In addition, five known formant trackers were used as references. Two most potential all-pole modeling methods (LP-FBCOV and QCP-FBCOV) were then used with a modern DNN-based tracker by proposing a novel formant tracking technique which combines benefits of datadriven and model-driven approaches: the formants predicted with the data-driven DNN were refined using the frequencies of the peaks in the all-pole spectra computed by the modeldriven LP-FBCOV and QCP-FBCOV methods.
The DNN-based formant trackers using the LP-FBCOV and QCP-FBCOV refinement were further compared to two conventional formant trackers (Wavesurfer and KARMA) and to one recently published DNN-based tracker (Deep Formants). With the QCP-FBCOV refinement, the DNNbased tracker outperformed the conventional reference formant trackers in all metrics. Compared to Deep Formants, the proposed DNN-tracker gave better performance in all other metrics except for FDR and FEE in F 3 where Deep Formants was just slightly better. In addition to these encouraging objective results, it is worth emphasising that the proposed QCP-FBCOV refinement technique can be used in principle to improve the performance of any existing DNN-based formant tracker which has been trained to map a speech signal frame into formants, that is, there is no need to re-train the DNN-based tracker used. However, it should be noted that the performance of the proposed method might depend on the accuracy of the estimated glottal closure instants, which are needed to generate the QCP weighting function [19]. Therefore, the robustness of the proposed method in various noisy conditions needs to be studied further. One way of improving the performance under degradations could be by training the DNN models for all noisy conditions of interest. Nevertheless, under clean conditions, the current study shows that the QCP-FBCOV method is a potential all-pole modeling technique to be used in formant tracking instead of the widely used conventional LP methods.
It is worth noting that the DNN-based formant tracking methods studied in this investigation (i.e. Deep Formants, DNN, DNN-LP-FBCOV, and DNN-QCP-FBCOV) are based on supervised learning and their computational complexity is relatively high compared to traditional model-based approaches. It is known that DNNs are resource hungry due to their need for training data and the architecture of the neural network adds more computational complexity when the trained network is used in formant tracking. The LSTMbased Deep Formants architecture has approximately 4M parameters, while the FFNN DNN architecture we propose has around 0.3M parameters. It is worth emphasizing, however, that the main contribution of this study, the QCP-FBCOV based refinement of formants, can be plugged into any existing pre-trained DNN-based tracker, which results only in a marginal increase in complexity. Compared to the conventional autocorrelation based LP, which has a computational complexity of O(n 2 ), our proposed QCP-FBCOVbased tracker has O(n 3 ) complexity, where n denotes the size of the covariance matrix (which is equal to the LP order, which was p = 12 in the experiments of the current study). However, the order of LP analysis being small, our proposed DNN-QCP-FBCOV-based formant tracking results only in a negligible increase in the overall computational complexity. There is an added computation complexity due to the computation of the temporal weighting function (w n in Eqs. 7 and 8), which calls for estimating glottal closure instants, which also requires LP inverse filtering and is proportional to O(n 3 ) computations.