Artificial intelligence to understand fluctuation of fetal brain activity by recognizing facial expressions

To examine whether artificial intelligence can achieve discoveries regarding fetal brain activity.

in utero pain or stress, 9 and a neutral expression and tongue expulsion have been unclear regarding brain function. 8 Despite the apparent importance of investigating fetal facial expressions, there have been no standard methods for objective and quantitative evaluation.
Artificial intelligence (AI) has recently advanced and become prominent in obstetrics and gynecology, [14][15][16][17][18] and we have reported on an AI classifier that can classify seven fetal facial expression categories and demonstrate the probabilities of each category as confidence scores. 14,15 The accuracy values of the classifier were 0.996, 0.996, 1.000, 1.000, 1.000, 1.000, 1.000, and 0.996 for eye blinking, neutral, mouthing, scowling, smiling, tongue expulsion, yawning, and all categories, respectively.
In the present study, we applied the AI classifier to video files of fetal facial images and investigated whether the confidence score variations could be detected using discrete Fourier transform (DFT) and chaotic analysis. This recognizes the order of time series data and allows analysis while preserving the data order for objectively and quantitatively investigating the nature of fetal brain activity.

| MATERIAL S AND ME THODS
We recorded and stored videos of fetal faces with a four-dimensional ultrasound technique obtained from consecutive singleton pregnancies of outpatients in routine conventional practice at 27 to 37 weeks of gestation. This was performed in MP4 format at a rate of 10.008 frames per second using a GE Voluson E10 BT20 (GE Healthcare).
Informed consent was obtained from all participants at Miyake Clinic between February 1 and December 31, 2021, with completely deidentified data enrolled. 14, 15 We transferred the videos to an offline AI system at Medical Data Labo, Japan. This retrospective, noninterventional study was approved by the institutional review board of Miyake Clinic (institutional review board number 2019-10). 14, 15 We converted each video frame into separated JPG-format images, from which the region of interest was cropped into a square and saved at 100 × 100 pixels. The AI classifier then classified each image and output the seven confidence scores belonging to each expression category. We then used the confidence scores to obtain a sevendimensional vector of the time series per fetus.
The confidence score profile was subjectively classified into two categories: dense and sparse, with the former showing substantial fluctuation of the numerical value and the latter showing little fluctuation. We divided the vector by a length of 200 frames, equivalent to a seemingly appropriate length of 20 s, consisting of either dense or sparse time. Each unit was defined as a set. Final sets of video frames with lengths exceeding 150 frames were used in the analysis.
We analyzed all of the sets with the following DFT and chaotic analysis by the fetus' sex, mother's parity, gestational week, and dense/ sparse profile.
We obtained the first five power spectra and corresponding periods for each facial category of all of the sets by: where i 2 = −1, j is the facial category number, and N is the number of video frames.
We applied the seven-dimensional vector to a practical algorithm to determine the character of strange attractors [19][20][21][22][23] to analyze multidimensional data. For a seven-dimensional vector of a time series, we reconstructed the vector x i by shifting time τ.
where τ is the time, j is the facial category number, m is the embedding dimension, and N is the number of video frames.
We then calculated the correlation dimension, D 2 , as follows: A sample confidence score profile comprising seven categories of fetal facial expressions. We recognized sparse and dense states.
where r is {r ∈ ℝ | r > 0} and Q is a Heaviside step function.
We then obtained generalized dimensions, 24 D q , as follows: where n(r) is the number of spheres required to cover the object (orbit) with a sphere of radius r, and q is {q ∈ ℝ}.
We obtained the capacity dimension, D 0 , as follows: We obtained the information dimension, D 1 , as follows: We obtained the α-f(α) profile graphs by using D qj . 19 We applied these chaos analyses not only for a seven-dimensional was used for all analyses as well as statistical analyses, and we used Mann-Whitney test, t test, variance test, and one-way analysis of variance (ANOVA). We set p < 0.05 as statistically significant. There was a significant difference in confidence scores among the categories of facial expressions (p < 1.00 × 10 −11 by one-way ANOVA) (Table 1, Figure 2). Mouthing was most prevalent (47%-49%), followed by neutral (37%-40%). The frequencies of other facial expressions were <5%; notably, scowling, smiling, and tongue expulsion were all <2%.

| RE SULTS
The power spectrum by DFT showed that mouthing and neutral components were the most prevalent (Table 2). Two hundred seconds, equal to the frame length, was the maximum period in all categories. Neutral had the largest power spectrum, followed by mouthing. There was a difference between categories for the power spectrum (p = 0.004 by one-way ANOVA). There were no differences in periods between the categories.
The mean value of D 2 tended to be smaller for mouthing. The median value tended to be smaller for neutral. Tongue expulsion tended to have larger mean and median values (Table 3). There were, however, no significant differences (p = 0.068).
In comparing confidence scores in the sparse and dense states of confidence score profiles, mouthing and neutral dominated in the dense state, at ≈40% each (Table 4). Even in the sparse state, these two categories were the most prevalent, with mouthing accounting for 65.7% and neutral for 18.3%. In all categories, there were significant differences between the sparse and dense state in mean, median, and variance (p < 0.001). Confidence scores varied significantly (p < 0.001).  Figure 3). There was no difference by variance between the two states.

| DISCUSS ION
This study presented a method for objectively and quantitatively analyzing changes in fetal facial expressions that appear related to fetal brain activity. To our knowledge, no such method has ever existed. We discovered that the expressions had two different states--dense and sparse--of confidence score profiles generated by AI (Table 6, Figure 3).
The DFT informed on the period of facial expression change. It is naturally impossible to compare with electroencephalogram (EEG), but we considered that this period reflected fetal brain activity. The confidence score list is time series data, and therefore the order should not be random and the DFT cannot evaluate the relationship between a given facial state and the states immediately before and after that state. We thus applied chaotic analysis, which included the order of time series data. 19 The EEG cycle in infants is in the range of 0.5 to 2.5 s, 25 which naturally cannot be directly compared with the cycle of fetal facial expression change. DFT indicated that the average period of facial expression was 66 to 73 s ( Table 2). Mouthing and neutral components were the most prevalent in the power spectrum. We observed that 200 s, equaling the number of frames in a set, was the maximum period for all categories. This finding suggests that longer recording of facial expressions is desirable. There was a difference between categories for the power spectrum (p = 0.004), and the spectrum was sometimes undetectable for scowling, smiling, tongue expulsion, and yawning. The 66-to 73-s period obtained in this study evidently is the first-ever finding observed regarding motor activity due to fetal brain activity. There were significant differences in the power spectrum, with mouthing and neutral clearly more prevalent.
We speculated that the states observed in mouthing and neutral in facial expression recognition were the fundamental, basic states of the fetal brain. Among humans, the fact that the lips are already moving in the fetal period, even if they are not vocalizing, could be preparation for suckling immediately after birth. The significance of a large power spectrum for mouthing is reasonable and pertinent.
Rare expressions such as scowling, smiling, and yawning are not considered to be fundamental actions in fetal life.
In our chaotic dimensional analysis of fetal facial expressions, we obtained embedding dimensions--D 0 , D 1 , and D 2 --for all expression categories in all sets. However, the chaotic attractor could not be determined for the less frequently observed categories (data not shown). Note: We performed discrete Fourier transform (DFT) for each case on every 200 video frames in each category. The power spectrum showed that mouthing and neutral components were dominant. We observed 200 s as the maximum period in all categories. Neutral was the largest power spectrum, followed by mouthing. There was a difference between categories for the power spectrum (p = 0.004 by one-way analysis of variance). There were no differences in periods between categories.

F I G U R E 2
Confidence scores of categories of fetal facial expressions. There was a significant difference among facial categories: p = 6.50 × 10 −12 by one-way analysis of variance. Mouthing was the most common, followed by neutral. The frequency of the other facial expressions was <5%. AI indicates artificial intelligence. Note: The mean value tended to be smaller for mouthing. The median value tended to be smaller for neutral. Tongue expulsion tended to have a larger mean and median value. There were, however, no significant differences (p = 0.068).

TA B L E 3 Correlation dimension, D 2 , for confidence scores of fetal facial expression categories
Abbreviation: SD, standard deviation. Note: The dense and sparse states had 6736 and 6071 frames, respectively. Mouthing and neutral were the majority in the dense state, at ≈40% each. Even in the sparse state, these two facial expression categories dominated, with mouthing accounting for 65.7% and neutral for 18.3%. In all categories, there were significant differences between the dense and sparse states in the mean and median values. There were also significant differences in variances between the two states in all categories. Confidence scores varied significantly. Note: We selected five power spectrums by order of size and compared the mean, standard deviation (SD), and median along with the corresponding period by t test, variance test, and Mann-Whitney test, respectively. Although there was almost no difference in the period, there was a significant difference in the median of the power spectrum between the dense and sparse states in all expression categories. Especially for mouthing and neutral, the power spectrum was three to five times higher in the dense state than in the sparse state. Abbreviation: DFT, discrete Fourier transform. a Not significant; b p < 0.05; c p < 0.0001.

TA B L E 4 Comparison of confidence scores in dense and sparse states of confidence score profiles
As we obtained D 2 by checking against the embedding dimension values, 20,21 we judged D 2 to be more reliable than D 0 and D 1 . The trends of increase and decrease of D 0 , D 1 , and D 2 were nearly the same, as represented in Figure 4. We then used D 2 as the representative dimension. For D 2 , there were no statistically significant differences by fetal sex, maternal age, or gestational week (data not shown). There was no significant difference in D 2 between categories (p = 0.068), although there was a trend toward a smaller median value for neutral.
This tendency suggests the low D 2 in neutral were the basic state, and the energy needed for fetal development might be lower in neutral in terms of facial muscle movement and brain activity in utero. The fact that D 2 was smaller in neutral seemed reasonable and pertinent, as did the fact that the power spectrum was larger.
We analyzed this difference by focusing on how the same case could have both sparse and dense states, as shown in Figures 1 and 6.
As Table 4 shows, there were significant differences in mean, median, and variance between the dense and sparse states; therefore, the confidence score varied significantly between the states. Confidence scores were significantly higher in the dense state than in the sparse state for all expression categories except mouthing. It is important that mouthing and neutral were most prevalent in the dense state, accounting for ≈40% each, while in the sparse state, mouthing dominated, at 65.7%, while neutral was 18.3%. As Table 5 shows, while there was no difference in the period observed in the dense and sparse states, there was a difference in the power spectrum. When we compared the power spectrum separately for the dense and sparse states, the values for the dense state were approximately two to five times higher (Table 2). However, the power values of 0.342 for mouthing and 0.487 for neutral were relatively high even in the sparse state, and we considered the values three to five times higher in the dense state to be highly relevant ( Table 5). The DFT analysis results also suggested that these two states indicated different brain activities.
A comparison of D 2 in the dense and sparse states showed that This study is the first report of a novel, potentially objective and quantitative assessment of fetal brain activity and development. It shows that noninvasively collected information from AI can be analyzed using DFT and chaotic analysis.

| CON CLUS IONS
We developed a novel method for analyzing changes in fetal facial expressions or fetal brain activity, using AI to quantify the expressions, and then using mathematical computational processing; in this case, DFT and chaotic analysis. Confidence scores by AI, DFT, and chaotic dimensional analysis found dense and sparse states for fetal brain and facial activity. Although dimensional interpretation of the fetal brain has yet to be established and requires further research, the method we presented herein is an objective and quantitative method for understanding the state of the fetal brain. to the validation, resources, manuscript review and editing, supervision, and project administration. All of the authors confirm that they had full access to all of the data in the study and they accept the responsibility to submit for publication. All of the authors approved the final version of the article.

CO N FLI C T O F I NTE R E S T
The authors declare no conflict of interest.

DATA AVA I L A B I L I T Y S TAT E M E N T
Research data are not shared.