Machine learning classification models for fetal skeletal development performance prediction using maternal bone metabolic proteins in goats

Experimental design and animal management

All the protocols for use of animal and experimental procedure in present study were approved by the Animal Care Committee according to the Animal Care and the Use Guidelines of the Institute of Subtropical Agriculture, Chinese Academy of Sciences (ISA-CAS), Changsha, China (No. KYNEAAM-2015-0009).

The experiment was conducted to discover the influence of MNR on the fetal bone growth performance in the middle (from days 45 to 100) and late gestation (from days 96 to 135) periods. Forty healthy female goats (Liuyang black goat, a Chinese local meat-production goat breed), in 2nd parity with the similar initial body weight and genetic background were collected and assigned to four groups (10 maternal goats for each) in a completely equally randomized design. These groups included the control (C) group in middle gestation (C-M), maternal nutrition restriction (R) group in middle gestation (R-M), C group in late gestation (C-L) and R group in late gestation (R-L), respectively.

The goats in control group were fed with 100% maintenance requirements of meat-producing goats of China (MpGC-2004), where, R group with 60% maintenance requirements of MpGC-2004 (Zhou et al., 2019). All experimental goats were provided by Liuyang Black Goats Nutritional Metabolism Innovation Breeding Base of ISA-CAS, natural crossing with breeding male goats after synchronization. The mating day was recorded to calculate the gestation day. All maternal goats used were free-range grazing before the experiments of maternal nutritional restriction. In the trial period, each pregnancy goat was kept in a well-ventilated individual cage with the proper temperature and humidity with free access to clean drinking water in an adaption of 3 days.

The average daily intake was measured and recorded for each pregnant goat. The ratio of concentrate to roughage (fresh crushed Miscanthus spp.) fed was set as 4:6 for middle gestation, and 6:4 for late gestation. The ingredients of feed concentrate on a dry matter (DM) basis are shown in Table 1. In general, the amount of intake increased gradually with increasing gestational days. The initial daily intake of pregnancy goats was around 0.85–0.95 kg, 1.15–1.25 kg for middle and late gestation periods, respectively. The feed amount of R group was adjusted weekly to 60% average daily intake of C group in the previous week. Each goat was fed twice daily at 08:30 h and 17:00 h. Daily intake of concentrate and forage was recorded.

In the gestation days 100 and 135, maternal goats were slaughtered and used to collect bone and blood samples. During the experimental period, some pregnancy goats aborted for some uncontrolled reasons. In total, twenty-four maternal goats (nine goats for C-M group, six goats for R-M group; four goats for C-L and five goats for R-L) were slaughtered to collect bone and blood samples. In addition, 34 fetal goats were collected for sampling in different treatments and gestation periods, including 10 fetuses in C-M, 10 fetuses in R-M, six fetuses in C-L and eight fetuses in R-L, respectively.

Sample collecting and analysis

The vein blood and umbilical cord blood samples were collected with normal vacuum blood collection tubes from jugular vein and umbilical cord before slaughter, respectively. All blood samples were allowed to stand for 2 h, centrifuged at 3,000 g/min for 15 min at 4 °C, and separated into 1.5 ml centrifugation tubes, stored at −80 °C refrigerator until further analysis. After slaughtered, the left femur and left humerus of the young and maternal goats were collected entirely and peeled off the associated soft tissue completely. The bone samples were washed with normal saline, dried with filter paper, and recorded the bone weights. The length and diameter of the middle spine of bone were measured with a Vernier Caliper and recorded all the parameters.

We used ELISA kits to determine the dynamic profiles of some skeleton ossification-associated parameters in the serum, like parathyroid hormone (PTH), bone alkaline phosphatase (BALP), osteocalcin (also known as bone gamma-carboxyglutamic acid containing protein, BGLAP), tartrate-resistant phosphatase (TRAP), type I collagen amino terminal peptide (INTP), and type I collagen carboxy terminal peptide (CTX-I) with the specific ELISA kits. Among, the PTH kit (catalog: CSB-E13082G; lot: C0351620332), BALP kit (catalog: CSB-E13080G; lot: G13035060), BGLAP kit (catalog: CSB-EL002682GO; lot: F30035061) were purchased from CUSABIO Biotech. Co., Ltd. The TRAP kit (catalog: MBS014917; lot: #0812015), INTP kit (catalog: MBS028250; lot: #0812015), and CTX-I kit (catalog: MBS267426; lot: #36271309) were made in USA, and purchased from Well Biological Science Co., Ltd.

Database construction and modelling

Original experimental data resources

In present work, the fetus and maternal femur and humerus bone weights were collected and measured in the conditions of maintenance nutrient requirements (MNR) and nutrition restriction (60% MNR) for goats in the middle and late gestation periods. We also determined the serum bone metabolic proteins of young and maternal goats, including PTH, BALP, BGLAP, INTP, TRAP, and CTX-I. In total, there were 61 fetus instances for the bone parameters and 58 instances for the goat serum bone metabolic proteins. As bone parameters there are femur weight (Fw, g), femur length (Fl, mm), femur diameter (Fd, mm), humerus weight (Hw, g), humerus length (Hl, mm), and humerus diameter (Hd, mm). The values were measured in three types of experimental conditions (ECs): animal type (Animal), gestation periods (Period), and nutrient restriction (Treat). Animal included 34 foetuses and 24 maternal goats, gestation periods were about middle and late gestations, and nutrient restriction parameters were 100% or 60% nutrient requirements for pregnancy maternal goats.

Fetal bone developmental profiles

The effects of MNR on gestation goats for fetal bone growth profiles (including weight, length and diameter of femur and humerus) are presented in Table 2. In the middle gestation, MNR improved the fetal bone weight, but decreased the length and diameter of bone (femur and humerus). In the late gestation, MNR enhanced the weight and diameter of humerus and femur, and humerus length, but decreased the femur length. However, for the local goat nutritional research, except the nutritional status providing the nutritional requirements of fetal skeletal development, the genetic factors of each individual might also play an important role in fetal skeletal development.

Nutritional restriction on goat bone metabolic indexes

Except the nutritional supplementation, the regulators or biomarkers of bone metabolism were also taken into consideration for fetal bone development. Here, we measured the goat bone metabolic proteins (PTH, BALP, BGLAP, TRAP, INTP and CTX-1) for maternal jugular venous serum and fetal umbilical cord serum (showed in Table 3 and Table 4, respectively). For maternal goats, the nutritional factor, MNR, reduced the levels of PTH, BALP, BGLAP, TRAP, INTP and CTX-1 in peripheral serum.

Under the malnutritional condition, low maternal BGLAP level represents low efficient maternal bone turnover rate. However, our results showed that the other biochemical markers of bone turnover, TRAcP, had no difference in malnutrition condition. Compared with significant differences in bone metabolic proteins for maternal individual, fetal bone metabolic proteins showed a minor variation in the control and malnutrition groups. The NTx and CTx were reduced in response to maternal malnutrition, CTx was obviously lower in the middle-gestation, while NTx significantly lower in late-gestation. The PTH for maternal malnutrition was higher or lower in middle- and late- gestations, respectively. Whereas, the fetal BALP, BGLAP, and TRAP showed no statistical difference between the control and malnutritional groups.

Machine learning predictive models

Seventy-two datasets were used to find the best ML model to predict six bone parameters by using seven ML methods such as KNN, SVM linear (linear kernels), SVM (RBF kernels), LR, DT, RF and XGB. The input features came from 6 one-hot encoding of ECs, six metabolic original data, six mixed MAs of metabolic data, 18 MAs of metabolic data for each ECs, 1 mixed probability, and three EC probabilities. The Python script was used Robust scaler for data scaling, cross-validation (n-fold CV, where n = 2, 3, 5, 10 and LOOCV), and balanced accuracy as performance score.

Table 5 presents the best accuracy (ACC) values obtained for all outputs using only LOOCV. For 5-, 10-fold CV results or other details, see the repository in GitLab (Munteanu & Liu, 2019a) or GitHub (Munteanu & Liu, 2019b).

The best CV method for a small dataset was the LOOCV one. The results based on this CV showed models with all ACC >0.70, with classifiers of five from six outputs with ACC between 0.75 and 0.90, and one classifier with ACC >0.90. In this study, we observed the performance of SVM method with bone dataset and LOOCV (all classifiers are SVM/SVM linear). The most preferred inputs were MAmix, and Metab. MAi was used only for one output and probabilities are missing. The best classifier with ACC of 0.911 was obtained for prediction of Fw by using SVM with RBF non-linear kernels and three types of inputs such as MAi, MAmix, and Metab. There were three outputs (Fl, Hw and Hl) where the best models used only six MAmix features (MAs for the set of ECs: MA-PTH-experim, MA-BALP-experim, MA-BGLAP-experim, MA-INTP-experim, MA-TRAP-experim, and MA-CTX-I-experim). The model for Fl prediction could achieve an ACC of 0.875 with only 6 input features (MAmix) and a SVM model with linear kernels (equivalent of a linear model). Fd prediction model increased a little with the ACC of 0.893 by adding to MAmix the Metab features (6 original metabolic features) and by using SVM with RBF non-linear kernels. Hw could be predicted with less ACC of 0.75, with only MAmix inputs and SVM with linear kernels. The model for Hd prediction had a good ACC of 0.857 using SVM with RBF kernels and MAmix + Metab inputs. An interesting pattern was observed for the preference of input information for each type of bone parameter: MAmix was used of bone length (Fl, Hl) prediction, and MAmix + Metab for bone diameter (Fd, Hd) prediction. In the case of bone weight (Fw, Hw), the preference of inputs was different.

The last test was the comparison of the best ACC values obtained by different inclusion of EC information into the model inputs: classical one-hot encoding of ECs as binary inputs for each categorical value versus MA encoding as difference between the original metabolic data and the average of this data in specific ECs. In Fig. 2, it could be observed that the classical use of categorical EC values and original metabolic features was unable to provide good results with the current small dataset for bone parameters prediction. Thus, one-hot encoding of ECs was able to classify only Fd with ACC of 0.714 (versus 0.893 with MAs) and Fw with ACC of 0.82 (versus 0.911 with MAs). One-hot encoding models were not able to predict Hd, Hl and Fl (ACC < 0.60). This test demonstrated the need of MAs to be able to classify low and high values of bone parameters. Additional details about the one-hot encoding results can be found in our repository notebooks (Munteanu & Liu, 2019a).

Feature importance

To evaluate the feature importance for the best model for each output, we presented a feature selection method by individual feature removal (see Bones_5_Feature-Importance-Pipelines-LOOCV.ipynb). The more an attribute is applied for making key decisions with decision trees, the higher its relative importance (Tang, Alelyani & Liu, 2014). This importance is calculated explicitly for each attribute in the dataset, allowing attributes to be ranked and compared to each other. The ranking of features was based on the difference between the ACC calculated without a feature (new-ACC) and original ACC calculated with all features (see Table 6). If the difference is smaller (more negative), it means that, if you remove that specific feature, the ACC of the model will suffer more (feature is more important). If the ACC difference is positive, it means that, if you remove that specific feature, you will obtain a better ACC (feature not important or even noise for the model). All details are available in our GitLub repository (FeatImpbyRemoval-LOOCV_SVMlinear_ACC.csv, FeatImpbyRemoval-LOOCV_ACC.csv).

In the case of Fw best model (SVM RBF, ACC = 0.911), the most important features were MA-PTH-Animal and MA-BALP-experim with a decrease of ACC of 0.054 and 0.036. In addition, MA-PTH-experim represents a noise for the model and its removal increased the ACC of the model with 0.035. Thus, the best model for Fw could be considered the one obtained without MA-PTH-experim, with test ACC of 0.946. Therefore, PTH moving average for each type of animal was important for the model but not the PTH moving average in mixed ECs. The feature importance for Fd best model (SVM RBF, ACC = 0.893) was different. The impact of feature removal of moving averages for INTP, PTH, CTX-I, and BALP in mixed ECs was very important, with differences in ACC between 0.125 and 0.232. The original metabolic features were less important. Fl best model (SVM linear, ACC = 0.875) preferred MA-CTX-I-experim and the ACC was improved with 0.036 by removing MA-PTH-experim. Therefore, the best model for Fl became the one without MA-PTH-experim, with a test ACC of 0.911.

The Hw best model (SVM linear, ACC = 0.750) showed preference for MA-CTX-I-experim and considered MA-PTH-experim as noise. By removal of this feature the model ACC was improved with 0.125. Thus, the new ACC of 0.875 represented the best performance for Hw. In contrast, Hd best model (SVM, ACC = 0.857) preferred MA-PTH-experim and MA-BALP-experim, the removal of MA-CTX-I-experim did not change the ACC, but the elimination of TRAP could increase the model performance with 0.018. Thus, the best model for Hd became the one without TRAP, with ACC of 0.875. Hl best model (SVM RBF, ACC = 0.696) preferred the same MA-PTH-experim and MA-BALP-experim, but it could remove MA-BGLAP-experim without any change of ACC.

As a general view for the feature importance, it is noticeable that the moving averages in mixed ECs were generally more important for the majority of the models. MA-PTH-experim was important for Fd, Hd, and Hl but its removal for Fw, Fl and Hw improved the model performance. In general, we observed the important role of PTH and BALP mixed with ECs.

With the best models and ML methods obtained Table 5, one feature models have been tested for all outputs. Therefore, Table 7 presents the ACC values for these models and the differences with the pool feature models. In the case of the ACC improvements, the text has a bold style. First of all, it should be pointed out that the current study is interested into models that include experimental conditions in order to use them to understand the relations with the fetus bones.

Fw models based on only one feature can be more accurate than the best model presented with the pool dataset. Thus, a better model with ACC values of 1.00 (LOOCV) could be obtained with PTH and TRAP or the mixed feature MA-TRAP-experim. A number of 18 features could be use separately to improve the best models with pool features. INTP and the mixed features with INTP are not able to generate any stable model. In consequence, the best model for Fw (ACC = 1.00) become the one that uses SVM (radial kernel) and MA-TRAP-experim.

In the case of Fd classifications, it can be observed that using simple features without experimental conditions such as BGLAP and PTH can be used as unique feature of prediction models with better ACC (maximum of 0.982). In contrast, other original features such as CTX-I, TRAP, INTP and BALP or the mixed features are not able to create better predictions. In the case of Fl, all the experimental features are improving the ACC to a maximum value of 1.00 for MA-PTH-experim, MA-BALP-experim, MA-INTP-experim, and MA-TRAP-experim.

Hd classifications could be improved if PTH, BGLAP, MA-BGLAP-experim, MA-TRAP-experim and MA-BALP-experim are used as single feature with SVM. The best improvements were obtained with PTH (ACC = 0.982) and MA-BGLAP-experim (ACC = 0.946). Hl classifications cannot be improved be using one feature. Hw can be improved to ACC = 1.00 by using only MA-INTP-experim. It implied that the maternal plasma metabolite (INTP) can be used to perfectly match or predict the fetal humerus growth performance (high or low weight) in the experimental conditions.

Nutritional restriction on bone metabolic profiles

Maternal limited nutrition intake during gestation impairs fetal growth and development by exacerbating deleterious outcomes (Abu-Saad & Fraser, 2010; Wu et al., 2004; Wu, Imhoff-Kunsch & Girard, 2012), such as intrauterine growth restriction (IUGR) (Sharma, Shastri & Sharma, 2016), birth defects and low birth weight (Abu-Saad & Fraser, 2010), bone development (Kueper et al., 2015), and low fetal immunity (Macpherson, De Aguero & Ganal-Vonarburg, 2017). Generally speaking, maternal instinct is to mobilize the body potential with the priority to meet the nutritional requirements of the fetal or placental growth in malnutrition condition (Wu, Imhoff-Kunsch & Girard, 2012). In present work, the biomarkers of bone resorption (BGLAP, TRAP, and CTX-1) or bone formation (BALP, and INTP) in serum reduced in goat group under nutrient restriction, indicating maternal malnutrition indeed influences the maternal or fetal bone metabolic turnover (Kovacs, 2001; Wu et al., 2004; Wu, Imhoff-Kunsch & Girard, 2012). According to the previous report, fetal growth and development is most vulnerable in the early-gestation (Wu et al., 2006). However, the influence of maternal nutrition intake also plays the vital role for fetal bone development in middle- and late- gestations. Under the malnutritional condition, low maternal osteocalcin (BGLAP) level represents low efficient maternal bone turnover, which may be decreased during pregnancy (Charles et al., 1985; Naylor et al., 2000).

The results also showed that fetal bone metabolites showed a minor discrepancy compared to high difference of mother in malnutritional condition. The BALP, a bone-specific isoform of ALP on the surface of osteoblasts, an important biomarker reflects the biosynthetic activity of bone-forming cells (Shipman, Holt & Gama, 2013; Tanaka et al., 1997). In addition, BGLAP level in the circulation system is also a symbol for evaluating the bone formation (Booth et al., 2012; Mendes et al., 2019). The serum BALP level decreased under the nutrition restriction, reflecting that malnutrition (protein deficiency, zinc deficiency or malnutrition) negatively regulates the secretion of maternal bone alkaline phosphatase (Sun et al., 2011). Maternal malnutrition might decrease the glucose metabolic process by decreasing osteocalcin level in maternal peripheral blood by regulating the insulin secretion and sensitivity (Ferron & Lacombe, 2014; Kanazawa et al., 2009) in middle-gestation.

In adults, the increasing TRAcP expression might be associated with the osteoporosis (Solberg et al., 2014), cortical bone mineral content and density (Gradin et al., 2012). Interestingly, in present work, we found that the biomarkers of bone turnover, the bone metabolic markers, including NTx, CTx, BALP, BGLAP and TRAP except PTH, decreased along with gestation time (from middle-gestation to late-gestation). The BGLAP concentration increased in maternal serum compared to that of fetus, reflecting enhanced osteoblastic activity. The bone turnover metabolic proteins decreased in late-gestation is due to maternal goats giving the priority to meet the nutritional requirements of fetal skeletal development through decreasing the self- bone turnover or resorption. The mother limits self–bone turnover/resorption rate or further to release more mineral contents of maternal bones to sustain the fetal bone development while undernutrition issues.

Machine learning predictive models for bone metabolic profiles

Machine learning (ML) is a kind of artificial intelligence with the statistical methods for clinical medicine data classification. Until right now, several ML techniques have been widely applied in clinical bone metabolism disease or bone researches with higher accuracy performance for diagnosis of osteopathy. In here, we briefly summarize a few successful applications of machine learning on osteopathy, such as the osteoporosis risk assessment for postmenopausal women (Yoo et al., 2013), pediatric bone age assessment (Halabi et al., 2019), the occurrence of bisphosphonate-related osteonecrosis (Kim et al., 2018), bone surface modifications (Dominguez-Rodrigo, 2019), trabecular bone mechanics (Sohail et al., 2019), or bone marrow associated with relapsed acute leukemia (Li et al., 2019a; Li et al., 2019b). For instance, the variability of sclerostin (a physiological inhibitor of bone formation) varied with sex, 25-OH-D and phosphorus levels in haemodialysis (HD) patients by multivariate regression analysis (Pietrzyk et al., 2019), low serum parathyroid hormone associated with malnutrition-inflammation complex in chronic kidney disease by Chis-square test, linear regression and multivariate logistic regression analysis (Dukkipati et al., 2010). In addition, the abnormalities of biochemical markers of bone turnover, ALP and PTH, are associated with mortality of HD patient across age by Cox proportional hazard models (Lertdumrongluk et al., 2013), and a decrease in intact parathyroid hormone (iPTH) is associated with higher mortality (Villa-Bellosta et al., 2017). There are also some previous works that combined with the machine learning algorithms with data processing analysis of moving average and feature selection to biological phenomena (Deng et al., 2017; Liu et al., 2016a; Liu et al., 2016b). However, none of them reported the association of fetal bone development with maternal bone metabolites by machine learning techniques.

In present work, seven ML methods, including KNN, SVM linear (linear kernels), SVM (RBF kernels), LR, DT, RF and XGB, were used to find the best ML model to predict 6 bone parameters. Then, the feature selection method within individual feature removal was used to evaluate the feature importance for each output (predictive parameter). To some extent, we got some good performance predictive models for matching bone profiles with serum bone metabolic biomarkers. Generally, it can be observed that isolated PTH and its mixed features are able to improve the prediction of bone dimensions. The next important features are BGLAP and its mixed features. Therefore, it could be concluded that some original features such as PTH and BGLAP are the most important for fetal bone development performance predictions and experimental-mixed features could be used to predict bone features in specific experimental conditions. With other words, the original features could predict bone properties but the experimental conditions could offer more details.