Next Article in Journal
On the Intermolecular Interactions in Thiophene-Cored Single-Stacking Junctions
Previous Article in Journal
Adipocyte-Derived Paracrine Factors Regulate the In Vitro Development of Bovine Mammary Epithelial Cells
Previous Article in Special Issue
Methylation Profile of Small Breast Cancer Tumors Evaluated by Modified MS–HRM
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Selectivity of Exhaled Breath Biomarkers of Lung Cancer in Relation to Cancer of Other Localizations

by
Elina M. Gashimova
1,*,
Azamat Z. Temerdashev
1,
Dmitry V. Perunov
2,
Vladimir A. Porkhanov
2,
Igor S. Polyakov
2 and
Ekaterina V. Dmitrieva
1
1
Department of Analytical Chemistry, Kuban State University, Stavropol’skaya St. 149, Krasnodar 350040, Russia
2
Research Institute—Regional Clinical Hospital N° 1 n.a. Prof. S.V. Ochapovsky, 1 May St. 167, Krasnodar 350086, Russia
*
Author to whom correspondence should be addressed.
Int. J. Mol. Sci. 2023, 24(17), 13350; https://doi.org/10.3390/ijms241713350
Submission received: 12 July 2023 / Revised: 23 August 2023 / Accepted: 23 August 2023 / Published: 28 August 2023
(This article belongs to the Special Issue New Diagnostic Tools and Biomarkers in Oncological Diseases)

Abstract

:
Lung cancer is a leading cause of death worldwide, mostly due to diagnostics in the advanced stage. Therefore, the development of a quick, simple, and non-invasive diagnostic tool to identify cancer is essential. However, the creation of a reliable diagnostic tool is possible only in case of selectivity to other diseases, particularly, cancer of other localizations. This paper is devoted to the study of the variability of exhaled breath samples among patients with lung cancer and cancer of other localizations, such as esophageal, breast, colorectal, kidney, stomach, prostate, cervix, and skin. For this, gas chromatography-mass spectrometry (GC-MS) was used. Two classification models were built. The first model separated patients with lung cancer and cancer of other localizations. The second model classified patients with lung, esophageal, breast, colorectal, and kidney cancer. Mann–Whitney U tests and Kruskal–Wallis H tests were applied to identify differences in investigated groups. Discriminant analysis (DA), gradient-boosted decision trees (GBDT), and artificial neural networks (ANN) were applied to create the models. In the case of classifying lung cancer and cancer of other localizations, average sensitivity and specificity were 68% and 69%, respectively. However, the accuracy of classifying groups of patients with lung, esophageal, breast, colorectal, and kidney cancer was poor.

1. Introduction

Cancer is considered as one of the main problems of healthcare. A vast number of various forms and manifestations of cancer are widespread [1,2]. The most optimal cancer treatment outcome is in case of diagnostics in the early stage. The majority of cancers are prone to the lack of symptoms in the early stage, which leads to a high mortality rate due to late diagnostics. Therefore, the issue of developing new non-invasive techniques to serve cancer diagnostics is at hand.
Exhaled breath is being actively explored as a source of cancer biomarkers [3,4,5]. Owing to its simplicity and convenience of sampling as well as non-invasiveness, the interest in exhaled breath is gaining momentum. Various scientists published the results of studies where cancer diagnostic methods based on exhaled breath analysis using different analytical tools were developed [6,7,8,9]. Gas chromatography coupled with mass spectrometry (GC-MS) has taken a dominant position in the field of exhaled breath analysis since it is able to provide the most complete information regarding the sample composition [10,11,12]. Additionally, other analytical methods, including ion mobility spectrometry (IMS) [13], selected ion flow tube mass spectrometry (SIFT-MS) [14,15], proton-transfer-reaction mass spectrometry (PTR-MS) [16,17,18], are widely applied for exhaled breath analysis. Electronic noses (e-noses) can be considered as a separate group of tools for exhaled breath analysis with the advantages of simplicity of construction, mobility of the device, and high speed of analysis [19]. Various e-nose configurations are known to be good candidates as exhaled breath analysis instruments: an e-nose based on metal oxide semiconductor sensors [20,21], a chemoresistive e-nose [22], Cyranose 320 [23], aeonose [24,25], or combined devices consisting of several types of sensors [26,27]. Exhaled breath sampling techniques and analytical methods differ in the studies, which can influence the results. Alveolar, end-tidal, or mixed exhaled breath can be a subject of analysis. The concentration of endogenous VOCs is higher in samples of alveolar air [28]. However, the sampling of alveolar air involves using sophisticated equipment, which restricts the mobility and velocity of sampling. Sampling of end-tidal exhaled air allows us to take more alveolar air, but the ratio of alveolar and dead space air in a sample may differ from one person to another, which contributes to a distortion of the results. Mixed exhaled air is highly diluted by dead space air; therefore, the number of endogenous VOCs is lower. However, this approach is simple, quick, and does not require sophisticated equipment. Obtaining reliable results using mixed exhaled air is possible only in the case of the strict controlling of ambient air as well as conducting the sampling procedure [29].
Attempts to create a diagnostic method using exhaled breath to reveal cancer of various localizations have already been demonstrated [30,31]. The majority of studies are devoted to the identification of an exact disease, as a rule, lung [21,24,26,27] or breast [6,23] cancer. Benzene, 2-propanol, styrene, and pentane were often assigned as metabolites linked with lung cancer development [32]. Breast cancer biomarkers in common significantly differ in various studies [23,33]. However, heptanal was noted as a biomarker in several studies [34,35]. Exhaled breath can be useful for diagnosing cancer of other localizations. For example, a diagnostic model was created in [36], which allowed for the identification of cirrhosis, and primary and secondary liver tumors. Ethane, (E)-2-nonene, acetaldehyde, and acetone contributed the most to the diagnostic accuracy. Ovarian cancer can be diagnosed with 89% accuracy using a diagnostic model based on decanal, nonanal, styrene, 2-butanone, and hexadecane, which were identified in exhaled breath using GC-MS [37]. Cyclohexanone, 2,2-dimethyldecane, dodecane, 4-ethyl-1-octyn-3-ol, ethylaniline, cyclooctylmethanol, trans-2-dodecen-1-ol, 3-hydroxy-2,4,4-trimethylpentyl, 2-methylpropanoate, and 6-t-butyl-2,2,9,9-tetramethyl-3,5-decadien-7-yne were assigned to colorectal cancer biomarkers [38].
Another interesting issue is to find alternative evidence that the tumor affects VOC levels in exhaled breath. It can be achieved by comparing exhaled breath profiles of patients before and after surgery. This approach was demonstrated on 84 patients with lung cancer [39]. Concentrations of 2,5-dimethylfurane, cyclohexanone, propyl cyclohexane, octanal, nonanal, decanal, and 2,2-dymethyldecane differed the most in exhaled breath of patients with lung cancer before and after surgery. An alternative approach is to study the VOC profile extracted by cancer cell lines. The authors [40] compared metabolite profiles of adenocarcinoma, squamous cell carcinoma, large cell carcinoma, small cell carcinoma cell lines, and one normal small airway epithelial cells. Benzaldehyde, 2-ethylhexanol, and 2,4-decadien-1-ol were found as potential lung cancer biomarkers. Comparing profiles of VOCs from different subjects allows one to trace metabolic pathways and obtain additional proof of biomarkers’ origins. Correlations between the results of exhaled breath and fecal samples of patients with gastric cancer were found in study [41].
Considering the highest mortality rate and sophisticated diagnostic procedures applied in clinical practice nowadays, the development of a non-invasive and accurate lung cancer diagnostic tool is the most urgent task [2,8,42]. A conventional approach to developing a diagnostic method is the comparison of healthy volunteers and patients with the studied disease. However, the accuracy of the diagnostic model can be unknown when it comes to other diseases. Therefore, it is essential to consider the accuracy of biomarkers not only in relation to heathy subjects, but the selectivity of potential biomarkers to other diseases. Some studies have considered the development of a diagnostic method able to simultaneously detect several cancer types, for example, an electronic nose was presented in [30] consisting of an array of cross-reactive nanosensors based on organically functionalized gold nanoparticles for diagnosing lung, breast, colorectal, and prostate cancer. VOC profiles of patients with lung cancer, lung cancer and COPD, COPD, and healthy subjects were compared in the study [42].
The paper is focused on the selectivity of exhaled breath analysis using GC-MS to distinguish lung cancer from cancer of other localizations. Breast, esophageal, colorectal, kidney, prostate, cervix, and skin cancer localizations were considered.

2. Results

The study includes two groups of cancer patients: 85 patients with lung cancer and 85 patients with cancer of other organs, including 11 patients with esophageal cancer, 22 patients with mammary cancer, 16 patients with colorectal cancer, 14 patients with kidney cancer, 7 patients with stomach cancer, 6 patients with prostate cancer, 5 patients with cervix cancer, and 4 patients with skin cancer. These samples of exhaled breath were analyzed using GC-MS.
VOCs and their ratios, which were different in lung cancer and other cancer localization groups, were found using a Mann–Whitney U test. Hexane (p = 0.013), acetonitrile (p = 0.036), 1-methylthiopropene (p = 0.010), 1-methylthiopropane (p = 0.006), and dimethyl sulfide (p = 0.021) show a significant difference between groups of patients with lung cancer and cancer of other localizations. Also, several ratios were significantly different between lung cancer and cancer of other localizations (Table 1).
The ratios were used as input values for the creation and validation of the diagnostic model using an artificial neural network (ANN). The accuracy for training, validation, and test datasets was calculated. The efficiency of Broyden–Fletcher–Goldfarb–Shanno (BFGS) and nonlinear conjugate gradient algorithms was compared for the creation of the model. To validate the model, three-fold cross-validation was implemented (Table 2). As seen from Table 2, the BFGS algorithm is better on a test dataset.
The variability of exhaled breath samples of patients with lung, esophageal, breast, colorectal, and kidney cancer was estimated by Kruskal–Wallis H tests. Each pairwise comparison was conducted using Mann-Whitney U tests with a subsequent adjustment of p-value for false discovery rate (FDR).
Some VOCs were found to be different in the studied groups (Table 3). Figure 1 represents a median and interquartile range of parameters with the lowest p-value.
Discriminant analysis (DA) was applied to classify groups of patients with lung, esophageal, breast, colorectal, and kidney cancer. Ratios of VOCs, which were significantly different between the groups, were used as input values. The DA classification matrix is presented in Table 4.
Figure 2 represents a scattering diagram of canonical values for exhaled breath samples depending on cancer localization.
In addition, the gradient-boosted decision trees (GBDT) algorithm was used to separate groups of patients with lung, esophageal, breast, colorectal, and kidney cancer. To validate the model, three-fold cross-validation was used. The performance for training and test datasets was calculated (Table 5).

3. Discussion

The development of a non-invasive cancer diagnostic method is an urgent challenge, which attracts the attention of many researchers worldwide [4,8,10,18,21]. Despite the attempts of many research groups to solve the problem, the breath test for cancer diagnostics has not yet been implemented in clinical practice. It can be explained by the many pitfalls that are often omitted during research. A conventional approach of biomarker identification assumes comparing a group of pathology with a group of healthy volunteers. However, the approach can lead to false-positive results linked to a lack of considering other disorders. An issue of this work was to compare groups of patients with cancer of various localizations. Breast, esophageal, colorectal, kidney, prostate, cervix, and skin cancers were considered. Not only peak areas but also their ratios were considered in terms of the difference between lung cancer and cancer of other localizations. The implementation of this approach was demonstrated earlier [43].
Taking into account difficulties concerning lung cancer diagnostics, the most essential task was to separate samples of exhaled breath of lung cancer patients and patients with cancer of other localizations. For this, a Mann–Whitney U test was applied. Acetonitrile, 1-methylthiopropene, 1-methylthiopropane, and dimethyl sulfide were different between patients with lung cancer and cancer of other localizations. Acetonitrile [44], dimethyl sulfide [45], and 1-methylthiopropene [46] were determined as lung cancer biomarkers earlier. Dimethyl sulfide was also listed as a putative biomarker of esophageal cancer [17]. The ratio of 1-methylthiopropane/acetone was different in groups of lung cancer and healthy volunteers in the previous work [43].
To create a model capable of separating patients with lung cancer and patients with other cancer localizations, ANN was used. ANN is one of the most powerful machine-learning algorithms. It was used in many research works to create diagnostic models [24,47]. Our previous research has shown that the diagnostic model created using ANN is more accurate than random forest, support vector machine, and logistic regression on the same dataset [43]. ANN is the most flexible method capable of revealing complex patterns that may be inaccessible to traditional algorithms. Therefore, ANN was used in this work to create a classification model to separate lung cancer patients from patients with cancer of other localizations. The efficiency of two algorithms: Broyden–Fletcher–Goldfarb–Shanno (BFGH) and nonlinear conjugate gradient was compared to train the ANN. The nonlinear conjugate gradient algorithm is attractive due to the simplicity of the iterations and lower storage requirements [48]. BFGS is one of the most effective quasi-Newton methods [49]. BFGS surpassed the conjugate gradient algorithm: the average sensitivity and specificity on the test dataset were 67% and 69% for BFGS and 56% and 57% for conjugate gradient. Accuracy, which is achieved by comparing lung cancer patients with healthy individuals, is significantly greater in most cases [50,51,52,53]. The accuracy obtained in our research is utterly inadequate for a large-scale screening due to the high number of expected false positives. The study has several limitations: the group of patients with other cancer localization includes uneven distribution of various cancer localizations. Another drawback is the sample size, which is too small to obtain reliable results. However, this study highlights the problem of differentiating various diseases through exhaled breath analysis. Prospectively, the diagnostic models aimed to identify lung cancer may classify patients with cancer of various localizations as lung cancer patients. Therefore, it is essential to compare not only samples of lung cancer patients and healthy volunteers but also consider other pathologies, which can be potentially confused with the disease.
Another task of this work was to evaluate the possibility of classifying patients with various cancer localizations, namely lung, esophageal, breast, colorectal, and kidney cancer, and find the parameters specific to each group. For this, a Kruskal–Wallis H test was used. As can be seen from Figure 2, there are no parameters that can classify each cancer in the separate groups. However, the level of dimethyl sulfide is elevated in the case of lung and esophageal cancer in comparison with other cancer localizations. The majority of ratios containing sulfuric compounds is higher in the case of esophageal and colorectal cancers. Dimethyl sulfide and ratios containing this component were significantly different in groups of lung and esophageal cancer as well as lung and kidney cancer. Levels of the set of VOCs and their ratios were equal for the rest of the cancer localizations (Table 2).
An attempt to classify lung, esophageal, breast, colorectal, and kidney cancer using DA was applied owing to the ability of visualization using a scattering diagram of canonical values. As shown in Figure 2, the exhaled breath samples of patients with cancer of various localizations cannot be separated. Most samples of esophageal, breast, colorectal, and kidney cancer are classified as lung cancer. ANN is one of the most effective machine learning algorithms [38]. It is worth noting that ANN works better when the groups have an equal number of cases. Considering the task of separation of groups with different numbers of observations, one of the most effective machine-learning algorithms is GBDT [45], which was applied to classify the exhaled breath samples of patients with cancer of different localizations. The accuracy of classification on the training data was relatively high for lung and esophageal cancer, but on the test data, it was significantly worse for all cancer localizations. Among the studied cancer types, the model better recognized lung and breast cancer on the test dataset (Table 5). Lung, breast, colorectal, and prostate cancers were classified through exhaled breath analysis using electronic nose based on cross-reactive nanosensors [30]. The groups of patients with lung, breast, and colon cancer were fully separated, but prostate and lung cancer and healthy individual groups were overlapped. Our study also demonstrates a better separation of lung and breast cancer, but accuracy is significantly lower. The main limitation of this part of the study is a small sample size with a lot of comparable groups, each of which contains a low number of samples.
The exhaled breath VOC profiles of lung cancer patients and patients suffering from other lung diseases (e.g., chronic obstructive pulmonary disease (COPD), asthma, pneumonia, pulmonary embolism, benign lung tumors) as well as healthy controls were compared in this study [42]. It was shown that the discrimination of lung cancer and healthy controls was better than between lung cancer and other lung diseases. The classification of 50 breast cancer patients, COPD patients, and healthy volunteers was fulfilled with 100% accuracy on test data using hemoresistive gas sensors and canonical analysis of principal coordinates [54].
The results obtained in this study additionally prove the assumption of obtaining a potentially incorrect diagnosis since the samples of patients with cancer of various localizations are poorly separated. The issue of separating cancer of various localizations is essential for the development of a reliable and accurate cancer diagnostic tool.

4. Materials and Methods

4.1. Materials

Acetone and 1-propanol (99.9%) were obtained from Ecos-1 (Moscow, Russia); o-xylene, p-xylene, m-xylene, and 2-propanol (>95%) were obtained from Vecton (Moscow, Russia); acetonitrile, ethanol, benzene, n-hexane, methanol, toluene (>95%) and octane, nonane, decane, and dodecane (≥99%) were purchased from Sigma-Aldrich (St. Louis, MO, USA).

4.2. Study Participants

The study includes 2 groups of cancer patients: 85 patients with lung cancer and 85 patients with cancer of other organs, including 11 patients with esophageal cancer, 22 patients with mammary cancer, 16 patients with colorectal cancer, 14 patients with kidney cancer, 7 patients with stomach cancer, 6 patients with prostate cancer, 5 patients with cervix cancer, and 4 patients with skin cancer. All patients involved in the study were examined in the State budgetary healthcare institution “Research Institute—Regional Clinical Hospital N° 1 named after Professor S.V. Ochapovsky”. Biopsy results were used for diagnosis confirmation. The samples were collected at the stage of the diagnosis verification before treatment. The data on the participants are summarized in Table 6.

4.3. Exhaled Breath Collection and GC-MS Analysis

Considering the simplicity and mobility of mixed expiratory breath sampling, this approach was chosen to collect the samples. Mixed expiratory breath was collected in 5 L Tedlar (Supelco, Bellefonte, PA, USA) sampling bags. Nitrogen was used for cleaning the bags. The participants provided the samples of exhaled breath in the hospital. Ambient air was used as a blank sample. The patients underwent overnight fasting before sampling. Smokers were not involved in the study if they smoked less than 2.5 h prior to breath collection. Exhaled breath was sampled after a 10 min rest of the participant in a separate sampling room. Patients were asked to breathe, hold their breath for 10 s, and breathe out into the sampling bag. The procedure was repeated until the sampling bag was filled. Sample treatment and chromatographic analysis conditions were optimized and applied earlier [43,55]. A PV-2 aspirator (Chromatec, Yoshkar-Ola, Russia) and Tenax TA (60–80 mesh, Chromatec, Yoshkar-Ola, Russia) sorbent tubes were applied to preconcentrate the samples. The rate and time of aspiration were 200 mL/min and 2.5 min, respectively. A system consisted of a gas chromatograph (Chromatec crystal 5000.2, Chromatec, Yoshkar-Ola, Russia) combined with a quadrupole mass spectrometer with an electron ionization source (Chromatec MSD, Chromatec, Yoshkar-Ola, Russia) and a two-stage thermal desorber TD2 (Chromatec, Yoshkar-Ola, Russia). Separation of analytes was conducted using a Supelco Supel-Q PLOT (30 m × 0.32 mm × 15 μm) column (Bellefonte, PA, USA). Data were acquired using Chromatec Analytic (Chromatec, Yoshkar-Ola, Russia) software and mass spectral library NIST 2017, Version 2.3 (Gatesburg, PA, USA). GC-MS analysis conditions are presented in Table 7.

4.4. Statistical Analysis

The chromatograms of exhaled breath samples were obtained in a full scan mode. Peak areas of individual VOCs were calculated in extracted ion chromatogram (EIC) mode. The influence of ambient air was eliminated by subtraction of room air peak area values from the exhaled breath ones. Negative results of subtraction were set to zero. VOCs with peak area values exceeding room air levels by at least 50% were selected for statistical analysis. Another criterion for selection was occurring the VOC in more than 50% of samples. The ratios of the compound peak areas to the main ones (present in more than 88% of the samples) as well as ratios of the main VOCs were considered for statistical analysis. The description of ratio calculations was presented earlier in detail [43].
StatSoft STATISTICA (version 10) was applied for statistical analysis and building the diagnostic model. At the first stage, the normality of data distribution was checked applying Kolmogorov–Smirnov test. The distribution was not normal. Therefore, a Mann–Whitney U test (p = 0.05) was used to select the VOCs and their ratios, which are different in groups of lung cancer patients and patients with cancer of other organs. A Kruskal–Wallis H test (p = 0.05) was used to assess the mean differences between groups of patients with lung, esophageal, breast, colorectal, and kidney cancer. Each pairwise comparison was conducted using a Mann–Whitney U test. A Type I error from multiple hypothesis testing was addressed with False Discovery Rate (FDR) correction using the Benjamini–Hochberg adjusted p-value cutoff of ≤0.05.
The diagnostic model for selecting lung cancer patients from patients with other cancer localizations was built by applying a multilayer perceptron artificial neural network (ANN). A recurrent feedforward ANN with fully connected one hidden layer was created. Model training was fulfilled using 2 algorithms: Broyden–Fletcher–Goldfarb–Shanno and nonlinear conjugate gradient. The model has the following structure: the input layer consists of 15 neurons corresponding to VOC, which were significantly different between the studied groups (Table 1), a hidden layer, including 7 neurons, and the output layer, including 2 neurons, which corresponded to the group of lung cancer or cancer of other organs. The hidden layer activation function was identity, output layer—SoftMax. The data were divided into 3 datasets: training (60%), validation (10%), and test (30%). To provide reliable results, three-fold cross-validation was used.
To create the model allowing classification of patients with different cancer localizations, the gradient-boosted decision trees (GBDT) algorithm was applied. The data were divided into 2 datasets: training (70%) and test (30%). The reliability of the results was provided by three-fold cross-validation.

5. Conclusions

In the present research, we shed a light on the problem of classifying patients with various diseases using exhaled breath analysis. The classical approach supposes the comparison of patients with certain diseases with healthy persons, but putative biomarkers can indicate not only the investigated diseases but other pathologies. To avoid incorrect classification, the other pathologies must be considered before the implementation of exhaled breath tests in clinical practice. Exhaled breath VOC profiles of patients with cancer of various localizations were considered in this work. The results obtained prove the assumption about overlapping the VOC profiles of patients with various cancer localizations. Further research is required to determine biomarkers specific to each cancer localization type for providing accurate diagnosis.

Author Contributions

Conceptualization, I.S.P., V.A.P. and A.Z.T.; methodology E.M.G. and A.Z.T.; software, E.M.G.; validation, E.M.G., E.V.D. and D.V.P.; formal analysis, E.M.G. and A.Z.T.; investigation E.M.G.; resources A.Z.T.; data curation, E.M.G.; writing—original draft preparation E.M.G.; writing—review and editing, A.Z.T., E.M.G., E.V.D. and D.V.P.; visualization, E.M.G. and E.V.D.; supervision A.Z.T.; project administration A.Z.T.; funding acquisition A.Z.T. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Russian Science Foundation and Kuban Science Foundation (project no. 22-13-20018) using the scientific equipment of the Center for Environmental Analysis at the Kuban State University.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by State budgetary healthcare institution “Research Institute—Regional Clinical Hospital N° 1 named after Professor S.V. Ochapovsky” (protocol N° 122, 19 December 2019).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Chhikara, K.P.; Parang, K. Global cancer statistics 2022: The trends projection analysis. Chem. Biol. Lett. 2023, 10, 451. [Google Scholar]
  2. Xia, C.; Dong, X.; Li, H.; Cao, M.; Sun, D.; He, S.; Yang, F.; Yan, X.; Zhang, S.; Li, N.; et al. Cancer statistics in China and United States, 2022: Profiles, trends, and determinants. Chin. Med. J. 2022, 135, 584–590. [Google Scholar] [PubMed]
  3. Ma, P.; Li, J.; Chen, Y.; Montano, B.A.Z.; Luo, H.; Zhang, D.; Zheng, H.; Liu, Y.; Lin, H.; Zhu, W.; et al. Non-invasive exhaled breath diagnostic and monitoring technologies. Microw. Opt. Technol. Lett. 2021, 65, 1475–1488. [Google Scholar]
  4. Peled, N.; Fuchs, V.; Kestenbaum, E.H.; Oscar, E.; Bitran, R. An update on the use of exhaled breath analysis for the early detection of lung cancer. Lung Cancer Targets Ther. 2021, 12, 81–92. [Google Scholar]
  5. Sharma, A.; Kumar, R.; Varadwaj, P. Smelling the disease: Diagnostic potential of breath analysis. Mol. Diagn. Ther. 2023, 27, 321–347. [Google Scholar]
  6. Leemans, M.; Bauer, P.; Cuzuel, V.; Audureau, E.; Fromantin, I. Volatile organic compounds analysis as a potential novel screening tool for breast cancer: A systematic review. Lung Cancer Targets Ther. 2021, 12, 81–92. [Google Scholar]
  7. Haddad, G.; Schouwenburg, S.; Altesha, A.; Xu, W.; Liu, G. Using breath analysis as a screening tool to detect gastric cancer: A systematic review. J. Breath Res. 2021, 15, 016013. [Google Scholar]
  8. Binson, V.A.; Subramoniam, M. Exhaled Breath Volatile Organic Compound Analysis for the Detection of Lung Cancer—A Systematic Review. J. Biomim. Biomater. Biomed. Eng. 2022, 56, 17–35. [Google Scholar]
  9. Le, T.; Priefer, R. Detection technologies of volatile organic compounds in the breath for cancer diagnoses. Talanta 2023, 265, 124767. [Google Scholar]
  10. Zou, Y.; Hu, Y.; Jiang, Z.; Chen, Y.; Zhou, Y.; Wang, Z.; Wang, Y.; Jiang, G.; Tan, Z.; Hua, F. Exhaled metabolic markers and relevant dysregulated pathways of lung cancer: A pilot study. Ann. Med. 2022, 54, 790–802. [Google Scholar]
  11. Xiang, C.; Yang, H.; Zhao, Z.; Deng, F.; Lv, Y.; Yang, Y.; Duan, Y.; Li, W.; Hu, B. Volatolomics analysis of exhaled breath and gastric-endoluminal gas for distinguishing early upper gastrointestinal cancer from benign. J. Breath Res. 2023, 17, 036004. [Google Scholar] [CrossRef]
  12. Sukaram, T.; Tansawat, R.; Apiparakoon, T.; Tiyarattanachai, T.; Marukatat, S.; Rerknimitr, R.; Chaiteerakij, R. Exhaled volatile organic compounds for diagnosis of hepatocellular carcinoma. Sci. Rep. 2022, 12, 5326. [Google Scholar] [CrossRef] [PubMed]
  13. Lamote, K.; Vynck, M.; Cleemput, J.V.; Thas, O.; Nackaerts, K.; Meerbeeck, J.P. Detection of malignant pleural mesothelioma in exhaled breath by multicapillary column/ion mobility spectrometry (MCC/IMS). J. Breath Res. 2016, 10, 046001. [Google Scholar] [CrossRef]
  14. Markar, S.R.; Chin, S.T.; Romano, A.; Wiggins, T.; Antonowicz, S.; Paraskeva, P.; Ziprin, P.; Darzi, A.; Hanna, G.B. Breath volatile organic compound profiling of colorectal cancer using selected ion flow-tube mass spectrometry. Ann. Surg. 2019, 269, 903–910. [Google Scholar] [CrossRef]
  15. Chandran, D.; Ooi, E.H.; Watson, D.I.; Kholmurodova, F.; Jaenisch, S.; Yazbeck, R. The use of selected ion flow tube-mass spectrometry technology to identify breath volatile organic compounds for the detection of head and neck squamous cell carcinoma: A pilot study. Medicina 2019, 55, 306. [Google Scholar] [CrossRef]
  16. Zhou, W.; Huang, C.; Zou, X.; Lu, Y.; Shen, C.; Ding, X.; Wang, H.; Jiang, H.; Chu, Y. Exhaled breath online measurement for cervical cancer patients and healthy subjects by proton transfer reaction mass spectrometry. Anal. Bioanal. Chem. 2017, 409, 5603–5612. [Google Scholar] [CrossRef]
  17. Zou, X.; Zhou, W.; Lu, Y.; Shen, C.; Hu, Z.; Wang, H.; Jiang, H.; Chu, Y. Exhaled gases online measurements for esophageal cancer patients and healthy people by proton transfer reaction mass spectrometry. J. Gastroenterol. Hepatol. 2016, 31, 1837–1843. [Google Scholar] [CrossRef]
  18. Sun, Y.; Chen, Y.; Sun, C.; Liu, H.; Wang, Y.; Jiang, X. Analysis of volatile organic compounds from patients and cell lines for the validation of lung cancer biomarkers by proton-transfer-reaction mass spectrometry. Anal. Methods 2019, 11, 3188–3197. [Google Scholar] [CrossRef]
  19. Scheepers, M.H.M.C.; Al-Difaie, Z.; Brandts, L.; Peeters, A.; Grinsven, B.; Bouvy, N.D. Diagnostic performance of electronic noses in cancer diagnoses using exhaled breath: A systematic review and meta-analysis. JAMA Netw. Open 2022, 5, e2219372. [Google Scholar] [CrossRef]
  20. Raspagliesi, F.; Bogani, G.; Benedetti, S.; Grassi, S.; Ferla, S.; Buratti, S. Detection of ovarian cancer through exhaled breath by electronic nose: A prospective study. Cancers 2020, 12, 2408. [Google Scholar] [CrossRef]
  21. Binson, V.A.; Subramoniam, M.; Mathew, L. Noninvasive detection of COPD and lung cancer through breath analysis using MOS sensor array based e-nose. Expert Rev. Mol. Diagn. 2021, 11, 1223–1233. [Google Scholar]
  22. Freddi, S.; Marzuoli, C.; Pagliara, S.; Drera, G.; Sangaletti, L. Targeting biomarkers in the gas phase through a chemoresistive electronic nose based on graphene functionalized with metal phthalocyanines. RSC Adv. 2023, 13, 251–263. [Google Scholar] [CrossRef] [PubMed]
  23. Leon-Martinez, L.D.; Rodriguez-Aguilar, M.; Gorocica-Rosete, P.; Dominguez-Reyes, C.A.; Martinez-Bustos, V.; Tenorio-Torres, J.A.; Ornelas-Rebolledo, O.; Cruz-Ramos, J.A.; Balderas-Segura, B.; Flores-Ramirez, R. Identification of profiles of volatile organic compounds in exhaled breath by means of an electronic nose as a proposal for a screening method for breast cancer: A case-control study. J. Breath Res. 2020, 14, 0406009. [Google Scholar]
  24. Kort, S.; Brusse-Keizer, M.; Gerritsen, J.W.; Schouwink, H.; Citgez, E.; Jongh, F.; Maten, J.; Samii, S.; Bogart, M.; Palen, J. Improving lung cancer diagnosis by combining exhaled-breath data and clinical parameters. ERJ Open Res. 2023, 6, 00221–02019. [Google Scholar] [CrossRef]
  25. Waltman, C.G.; Marcelissen, T.A.T.; Roermund, J.G.H. Exhaled-breath testing for prostate cancer based on volatile organic compound profiling using an electronic nose device (Aeonose™): A preliminary report. Eur. Urol. Focus 2020, 6, 1220–1225. [Google Scholar] [CrossRef] [PubMed]
  26. Li, W.; Liu, H.; Xie, D.; He, Z.; Pi, X. Lung cancer screening based on type-different sensor arrays. Sci. Rep. 2017, 7, 1969. [Google Scholar] [CrossRef]
  27. Liu, B.; Yu, H.; Zeng, X.; Zhang, D.; Gong, J.; Tian, L.; Qian, J.; Zhao, L.; Zhang, S.; Liu, R. Lung cancer detection via breath by electronic nose enhanced with a sparse group feature selection approach. Sens. Actuators B Chem. 2021, 339, 129896. [Google Scholar] [CrossRef]
  28. Miekisch, W.; Kischkel, S.; Sawacki, A.; Liebau, T.; Mieth, M.; Schubert, J.K. Impact of sampling procedures on the results of breath analysis. J. Breath Res. 2008, 2, 026007. [Google Scholar] [CrossRef]
  29. Di Gilio, A.; Palmisani, J.; Ventrella, G.; Facchini, L.; Catino, A.; Varesano, N.; Pizzutilo, P.; Galetta, D.; Borelli, M.; Barbieri, P.; et al. Breath Analysis: Comparison among Methodological Approaches for Breath Sampling. Molecules 2020, 25, 5823. [Google Scholar] [CrossRef]
  30. Peng, G.; Hakim, M.; Broza, Y.Y.; Billan, S.; Abdah-Bortnyak, R.; Kuten, A.; Tisch, U.; Haick, H. Detection of lung, breast, colorectal, and prostate cancers from exhaled breath using a single array of nanosensors. Br. J. Cancer 2010, 103, 542–551. [Google Scholar] [CrossRef]
  31. Chung, J.; Akter, S.; Han, S.; Shin, Y.; Choi, T.G.; Kang, I.; Kim, S.S. Diagnosis by volatile organic compounds in exhaled breath from patients with gastric and colorectal cancers. Int. J. Mol. Sci. 2022, 24, 129. [Google Scholar] [CrossRef] [PubMed]
  32. Hua, Q.; Zhu, Y.; Liu, H. Detection of volatile organic compounds in exhaled breath to screen lung cancer: A systematic review. Future Oncol. 2018, 14, 1647–1662. [Google Scholar] [CrossRef] [PubMed]
  33. Wang, C.; Sun, B.; Guo, L.; Wang, X.; Ke, C.; Liu, S.; Zhao, W.; Luo, S.; Guo, Z.; Zhang, Y.; et al. Volatile organic metabolites identify patients with breast cancer, cyclomastopathy and mammary gland fibroma. Sci. Rep. 2014, 4, 45383. [Google Scholar] [CrossRef]
  34. Li, J.; Peng, Y.; Liu, Y.; Li, W.; Jin, Y.; Tang, Z.; Duan, Y. Investigation of potential breath biomarkers for the early diagnosis of breast cancer using gas chromatography–mass spectrometry. Clin. Chim. Acta 2014, 436, 59–67. [Google Scholar] [CrossRef] [PubMed]
  35. Phillips, M.; Cataneo, R.N.; Ditkoff, B.A.; Fisher, P.; Greenberg, J.; Gunawardena, R.; Kwon, C.S.; Tietje, O.; Wong, C. Prediction of breast cancer using volatile biomarkers in the breath. Breast Cancer Res Treat. 2006, 99, 19–21. [Google Scholar] [CrossRef]
  36. Miller-Atkins, G.; Acevedo-Moreno, L.; Grove, D.; Dweik, R.A.; Tonelli, A.R.; Brown, J.M.; Allende, D.S.; Aucejo, F.; Rotroff, D.M. Breath metabolomics provides an accurate and noninvasive approach for screening cirrhosis, primary, and secondary liver tumors. Hepatol. Commun. 2020, 4, 1041–1055. [Google Scholar] [CrossRef]
  37. Amal, H.; Shi, D.Y.; Ionescu, R.; Zhang, W.; Hua, Q.L.; Pan, Y.Y.; Tao, L.; Liu, H.; Haick, H. Assessment of ovarian cancer conditions from exhaled breath. Int. J. Cancer 2015, 136, E614–E622. [Google Scholar] [CrossRef]
  38. Wang, C.; Ke, C.; Wang, X.; Chi, C.; Guo, L.; Luo, S.; Guo, Z.; Xu, G.; Zhang, F.; Li, E. Noninvasive detection of colorectal cancer by analysis of exhaled breath. Anal. Bioanal. Chem. 2014, 406, 4757–4763. [Google Scholar] [CrossRef]
  39. Wang, P.; Huang, Q.; Meng, S.; Mu, T.; Liu, Z.; He, M.; Li, Q.; Zhao, S.; Wang, S.; Qiu, M. Identification of lung cancer breath biomarkers based on perioperative breathomics testing: A prospective observational study. EClinicalMedicine 2022, 47, 101384. [Google Scholar] [CrossRef]
  40. Jia, Z.; Zhang, H.; Ong, C.N.; Patra, A.; Lu, Y.; Lim, C.T.; Venkatesan, T. Detection of lung cancer: Concomitant volatile organic compounds and metabolomic profiling of six cancer cell lines of different histological origins. ACS Omega 2018, 3, 5131–5140. [Google Scholar] [CrossRef]
  41. Bhandari, M.P.; Polaka, I.; Vangravs, R.; Mezmale, L.; Veliks, V.; Kirshners, A.; Mochalski, P.; Dias-Neto, E.; Leja, M. Volatile markers for cancer in exhaled breath—Could they be the signature of the gut microbiota? Molecules 2023, 28, 3488. [Google Scholar] [CrossRef]
  42. Tirzite, M.; Bukovskis, M.; Strazda, G.; Jurka, N.; Taivans, I. Detection of lung cancer in exhaled breath with an electronic nose using support vector machine analysis. J. Breath Res. 2017, 11, 036009. [Google Scholar] [CrossRef] [PubMed]
  43. Gashimova, E.M.; Temerdashev, A.Z.; Porkhanov, V.A.; Polyakov, I.S.; Perunov, D.V.; Dmitrieva, E.V. Non-invasive exhaled breath and skin analysis to diagnose lung cancer: Study of age effect on diagnostic accuracy. ACS Omega 2022, 7, 42613–42628. [Google Scholar] [CrossRef] [PubMed]
  44. Sakumura, Y.; Koyama, Y.; Tokutake, H.; Hida, T.; Sato, K.; Itoh, T.; Akamatsu, T.; Shin, W. Diagnosis by volatile organic compounds in exhaled breath from lung cancer patients using support vector machine algorithm. Sensors 2017, 17, 287. [Google Scholar] [CrossRef] [PubMed]
  45. Rudnicka, J.; Walczak, M.; Kowalkowski, T.; Jezierski, T.; Buszewski, B. Determination of volatile organic compounds as potential markers of lung cancer by gas chromatography-mass spectrometry versus trained dogs. Sens. Actuators B 2014, 202, 615–621. [Google Scholar] [CrossRef]
  46. Temerdashev, A.Z.; Gashimova, E.M.; Porkhanov, V.A.; Polyakov, I.S.; Perunov, D.V.; Dmitrieva, E.V. Non-invasive lung cancer diagnostics through metabolites in exhaled breath: Influence of the disease variability and comorbidities. Metabolites 2023, 13, 203. [Google Scholar] [CrossRef]
  47. Wang, M.; Sheng, J.; Wu, Q.; Zou, Y.; Hu, Y.; Ying, K.; Wan, H.; Wang, P. Confounding effect of benign pulmonary diseases in selecting volatile organic compounds as markers of lung cancer. J. Breath Res. 2018, 12, 046014. [Google Scholar] [CrossRef]
  48. Jiang, X.; Zhu, Y.; Jian, J. Two efficient nonlinear conjugate gradient methods with restart procedures and their applications in image restoration. Nonlinear Dyn. 2023, 111, 5469–5498. [Google Scholar] [CrossRef]
  49. Ibrahim, M.A.H.; Mama, M.; Leong, W.J. BFGS method: A new search direction. Sains Malays. 2014, 43, 1591–1597. [Google Scholar]
  50. Wang, Y.; Hu, Y.; Wang, D.; Yu, K.; Wang, L.; Zou, Y.; Zhao, C.; Zhang, X.; Wang, P.; Ying, K. The analysis of volatile organic compounds biomarkers for lung cancer in exhaled breath, tissues and cell lines. Cancer Biomark. 2012, 11, 129–137. [Google Scholar] [CrossRef]
  51. Rudnickaa, J.; Kowalkowskia, T.; Buszewski, B. Searching for selected VOCs in human breath samples as potential markers of lung cancer. Lung Cancer 2019, 135, 123–129. [Google Scholar] [CrossRef] [PubMed]
  52. Meng, S.; Li, Q.; Zhou, Z.; Li, H.; Liu, X.; Pan, S.; Li, M.; Wang, L.; Guo, Y.; Qiu, M.; et al. Assessment of an exhaled breath test using high-pressure photon ionization time-of-flight mass spectrometry to detect lung cancer. JAMA Netw. Open 2021, 4, e213486. [Google Scholar] [CrossRef] [PubMed]
  53. Zou, Y.; Wang, Y.; Jiang, Z.; Zhou, Y.; Chen, Y.; Hu, Y.; Jiang, G.; Xie, D. Breath profile as composite biomarkers for lung cancer diagnosis. Lung Cancer 2021, 154, 206–213. [Google Scholar] [CrossRef] [PubMed]
  54. Rodriguez-Aguilar, M.; Leon-Martinez, L.D.; Gorocica-Rosete, P.; Perez-Padilla, R.; Dominguez-Reyes, C.A.; Tenorio-Torres, J.A.; Ornelas-Rebolledo, O.; Mehta, G.; Zamora-Mendoza, B.N.; Flores-Ramirez, R. Application of chemoresistive gas sensors and chemometric analysis to differentiate the fingerprints of global volatile organic compounds from diseases. Preliminary results of COPD, lung cancer and breast cancer. Clin. Chim. Acta 2021, 518, 83–92. [Google Scholar] [CrossRef]
  55. Gashimova, E.M.; Temerdashev, A.Z.; Porkhanov, V.A.; Polyakov, I.S.; Perunov, D.V.; Azaryan, A.A.; Dmitrieva, E.V. Evaluation of the possibility of volatile organic compounds determination in exhaled air by gas chromatography for the noninvasive diagnostics of lung cancer. J. Anal. Chem. 2019, 73, 472–479. [Google Scholar] [CrossRef]
Figure 1. Median and interquartile range of peak areas and their ratios with the highest difference between groups of lung, esophageal, breast, colorectal, and kidney cancer.
Figure 1. Median and interquartile range of peak areas and their ratios with the highest difference between groups of lung, esophageal, breast, colorectal, and kidney cancer.
Ijms 24 13350 g001
Figure 2. Scattering diagram of canonical values for exhaled breath samples depending on cancer localization.
Figure 2. Scattering diagram of canonical values for exhaled breath samples depending on cancer localization.
Ijms 24 13350 g002
Table 1. VOCs and their ratios, which have a significant difference between groups of patients with lung cancer and cancer of other localizations.
Table 1. VOCs and their ratios, which have a significant difference between groups of patients with lung cancer and cancer of other localizations.
VOC or VOC Ratiop-Value
Acetonitrile0.035681
1-Methylthiopropene0.010414
1-Methylthiopropane0.005953
Dimethyl sulfide0.021984
2-Butanone/1-methylthiopropene0.000567
2-Butanone/1-methylthiopropane0.002528
2-Butanone/dimethyl sulfide0.004545
Allyl methyl sulfide/dimethyl disulfide0.033031
Allyl methyl sulfide/acetone0.045749
2-Pentanone/1-methylthiopropene0.008109
2-Pentanone/1-methylthiopropane0.024048
2-Pentanone/dimethyl sulfide0.030791
Dimethyl disulfide/1-methylthiopropene0.044414
Dimethyl disulfide/dimethyl sulfide0.031523
1-Methylthiopropene/isoprene0.047466
1-Methylthiopropene/acetone0.012438
1-Methylthiopropane/isoprene0.041529
1-Methylthiopropane/acetone0.025859
Dimethyl sulfide/acetone0.049961
Table 2. Performance of ANN models.
Table 2. Performance of ANN models.
AlgorithmDatasetTraining DatasetValidation DatasetTest Dataset
SensitivitySpecificitySensitivitySpecificitySensitivitySpecificity
Broyden–Fletcher–Goldfarb–Shanno1626289606668
27756701005769
3808770507971
Nonlinear conjugate gradient1666389805543
28148701007159
3437730754179
Table 3. Statistically significantly different parameters in groups of patients with lung, esophageal, breast, colorectal, and kidney cancer (bold p-values were statistically significant).
Table 3. Statistically significantly different parameters in groups of patients with lung, esophageal, breast, colorectal, and kidney cancer (bold p-values were statistically significant).
VOC or VOC RatioKruskal–Wallis H Test p-ValueMann–Whitney U Test p-Value
Lung vs. EsophagealLung vs. BreastLung vs. ColorectalLung vs. KidneyEsophageal vs. BreastEsophageal vs. ColorectalEsophageal vs. KidneyBreast vs. ColorectalBreast vs. KidneyColorectal vs. Kidney
Dimethyl sulfide0.00560.00760.07430.86340.00400.17520.13230.97820.46880.13130.1095
2-butanone/1-methylthiopropene0.00370.01430.04800.04030.00580.59290.60440.86950.98820.64960.7870
2-butanone/1-methylthiopropane0.00930.00890.45930.10860.01010.09290.17480.44340.35170.22990.4669
2-butanone/dimethyl sulfide0.00570.01240.18490.06830.00540.25190.30011.00000.83600.22990.1834
2-pentanone/dimethyl sulfide0.00950.00190.05690.32650.11840.13140.03180.08460.50590.93530.5194
Dimethyl sulfide/acetone0.00570.00790.07430.39480.00490.25990.09830.42730.46880.32230.1190
Dimethyl disulfide/dimethyl sulfide0.00770.66200.74900.01580.00140.63310.24620.08460.08910.03630.5194
1-methylthiopropene/acetone0.03210.06740.06710.01790.11150.64680.94100.33800.56430.85840.4928
Dimethyl disulfide/1-methylthiopropene0.02720.93580.30350.02190.00570.71680.28870.05890.39940.22370.9503
Table 4. Classification matrix of DA model.
Table 4. Classification matrix of DA model.
CancerSample Classification Results
LungEsophagealBreastColorectalKidneyTotalPercentage of Correct Classification, %
Lung8131008595
Esophageal91010119
Breast182200229
Colorectal1200401625
Kidney120011147
Table 5. Classification matrix of GBDT model.
Table 5. Classification matrix of GBDT model.
Sample Classification Results
DataCancerDatasetLungEsophagealBreastColorectalKidneyTotalPercentage of Correct Classification, %
TrainingLung14631605682
Esophageal34001850
Breast612501414
Colorectal301701164
Kidney42111911
TestLung2031502969
Esophageal12000367
Breast5201080
Colorectal31010520
Kidney2003050
TrainingLung24075235770
Esophageal22003729
Breast436021540
Colorectal421121010
Kidney404021010
TestLung1656012857
Esophageal2000240
Breast31300743
Colorectal3030060
Kidney30001425
TrainingLung34615225682
Esophageal070007100
Breast00150015100
Colorectal00011011100
Kidney000099100
TestLung1806322962
Esophageal1021040
Breast20401757
Colorectal1020250
Kidney2120050
Table 6. Information on the participants.
Table 6. Information on the participants.
Group of PatientsNumberMale/FemaleAge (Median, Range)Smokers
Lung cancer8563/2266, 30–7932
Esophageal cancer1110/161, 45–745
Breast cancer222260, 30–731
Colorectal cancer166/1066, 35–851
Kidney cancer148/663, 49–816
Stomach cancer75/264, 54–792
Prostate cancer6669, 58–762
Cervix cancer4457, 42–611
Skin cancer42/264, 59–671
Table 7. GC-MS analysis conditions.
Table 7. GC-MS analysis conditions.
EquipmentParameterValue
Thermal desorberCarrier gasHelium
Carries gas flow rate (desorption from the sorption tube), mL/min30
Desorption temperature, °C250
Initial trap temperature, °C−10
Final trap temperature, °C250
Carrier gas flow rate (desorption from the trap), mL/min50
Desorption time, min5
Speed of the trap heating, °C/min2000
GC-MSCarrier gasHelium
Injector temperature, °C250
Split ratio1:10
Ion source temperature, °C200
Transfer line temperature, °C250
Scan range, amu29–250
Electron impact ionization, eV70
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Gashimova, E.M.; Temerdashev, A.Z.; Perunov, D.V.; Porkhanov, V.A.; Polyakov, I.S.; Dmitrieva, E.V. Selectivity of Exhaled Breath Biomarkers of Lung Cancer in Relation to Cancer of Other Localizations. Int. J. Mol. Sci. 2023, 24, 13350. https://doi.org/10.3390/ijms241713350

AMA Style

Gashimova EM, Temerdashev AZ, Perunov DV, Porkhanov VA, Polyakov IS, Dmitrieva EV. Selectivity of Exhaled Breath Biomarkers of Lung Cancer in Relation to Cancer of Other Localizations. International Journal of Molecular Sciences. 2023; 24(17):13350. https://doi.org/10.3390/ijms241713350

Chicago/Turabian Style

Gashimova, Elina M., Azamat Z. Temerdashev, Dmitry V. Perunov, Vladimir A. Porkhanov, Igor S. Polyakov, and Ekaterina V. Dmitrieva. 2023. "Selectivity of Exhaled Breath Biomarkers of Lung Cancer in Relation to Cancer of Other Localizations" International Journal of Molecular Sciences 24, no. 17: 13350. https://doi.org/10.3390/ijms241713350

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop