Implementation of Machine Learning on Human Frequency-Following Responses: A Tutorial

 

 Buy Article Permissions and Reprints

Abstract

The frequency-following response (FFR) provides enriched information on how acoustic stimuli are processed in the human brain. Based on recent studies, machine learning techniques have demonstrated great utility in modeling human FFRs. This tutorial focuses on the fundamental principles, algorithmic designs, and custom implementations of several supervised models (linear regression, logistic regression, k-nearest neighbors, support vector machines) and an unsupervised model (k-means clustering). Other useful machine learning tools (Markov chains, dimensionality reduction, principal components analysis, nonnegative matrix factorization, and neural networks) are discussed as well. Each model's applicability and its pros and cons are explained. The choice of a suitable model is highly dependent on the research question, FFR recordings, target variables, extracted features, and their data types. To promote understanding, an example project implemented in Python is provided, which demonstrates practical usage of several of the discussed models on a sample dataset of six FFR features and a target response label.

Publication History

Article published online:
26 October 2022

© 2022. Thieme. All rights reserved.

Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA

• References

• 1 Harris CR, Millman KJ, van der Walt SJ. et al. Array programming with NumPy. Nature 2020; 585 (7825): 357-362
  CrossrefPubMedGoogle Scholar
• 2 The Pandas Development Team. Pandas-Dev/Pandas: Pandas. Zenodo. 2021 https://doi.org/10.5281/zenodo.3509134
  CrossrefPubMedGoogle Scholar
• 3 Hunter JD. Matplotlib: a 2D graphics environment. Comput Sci Eng 2007; 9: 90-95
  CrossrefPubMedGoogle Scholar
• 4 Pedregosa F, Varoquaux G, Gramfort A. et al. Scikit-learn: machine learning in python. J Mach Learn Res 2011; 12: 2825-2830
  PubMedGoogle Scholar
• 5 Paszke A, Gross S, Massa F. et al. PyTorch: an imperative style, high-performance deep learning library. ArXiv191201703 Cs Stat.. Published online December 3, 2019. Accessed December 29, 2021 at: http://arxiv.org/abs/1912.01703
  PubMedGoogle Scholar
• 6 Waskom ML. Seaborn: statistical data visualization. J Open Source Softw 2021; 6: 3021
  CrossrefPubMedGoogle Scholar
• 7 Skoe E, Kraus N. Auditory brain stem response to complex sounds: a tutorial. Ear Hear 2010; 31 (03) 302-324
  CrossrefPubMedGoogle Scholar
• 8 Galbraith GC, Olfman DM, Huffman TM. Selective attention affects human brain stem frequency-following response. Neuroreport 2003; 14 (05) 735-738
  CrossrefPubMedGoogle Scholar
• 9 Anderson S, White-Schwoch T, Choi HJ, Kraus N. Partial maintenance of auditory-based cognitive training benefits in older adults. Neuropsychologia 2014; 62: 286-296
  CrossrefPubMedGoogle Scholar
• 10 Hornickel J, Skoe E, Nicol T, Zecker S, Kraus N. Subcortical differentiation of stop consonants relates to reading and speech-in-noise perception. Proc Natl Acad Sci U S A 2009; 106 (31) 13022-13027
  CrossrefPubMedGoogle Scholar
• 11 White-Schwoch T, Kraus N. Physiologic discrimination of stop consonants relates to phonological skills in pre-readers: a biomarker for subsequent reading ability?(†). Front Hum Neurosci 2013; 7: 899
  CrossrefPubMedGoogle Scholar
• 12 Aiken SJ, Picton TW. Envelope following responses to natural vowels. Audiol Neurotol 2006; 11 (04) 213-232
  CrossrefPubMedGoogle Scholar
• 13 Krishnan A. Human frequency-following responses to two-tone approximations of steady-state vowels. Audiol Neurotol 1999; 4 (02) 95-103
  CrossrefPubMedGoogle Scholar
• 14 Stump K, Jeng FC. Frequency-following responses elicited by a consonant-vowel with an intonation. Proc Meet Acoust 2018; 35: 050001
  CrossrefPubMedGoogle Scholar
• 15 Krishnan A, Xu Y, Gandour JT, Cariani PA. Human frequency-following response: representation of pitch contours in Chinese tones. Hear Res 2004; 189 (1-2): 1-12
  CrossrefPubMedGoogle Scholar
• 16 Aiken SJ, Picton TW. Envelope and spectral frequency-following responses to vowel sounds. Hear Res 2008; 245 (1-2): 35-47
  CrossrefPubMedGoogle Scholar
• 17 Jeng FC, Chung HK, Lin CD, Dickman B, Hu J. Exponential modeling of human frequency-following responses to voice pitch. Int J Audiol 2011; 50 (09) 582-593
  CrossrefPubMedGoogle Scholar
• 18 Krishnan A. Human frequency-following responses: representation of steady-state synthetic vowels. Hear Res 2002; 166 (1-2): 192-201
  CrossrefPubMedGoogle Scholar
• 19 Krizman J, Kraus N. Analyzing the FFR: a tutorial for decoding the richness of auditory function. Hear Res 2019; 382: 107779
  CrossrefPubMedGoogle Scholar
• 20 Xie Z, Reetzke R, Chandrasekaran B. Taking attention away from the auditory modality: context-dependent effects on early sensory encoding of speech. Neuroscience 2018; 384: 64-75
  CrossrefPubMedGoogle Scholar
• 21 Krishnan A, Gandour JT, Bidelman GM. The effects of tone language experience on pitch processing in the brainstem. J Neurolinguist 2010; 23 (01) 81-95
  CrossrefPubMedGoogle Scholar
• 22 Jeng FC, Hu J, Dickman B. et al. Cross-linguistic comparison of frequency-following responses to voice pitch in American and Chinese neonates and adults. Ear Hear 2011; 32 (06) 699-707
  CrossrefPubMedGoogle Scholar
• 23 Musacchia G, Sams M, Skoe E, Kraus N. Musicians have enhanced subcortical auditory and audiovisual processing of speech and music. Proc Natl Acad Sci U S A 2007; 104 (40) 15894-15898
  CrossrefPubMedGoogle Scholar
• 24 Wong PCM, Skoe E, Russo NM, Dees T, Kraus N. Musical experience shapes human brainstem encoding of linguistic pitch patterns. Nat Neurosci 2007; 10 (04) 420-422
  CrossrefPubMedGoogle Scholar
• 25 Krizman J, Lindley T, Bonacina S, Colegrove D, White-Schwoch T, Kraus N. Play sports for a quieter brain: evidence from division I collegiate athletes. Sports Health 2020; 12 (02) 154-158
  CrossrefPubMedGoogle Scholar
• 26 Kraus N, Lindley T, Colegrove D. et al. The neural legacy of a single concussion. Neurosci Lett 2017; 646: 21-23
  CrossrefPubMedGoogle Scholar
• 27 Kraus N, Thompson EC, Krizman J, Cook K, White-Schwoch T, LaBella CR. Auditory biological marker of concussion in children. Sci Rep 2016; 6: 39009
  CrossrefPubMedGoogle Scholar
• 28 Chandrasekaran B, Hornickel J, Skoe E, Nicol T, Kraus N. Context-dependent encoding in the human auditory brainstem relates to hearing speech in noise: implications for developmental dyslexia. Neuron 2009; 64 (03) 311-319
  CrossrefPubMedGoogle Scholar
• 29 Jeng FC, Stehura KA, Hart BN, Giordano AT. Contralateral noise degrades frequency-coding accuracy in normal-hearing adults – preliminary results. Proc Meet Acoust 2021; 45: 050001
  CrossrefPubMedGoogle Scholar
• 30 Bidelman GM, Krishnan A. Effects of reverberation on brainstem representation of speech in musicians and non-musicians. Brain Res 2010; 1355: 112-125
  CrossrefPubMedGoogle Scholar
• 31 Dimitrijevic A, Alsamri J, John MS, Purcell D, George S, Zeng FG. Human envelope following responses to amplitude modulation: effects of aging and modulation depth. Ear Hear 2016; 37 (05) e322-e335
  CrossrefPubMedGoogle Scholar
• 32 Kraus N, Anderson S. The effects of aging on auditory processing. Hear J 2013; 66 (01) 36
  CrossrefPubMedGoogle Scholar
• 33 White-Schwoch T, Krizman J, Nicol T, Kraus N. Case studies in neuroscience: cortical contributions to the frequency-following response depend on subcortical synchrony. J Neurophysiol 2021; 125 (01) 273-281
  CrossrefPubMedGoogle Scholar
• 34 Gnanateja GN, Rupp K, Llanos F. et al. Frequency-following responses to speech sounds are highly conserved across species and contain cortical contributions. eNeuro 2021;8:ENEURO.0451-21.2021
  PubMedGoogle Scholar
• 35 Chou M-S, Lin C-D, Wang T-C, Jeng F-C. Recording frequency-following responses to voice pitch in guinea pigs: preliminary results. Percept Mot Skills 2014; 118 (03) 681-690
  CrossrefPubMedGoogle Scholar
• 36 Worden FG, Marsh JT. Frequency-following (microphonic-like) neural responses evoked by sound. Electroencephalogr Clin Neurophysiol 1968; 25 (01) 42-52
  CrossrefPubMedGoogle Scholar
• 37 Anderson S, White-Schwoch T, Parbery-Clark A, Kraus N. Reversal of age-related neural timing delays with training. Proc Natl Acad Sci U S A 2013; 110 (11) 4357-4362
  CrossrefPubMedGoogle Scholar
• 38 Russo NM, Nicol TG, Zecker SG, Hayes EA, Kraus N. Auditory training improves neural timing in the human brainstem. Behav Brain Res 2005; 156 (01) 95-103
  CrossrefPubMedGoogle Scholar
• 39 Song JH, Skoe E, Wong PCM, Kraus N. Plasticity in the adult human auditory brainstem following short-term linguistic training. J Cogn Neurosci 2008; 20 (10) 1892-1902
  CrossrefPubMedGoogle Scholar
• 40 Skoe E, Kraus N. Musical training heightens auditory brainstem function during sensitive periods in development. Front Psychol 2013; 4: 622
  CrossrefPubMedGoogle Scholar
• 41 Jeng FC, Lin CD, Wang TC. Subcortical neural representation to Mandarin pitch contours in American and Chinese newborns. J Acoust Soc Am 2016; 139 (06) EL190
  CrossrefPubMedGoogle Scholar
• 42 Ribas-Prats T, Almeida L, Costa-Faidella J. et al. The frequency-following response (FFR) to speech stimuli: a normative dataset in healthy newborns. Hear Res 2019; 371: 28-39
  CrossrefPubMedGoogle Scholar
• 43 Jeng FC, Schnabel EA, Dickman BM. et al. Early maturation of frequency-following responses to voice pitch in infants with normal hearing. Percept Mot Skills 2010; 111 (03) 765-784
  CrossrefPubMedGoogle Scholar
• 44 Anderson S, Parbery-Clark A, White-Schwoch T, Kraus N. Development of subcortical speech representation in human infants. J Acoust Soc Am 2015; 137 (06) 3346-3355
  CrossrefPubMedGoogle Scholar
• 45 Font-Alaminos M, Cornella M, Costa-Faidella J. et al. Increased subcortical neural responses to repeating auditory stimulation in children with autism spectrum disorder. Biol Psychol 2020; 149: 107807
  CrossrefPubMedGoogle Scholar
• 46 Jeng FC, Lin CD, Sabol JT. et al. Pitch perception and frequency-following responses elicited by lexical-tone chimeras. Int J Audiol 2016; 55 (01) 53-63
  CrossrefPubMedGoogle Scholar
• 47 Vanheusden FJ, Bell SL, Chesnaye MA, Simpson DM. Improved detection of vowel envelope frequency following responses using Hotelling's T2 analysis. Ear Hear 2019; 40 (01) 116-127
  CrossrefPubMedGoogle Scholar
• 48 Jones MK, Kraus N, Bonacina S, Nicol T, Otto-Meyer S, Roberts MY. Auditory processing differences in toddlers with autism spectrum disorder. J Speech Lang Hear Res 2020; 63 (05) 1608-1617
  CrossrefPubMedGoogle Scholar
• 49 Hornickel J, Kraus N. Unstable representation of sound: a biological marker of dyslexia. J Neurosci 2013; 33 (08) 3500-3504
  CrossrefPubMedGoogle Scholar
• 50 Cunningham J, Nicol T, Zecker SG, Bradlow A, Kraus N. Neurobiologic responses to speech in noise in children with learning problems: deficits and strategies for improvement. Clin Neurophysiol 2001; 112 (05) 758-767
  CrossrefPubMedGoogle Scholar
• 51 King C, Warrier CM, Hayes E, Kraus N. Deficits in auditory brainstem pathway encoding of speech sounds in children with learning problems. Neurosci Lett 2002; 319 (02) 111-115
  CrossrefPubMedGoogle Scholar
• 52 Rauterkus G, Moncrieff D, Stewart G, Skoe E. Baseline, retest, and post-injury profiles of auditory neural function in collegiate football players. Int J Audiol 2021; 60 (09) 650-662
  CrossrefPubMedGoogle Scholar
• 53 Ribas-Prats T, Arenillas-Alcón S, Lip-Sosa DL. et al. Deficient neural encoding of speech sounds in term neonates born after fetal growth restriction. Dev Sci 2022; 25 (03) e13189
  CrossrefPubMedGoogle Scholar
• 54 Musacchia G, Hu J, Bhutani VK. et al. Frequency-following response among neonates with progressive moderate hyperbilirubinemia. J Perinatol 2020; 40 (02) 203-211
  CrossrefPubMedGoogle Scholar
• 55 Chandrasekaran B, Krishnan A, Gandour JT. Relative influence of musical and linguistic experience on early cortical processing of pitch contours. Brain Lang 2009; 108 (01) 1-9
  CrossrefPubMedGoogle Scholar
• 56 Krizman J, Marian V, Shook A, Skoe E, Kraus N. Subcortical encoding of sound is enhanced in bilinguals and relates to executive function advantages. Proc Natl Acad Sci U S A 2012; 109 (20) 7877-7881
  CrossrefPubMedGoogle Scholar
• 57 Gardi J, Salamy A, Mendelson T. Scalp-recorded frequency-following responses in neonates. Audiology 1979; 18 (06) 494-506
  CrossrefPubMedGoogle Scholar
• 58 Galbraith GC. Two-channel brain-stem frequency-following responses to pure tone and missing fundamental stimuli. Electroencephalogr Clin Neurophysiol 1994; 92 (04) 321-330
  CrossrefPubMedGoogle Scholar
• 59 Krishnan A, Parkinson J. Human frequency-following response: representation of tonal sweeps. Audiol Neurotol 2000; 5 (06) 312-321
  CrossrefPubMedGoogle Scholar
• 60 Van Dyke KB, Lieberman R, Presacco A, Anderson S. Development of phase locking and frequency representation in the infant frequency-following response. J Speech Lang Hear Res 2017; 60: 1-12
  CrossrefPubMedGoogle Scholar
• 61 Galbraith GC, Amaya EM, de Rivera JM. et al. Brain stem evoked response to forward and reversed speech in humans. Neuroreport 2004; 15 (13) 2057-2060
  CrossrefPubMedGoogle Scholar
• 62 Xie Z, Reetzke R, Chandrasekaran B. Machine learning approaches to analyze speech-evoked neurophysiological responses. J Speech Lang Hear Res 2019; 62 (03) 587-601
  CrossrefPubMedGoogle Scholar
• 63 Lemos FA, da Silva Nunes AD, de Souza Evangelista CK, Escera C, Taveira KVM, Balen SA. Frequency-following response in newborns and infants: a systematic review of acquisition parameters. J Speech Lang Hear Res 2021; 64 (06) 2085-2102
  CrossrefPubMedGoogle Scholar
• 64 Rogers S, Girolami M. A First Course in Machine Learning. 2nd ed.. Chapman and Hall/CRC; 2020
  Google Scholar
• 65 Yi HG, Xie Z, Reetzke R, Dimakis AG, Chandrasekaran B. Vowel decoding from single-trial speech-evoked electrophysiological responses: a feature-based machine learning approach. Brain Behav 2017; 7 (06) e00665
  CrossrefPubMedGoogle Scholar
• 66 Hart BN, Jeng FC. A demonstration of machine learning in detecting frequency following responses in American neonates. Percept Mot Skills 2021; 128 (01) 48-58
  CrossrefPubMedGoogle Scholar
• 67 Llanos F, Xie Z, Chandrasekaran B. Hidden Markov modeling of frequency-following responses to Mandarin lexical tones. J Neurosci Methods 2017; 291: 101-112
  CrossrefPubMedGoogle Scholar
• 68 Cheng FY, Xu C, Gold L, Smith S. Rapid enhancement of subcortical neural responses to sine-wave speech. Front Neurosci 2021; 15: 747303
  CrossrefPubMedGoogle Scholar
• 69 Hart B, Jeng FC. Machine learning in detecting frequency-following responses. Proc Meet Acoust 2018; 35: 050002
  CrossrefPubMedGoogle Scholar
• 70 Llanos F, Xie Z, Chandrasekaran B. Biometric identification of listener identity from frequency following responses to speech. J Neural Eng 2019; 16 (05) 056004
  CrossrefPubMedGoogle Scholar
• 71 Llanos F, McHaney JR, Schuerman WL, Yi HG, Leonard MK, Chandrasekaran B. Non-invasive peripheral nerve stimulation selectively enhances speech category learning in adults. NPJ Sci Learn 2020; 5: 12
  CrossrefPubMedGoogle Scholar
• 72 Marsland S. Machine Learning: An Algorithmic Perspective. 2nd ed.. Chapman and Hall/CRC; 2015
  Google Scholar
• 73 Campbell A. Python Machine Learning: Complete and Clear Introduction to the Basics of Machine Learning with Python. Comprehensive Guide to Data Science and Analytics. Independently Published; 2020
  Google Scholar
• 74 Cristianini N, Shawe-Taylor J. An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods: And Other Kernel-Based Learning Methods. 1st ed.. Cambridge University Press; 2000
  CrossrefGoogle Scholar
• 75 Chen H, Tino P, Yao X. Probabilistic classification vector machines. IEEE Trans Neural Netw 2009; 20 (06) 901-914
  CrossrefPubMedGoogle Scholar
• 76 Rabiner LR. A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 1989; 77: 257-286
  CrossrefPubMedGoogle Scholar
• 77 Lin TH, Tsao Y. Source separation in ecoacoustics: a roadmap towards versatile soundscape information retrieval. Remote Sens Ecol Conserv 2020; 6: 236-247
  CrossrefPubMedGoogle Scholar
• 78 Lee DD, Seung HS. Learning the parts of objects by non-negative matrix factorization. Nature 1999; 401 (6755): 788-791
  CrossrefPubMedGoogle Scholar
• 79 Smaragdis P, Févotte C, Mysore GJ, Mohammadiha N, Hoffman M. Static and dynamic source separation using nonnegative factorizations: a unified view. IEEE Signal Process Mag 2014; 31: 66-75
  CrossrefPubMedGoogle Scholar
• 80 Xu L, Chen X, Zhou N, Li Y, Zhao X, Han D. Recognition of lexical tone production of children with an artificial neural network. Acta Otolaryngol 2007; 127 (04) 365-369
  CrossrefPubMedGoogle Scholar
• 81 Zhou N, Zhang W, Lee CY, Xu L. Lexical tone recognition with an artificial neural network. Ear Hear 2008; 29 (03) 326-335
  CrossrefPubMedGoogle Scholar
• 82 Xu L, Zhang W, Zhou N. et al. Mandarin Chinese tone recognition with an artificial neural network. J Otol 2006; 1: 30-34
  CrossrefPubMedGoogle Scholar