Abstract
Proprioception plays an essential role in natural motor control, and we argue that it will serve an equally important function in artificial control of motor prosthetic devices. An artificial sensory feedback signal that could substitute for proprioception in a Brain-Computer Interface (BCI) must be sufficiently informative to be used alone when vision is not available (sensory substitution), and it should integrate with vision to improve motor performance when it is (sensory augmentation). Achieving these qualities with an artificial signal requires a high-bandwidth channel, which can be achieved with an invasive neural interface. With invasive electrode arrays, we can manipulate the activity of populations of neurons using intracortical electrical microstimulation (ICMS), effectively transmitting useful information directly to the neural circuits where it is needed. To date, the dominant strategy for encoding artificial somatosensation has been biomimetic—trying to replicate, at the single neuron level, the neural activity seen during natural sensory processing. Here, we argue for a different, though complementary, learning-based approach. We propose taking advantage of the natural plasticity of the sensorimotor system, and asking the brain to learn, de novo, an artificial input. We hypothesize that the statistical dependencies, such as temporal correlations, that will be imposed on a natural (vision) and an artificial sensory input (ICMS) will be enough to drive learning and, ultimately, integration of the two inputs. Therefore we suggest that such a learning-based approach can achieve sensory substitution and augmentation of vision, the two desired properties of an artificial sensory feedback signal for clinical motor neural prostheses.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol CB 14(3):257–262. doi:10.1016/j.cub.2004.01.029
Alam M, Chen X, Fernandez E (2013) A low-cost multichannel wireless neural stimulation system for freely roaming animals. J Neural Eng 10(6):066010. doi:10.1088/1741-2560/10/6/066010
Bach-y-Rita P, Kercel SW (2002) Sensori-“motor” coupling by observed and imagined movement. Intellectica 35:287–297
Bach-y-Rita P, Kercel SW (2003) Sensory substitution and the human-machine interface. Trends Cogn Sci 7(12):541–546
Bach-y-Rita P, Collins CC, Saunders FA, White B, Scadden L (1969) Vision substitution by tactile image projection. Nature 221(5184):963–964
Barrese JC, Rao N, Paroo K, Triebwasser C, Vargas-Irwin C, Franquemont L, Donoghue JP (2013) Failure mode analysis of silicon-based intracortical microelectrode arrays in non-human primates. J Neural Eng 10(6):066014. doi:10.1088/1741-2560/10/6/066014
Berg JA, Dammann JF, Tenore FV, Tabot Ga, Boback JL, Manfredi LR, Peterson ML, Katyal KD, Johannes MS, Makhlin a, Wilcox R, Franklin RK, Vogelstein RJ, Hatsopoulos NG, Bensmaia SJ (2013) Behavioral demonstration of a somatosensory neuroprosthesis. IEEE Trans Neural Syst Rehabil Eng 21(3):500–507. doi:10.1109/TNSRE.2013.2244616 (a publication of the IEEE Engineering in Medicine and Biology Society)
Bin G, Gao X, Wang Y, Li Y, Hong B, Gao S (2011) A high-speed BCI based on code modulation VEP. J Neural Eng 8(2):025015. doi:10.1088/1741-2560/8/2/025015
Brockmeier AJ, Choi JS, Emigh MS, Li L, Francis JT, Principe JC (2012) Subspace matching thalamic microstimulation to tactile evoked potentials in rat somatosensory cortex. In: Annual international conference of the IEEE engineering in Medicine and Biology Society, IEEE Engineering in Medicine and Biology Society, pp 2957–2960
Burge J, Girshick AR, Banks MS (2010) Visual-haptic adaptation is determined by relative reliability. The Journal of neuroscience : the official journal of the Society for Neuroscience 30(22):7714–7721. doi:10.1523/JNEUROSCI.6427-09.2010
Butovas S, Schwarz C (2003) Spatiotemporal effects of microstimulation in rat neocortex: a parametric study using multielectrode recordings. J Neurophysiol 90(5):3024–3039. doi:10.1152/jn.00245.2003
Chestek CA, Gilja V, Nuyujukian P, Foster JD, Fan JM, Kaufman MT, Churchland MM, Rivera-Alvidrez Z, Cunningham JP, Ryu SI, Shenoy KV (2011) Long-term stability of neural prosthetic control signals from silicon cortical arrays in rhesus macaque motor cortex. J Neural Eng 8(4):045005. doi:10.1088/1741-2560/8/4/045005
Choi JS, DiStasio MM, Brockmeier AJ, Francis JT (2012) An electric field model for prediction of somatosensory (S1) cortical field potentials induced by ventral posterior lateral (VPL) thalamic microstimulation. IEEE Trans Neural Syst Rehabil Eng 20(2):161–169. doi:10.1109/TNSRE.2011.2181417 (a publication of the IEEE Engineering in Medicine and Biology Society)
Doty RW (1969) Electrical stimulation of the brain in behavioral context. Annu Rev Psychol 20:289–320. doi:10.1146/annurev.ps.20.020169.001445
Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433. doi:10.1038/415429a
Fagg AH, Hatsopoulos NG, de Lafuente V, Moxon KA, Nemati S, Rebesco JM, Romo R, Solla SA, Reimer J, Tkach D, Pohlmeyer EA, Miller LE (2007) Biomimetic brain machine interfaces for the control of movement. J Neurosci Off J Soc Neurosci 27(44):11842–11846. doi:10.1523/JNEUROSCI.3516-07.2007
Fetsch CR, Pouget A, DeAngelis GC, Angelaki DE (2012) Neural correlates of reliability-based cue weighting during multisensory integration. Nat Neurosci 15(1):146–154. doi:10.1038/nn.2983
Frey SH, Bogdanov S, Smith JC, Watrous S, Breidenbach WC (2008) Chronically deafferented sensory cortex recovers a grossly typical organization after allogenic hand transplantation. Curr Biol CB 18(19):1530–1534. doi:10.1016/j.cub.2008.08.051
Georgopoulos AP, Massey JT (1988) Cognitive spatial-motor processes. 2. Information transmitted by the direction of two-dimensional arm movements and by neuronal populations in primate motor cortex and area 5. Exp Brain Res 69(2):315–326
Gilja V, Chestek CA, Diester I, Henderson JM, Deisseroth K, Shenoy KV (2011) Challenges and opportunities for next-generation intracortically based neural prostheses. IEEE Trans Bio-med Eng 58(7):1891–1899. doi:10.1109/TBME.2011.2107553
Gilja V, Nuyujukian P, Chestek CA, Cunningham JP, Yu BM, Fan JM, Churchland MM, Kaufman MT, Kao JC, Ryu SI, Shenoy KV (2012) A high-performance neural prosthesis enabled by control algorithm design. Nat Neurosci 15(12):1752–1757. doi:10.1038/nn.3265
Gomez-Rodriguez M, Peters J, Hill J, Schölkopf B, Gharabaghi A, Grosse-Wentrup M (2011) Closing the sensorimotor loop: haptic feedback facilitates decoding of motor imagery. J Neural Eng 8(3):036005. doi:10.1088/1741-2560/8/3/036005
Gu Y, Angelaki DE, Deangelis GC (2008) Neural correlates of multisensory cue integration in macaque MSTd. Nat Neurosci 11(10):1201–1210. doi:10.1038/nn.2191
Hatsopoulos NG, Donoghue JP (2009) The science of neural interface systems. Annu Rev Neurosci 32:249–266. doi:10.1146/annurev.neuro.051508.135241
Heming E, Sanden A, Kiss ZH (2010) Designing a somatosensory neural prosthesis: percepts evoked by different patterns of thalamic stimulation. J Neural Eng 7(6):064001. doi:10.1088/1741-2560/7/6/064001
Heming EA, Choo R, Davies JN, Kiss ZH (2011) Designing a thalamic somatosensory neural prosthesis: consistency and persistence of percepts evoked by electrical stimulation. IEEE Trans Neural Syst Rehabil Eng 19(5):477–482. doi:10.1109/TNSRE.2011.2152858 (a publication of the IEEE Engineering in Medicine and Biology Society)
Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554. doi:10.1162/neco.2006.18.7.1527
Histed MH, Bonin V, Reid RC (2009) Direct activation of sparse, distributed populations of cortical neurons by electrical microstimulation. Neuron 63(4):508–522. doi:10.1016/j.neuron.2009.07.016
Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, Branner A, Chen D, Penn RD, Donoghue JP (2006) Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442(7099):164–171. doi:10.1038/nature04970
Hochberg LR, Bacher D, Jarosiewicz B, Masse NY, Simeral JD, Vogel J, Haddadin S, Liu J, Cash SS, van der Smagt P, Donoghue JP (2012) Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485(7398):372–375. doi:10.1038/nature11076
Houweling AR, Brecht M (2008) Behavioural report of single neuron stimulation in somatosensory cortex. Nature 451(7174):65–68. doi:10.1038/nature06447
Hyvarinen J (1982) Posterior parietal lobe of the primate brain. Physiol Rev 62(3):1060–1129
Johansson RS, Flanagan JR (2009) Coding and use of tactile signals from the fingertips in object manipulation tasks. Nat Rev Neurosci 10(5):345–359. doi:10.1038/nrn2621
Johansson RS, Westling G (1984) Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Exp Brain Res 56(3):550–564
Kaas JH, Nelson RJ, Sur M, Lin CS, Merzenich MM (1979) Multiple representations of the body within the primary somatosensory cortex of primates. Science 204(4392):521–523
Kalaska JF (1996) Parietal cortex area 5 and visuomotor behavior. Can J Physiol Pharmacol 74(4):483–498
Kozai TD, Marzullo TC, Hooi F, Langhals NB, Majewska AK, Brown EB, Kipke DR (2010) Reduction of neurovascular damage resulting from microelectrode insertion into the cerebral cortex using in vivo two-photon mapping. J Neural Eng 7(4):046011. doi:10.1088/1741-2560/7/4/046011
Krubitzer LA, Kaas JH (1990) The organization and connections of somatosensory cortex in marmosets. J Neurosci Off J Soc Neurosci 10(3):952–974
Kuiken TA, Li G, Lock BA, Lipschutz RD, Miller LA, Stubblefield KA, Englehart KB (2009) Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. J Am Med Assoc JAMA 301(6):619–628. doi:10.1001/jama.2009.116
Lacquaniti F, Guigon E, Bianchi L, Ferraina S, Caminiti R (1995) Representing spatial information for limb movement: role of area 5 in the monkey. Cereb Cortex 5(5):391–409
Lebedev MA, Nicolelis MA (2006) Brain-machine interfaces: past, present and future. Trends Neurosci 29(9):536–546. doi:10.1016/j.tins.2006.07.004
Lebedev MA, Tate AJ, Hanson TL, Li Z, O’Doherty JE, Winans JA, Ifft PJ, Zhuang KZ, Fitzsimmons NA, Schwarz DA, Fuller AM, An JH, Nicolelis MA (2011) Future developments in brain-machine interface research. Clinics 66(Suppl 1):25–32
Ledochowitsch P, Koralek AC, Moses D, Carmena JM, Maharbiz MM (2013) Sub-mm functional decoupling of electrocortical signals through closed-loop BMI learning. In: Annual international conference of the IEEE engineering in Medicine and Biology Society, IEEE Engineering in Medicine and Biology Society, pp 5622–5625. doi:10.1109/EMBC.2013.6610825
Li Z, O’Doherty JE, Lebedev MA, Nicolelis MA (2011) Adaptive decoding for brain-machine interfaces through Bayesian parameter updates. Neural Comput 23(12):3162–3204. doi:10.1162/NECO_a_00207
London BM, Jordan LR, Jackson CR, Miller LE (2008) Electrical stimulation of the proprioceptive cortex (area 3a) used to instruct a behaving monkey. IEEE Trans Neural Syst Rehabil Eng 16(1):32–36. doi:10.1109/TNSRE.2007.907544 (a publication of the IEEE Engineering in Medicine and Biology Society)
Makin JG, Fellows MR, Sabes PN (2013) Learning multisensory integration and coordinate transformation via density estimation. PLoS Comput Biol 9(4):e1003035. doi:10.1371/journal.pcbi.1003035
McGuire LM, Sabes PN (2009) Sensory transformations and the use of multiple reference frames for reach planning. Nat Neurosci 12(8):1056–1061. doi:10.1038/nn.2357
Merzenich MM, Kaas JH, Wall J, Nelson RJ, Sur M, Felleman D (1983a) Topographic reorganization of somatosensory cortical areas 3b and 1 in adult monkeys following restricted deafferentation. Neuroscience 8(1):33–55
Merzenich MM, Kaas JH, Wall JT, Sur M, Nelson RJ, Felleman DJ (1983b) Progression of change following median nerve section in the cortical representation of the hand in areas 3b and 1 in adult owl and squirrel monkeys. Neuroscience 10(3):639–665
Mulliken GH, Musallam S, Andersen Ra (2008) Decoding trajectories from posterior parietal cortex ensembles. J Neurosci Off J Soc Neurosci 28(48):12913–12926. doi:10.1523/JNEUROSCI.1463-08.2008
Nau A, Bach M, Fisher C (2013) Clinical tests of ultra-low vision used to evaluate rudimentary visual perceptions enabled by the BrainPort vision device. Transl Vis Sci Technol 2(3):1. doi:10.1167/tvst.2.3.1
O’Doherty JE, Lebedev MA, Hanson TL, Fitzsimmons NA, Nicolelis MA (2009) A brain-machine interface instructed by direct intracortical microstimulation. Front Integr Neurosci 3:20. doi:10.3389/neuro.07.020.2009
O’Doherty JE, Lebedev MA, Ifft PJ, Zhuang KZ, Shokur S, Bleuler H, Nicolelis MA (2011) Active tactile exploration using a brain-machine-brain interface. Nature 479(7372):228–231. doi:10.1038/nature10489
O’Doherty JE, Lebedev MA, Li Z, Nicolelis MA (2012) Virtual active touch using randomly patterned intracortical microstimulation. IEEE Trans Neural Syst Rehabil Eng 20(1):85–93. doi:10.1109/TNSRE.2011.2166807 (a publication of the IEEE Engineering in Medicine and Biology Society)
Omrani M, Pruszynki AJ, Scott SH (2013) Temporal evolution of task dependent signal in sensory-motor cortices. In: Annual meeting of the Society for Neuroscience, San Diego, 13 November 2013
Orsborn AL, Dangi S, Moorman HG, Carmena JM (2012) Closed-loop decoder adaptation on intermediate time-scales facilitates rapid BMI performance improvements independent of decoder initialization conditions. IEEE Trans Neural Syst Rehabil Eng 20(4):468–477. doi:10.1109/TNSRE.2012.2185066 (a publication of the IEEE Engineering in Medicine and Biology Society)
Parker PA, Scott RN (1986) Myoelectric control of prostheses. Crit Rev Biomed Eng 13(4):283–310
Penfield W, Boldrey E (1937) Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain 60:398–443. doi:10.1093/brain/60.4.389
Polikov VS, Tresco PA, Reichert WM (2005) Response of brain tissue to chronically implanted neural electrodes. J Neurosci Methods 148(1):1–18. doi:10.1016/j.jneumeth.2005.08.015
Pons TP, Garraghty PE, Ommaya AK, Kaas JH, Taub E, Mishkin M (1991) Massive cortical reorganization after sensory deafferentation in adult macaques. Science 252(5014):1857–1860
Prud’homme MJ, Kalaska JF (1994) Proprioceptive activity in primate primary somatosensory cortex during active arm reaching movements. J Neurophysiol 72(5):2280–2301
Recanzone GH, Merzenich MM, Dinse HR (1992) Expansion of the cortical representation of a specific skin field in primary somatosensory cortex by intracortical microstimulation. Cereb Cortex 2(3):181–196
Rincon-Gonzalez L, Naufel SN, Santos VJ, Helms Tillery S (2012) Interactions between tactile and proprioceptive representations in haptics. J Mot Behav 44(6):391–401. doi:10.1080/00222895.2012.746281
Sainburg RL, Poizner H, Ghez C (1993) Loss of proprioception produces deficits in interjoint coordination. J Neurophysiol 70:2136–2147
Schalk G, Miller KJ, Anderson NR, Ja Wilson, Smyth MD, Ojemann JG, Moran DW, Wolpaw JR, Leuthardt EC (2008) Two-dimensional movement control using electrocorticographic signals in humans. J Neural Eng 5(1):75–84. doi:10.1088/1741-2560/5/1/008
Schiefer MA, Polasek KH, Tiolo RJ, Pinault GCJ, Tyler DJ (2010) Selective stimulation of the human femoral nerve with a flat interface nerve electrode. J Neural Eng 7(2). doi:10.1088/1741-2560/7/2/026006.Selective
Shanechi MM, Williams ZM, Wornell GW, Hu RC, Powers M, Brown EN (2013) A real-time brain-machine interface combining motor target and trajectory intent using an optimal feedback control design. PLoS ONE 8(4):e59049. doi:10.1371/journal.pone.0059049
Shokur S, O’Doherty JE, Winans JA, Bleuler H, Lebedev MA, Nicolelis MA (2013) Expanding the primate body schema in sensorimotor cortex by virtual touches of an avatar. Proc Natl Acad Sci USA 110(37):15121–15126. doi:10.1073/pnas.1308459110
Simani MC, McGuire LM, Sabes PN (2007) Visual-shift adaptation is composed of separable sensory and task-dependent effects. J Neurophysiol 98(5):2827–2841. doi:10.1152/jn.00290.2007
Sober SJ, Sabes PN (2005) Flexible strategies for sensory integration during motor planning. Nat Neurosci 8(4):490–497. doi:10.1038/nn1427
Stoney SD Jr, Thompson WD, Asanuma H (1968) Excitation of pyramidal tract cells by intracortical microstimulation: effective extent of stimulating current. J Neurophysiol 31(5):659–669
Suminski AJ, Tkach DC, Fagg AH, Hatsopoulos NG (2010) Incorporating feedback from multiple sensory modalities enhances brain-machine interface control. J Neurosci Off J Soc Neurosc 30(50):16777–16787. doi:10.1523/JNEUROSCI.3967-10.2010
Tabot GA, Dammann JF, Berg JA, Tenore FV, Boback JL, Vogelstein RJ, Bensmaia SJ (2013) Restoring the sense of touch with a prosthetic hand through a brain interface. Proc Natl Acad Sci USA 110(45):18279–18284. doi:10.1073/pnas.1221113110
Tehovnik EJ (1996) Electrical stimulation of neural tissue to evoke behavioral responses. J Neurosci Methods 65(1):1–17
Tehovnik EJ, Tolias AS, Sultan F, Slocum WM, Logothetis NK (2006) Direct and indirect activation of cortical neurons by electrical microstimulation. J Neurophysiol 96(2):512–521. doi:10.1152/jn.00126.2006
van Beers RJ, Sittig AC, Gon JJ (1999) Integration of proprioceptive and visual position-information: an experimentally supported model. J Neurophysiol 81(3):1355–1364
van Beers RJ, Wolpert DM, Haggard P (2002) When feeling is more important than seeing in sensorimotor adaptation. Curr Biol CB 12(10):834–837
Velliste M, Perel S, Spalding MC, Whitford AS, Schwartz AB (2008) Cortical control of a prosthetic arm for self-feeding. Nature 453(7198):1098–1101. doi:10.1038/nature06996
Verstynen T, Sabes PN (2011) How each movement changes the next: an experimental and theoretical study of fast adaptive priors in reaching. J Neurosci Off J Soc Neurosci 31(27):10050–10059. doi:10.1523/JNEUROSCI.6525-10.2011
Vuillerme N, Hlavackova P, Franco C, Diot B, Demongeot J, Payan Y (2011) Can an electro-tactile vestibular substitution system improve balance in patients with unilateral vestibular loss under altered somatosensory conditions from the foot and ankle? In: Annual international conference of the IEEE Engineering in Medicine and Biology Society, IEEE Engineering in Medicine and Biology Society, pp 1323–1326. doi:10.1109/IEMBS.2011.6090311
Wallace MT, Stein BE (1997) Development of multisensory neurons and multisensory integration in cat superior colliculus. J Neurosci Off J Soc Neurosci 17(7):2429–2444
Warren JP, Tillery SI (2011) Tactile perception: do distinct subpopulations explain differences in mislocalization rates of stimuli across fingertips? Neurosci Lett 505(1):1–5. doi:10.1016/j.neulet.2011.04.057
Weber DJ, Friesen R, Miller LE (2012) Interfacing the somatosensory system to restore touch and proprioception: essential considerations. J Mot Behav 44(6):403–418. doi:10.1080/00222895.2012.735283
Woolley AJ, Desai HA, Otto KJ (2013) Chronic intracortical microelectrode arrays induce non-uniform, depth-related tissue responses. J Neural Eng 10(2):026007. doi:10.1088/1741-2560/10/2/026007
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 The Author(s)
About this chapter
Cite this chapter
Dadarlat, M.C., O’Doherty, J.E., Sabes, P.N. (2014). A Learning-Based Approach to Artificial Sensory Feedback. In: Guger, C., Vaughan, T., Allison, B. (eds) Brain-Computer Interface Research. SpringerBriefs in Electrical and Computer Engineering. Springer, Cham. https://doi.org/10.1007/978-3-319-09979-8_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-09979-8_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-09978-1
Online ISBN: 978-3-319-09979-8
eBook Packages: Computer ScienceComputer Science (R0)