Proximal auditory AR rehabilitation: system integration and wellness applications—from hearing support up toward vestibular rehabilitation

We present a new rehabilitation system based on novel principles, which consists of an auditory augmented reality (AR) headset we originated. The auditory AR headset, which does not cover both ears, allows users to hear both Real and Virtual environmental sounds at the same time. It can also be used in combination with Hearing Aids. We have studied a system to support hearing-impaired people and conducted a test evaluation. The system was able to provide convenience akin to “reading glasses for sound” to those who had mild hearing disabilities. Furthermore, by combining the system with surrounding speakers, a completely novel virtual auditory illusion was created in which the sound image jumps into the ear and runs away. We name this “proximal auditory AR (PAAR)” system. This system directly affects the unconscious level of reflexes for maintaining a standing position and can generate very subtle body motion disturbance. Using this system, we can modulate the standing posture and observe the autonomic nerve system's ability to subliminally compensate for the disturbance, using a stabilometer that measures body sways by center of pressure (COP). We observed a significant difference in the declination of COP only when using the PAAR, which is combined with array speakers and the auditory AR headphone, compared using a conventional closed-type and a bone-conduction headphone. By analyzing such big data of physical movement through machine learning, we expect to realize new systems for diagnosis, rehabilitation, function maintenance, and fall prevention.

The auditory AR headset, equipped with a piezoelectric actuator as the sound oscillator, has high-pass filter-like characteristics that enhance the audible band above 1,000 Hz (Figure 1B).Therefore, the headset was expected to assist language listening with minimal energy.Also, because the AR headset does not cover the auricle, it can be used alongside hearing aids worn by the hearing impaired.When the actuator was tested on hearing-impaired people, it was found to provide "reading glasses"-like convenience for those with mild hearing impairment.However, the ears-free auditory AR headphone had limits to improve the severely hearing impaired people, because of its amplitude limit and frequency characteristics.We thus developed another system called proximal auditory AR (PAAR), in which features of the auditory AR headphones would stand out.
Since it does not cover the auricle, it can be used in combination with array speakers.This system reproduces a continuous sound between remote-field and proximal-field virtual direct sound, creating a virtual sound image space in which the sound approaches and jumps into and out of the ear, especially inside and outside the head.This feature was utilized to construct a new vestibular rehabilitation system.
In this article, we propose a wellness application using a "proximal auditory AR" system based on a "full-ears-free headphone."First, an overview of the system is presented, followed by participant measurements of vestibular function training.The system artificially induces a very slight "body sway," and uses this artificial sway for functional training of standing posture holding.
A bone-conduction headphone has recently become popular.It vibrates the skull, and because the signal is directly connected to the inner ear, the speed of sound transmission is fast, in the order of milliseconds.Therefore, the virtual impact of the binaural signal is small.This virtual impact is discussed in the following.
Attempts have been made to create a new vestibular rehabilitation system that works on the extrapyramidal system by subliminal stimulation of involuntary movements for standing and body balance.

System setup
Two speakers are placed 100 cm apart from the midline on either side.This presents the participant with far-distant sounds approaching from a remote place.The arrayed speakers placed to the right and left were adjusted to a height of 150 cm, with the vibrating surfaces facing each other so that the sound would reach the participant's ears perpendicularly.For the near-directed sound, three headphones were used: bone-conduction headphone, conventional closed-type headphone, and ears-free headphone.The combinations are described in Section 2.2.A stabilometer was placed between the left and right array speakers, and the participants were asked to stand on it with wearing each headphone.For the spatial sound operation Spat 5.0 for MAX system by IRCAM (French National Institute for Research and Coordination in Acoustics/Music, Paris, France) is used.We use a Wii Balance Board as a stabilometer to measure body sways by center of pressure (COP), which is produced by Nintendo (Kyoto, Japan).The measurement software for recording body sway data from the stabilometer was WBBSS (Wii Balance Board Stabilometry System), which is published by Assistant Professor Yuki Hyohdoh of Kochi University (3).

Stimuli
For the stimulus sound source, a jingle with an attack was used to induce a sense of spatial localization in the listener.Specifically, a small metallic cymbal sound used in Tibetan Buddhism was processed to 1,000 ms by extracting only the vicinity of the attack.

Process
Participants wore headphones and stood on a stabilometer at the center of the array speakers.The stimulus was the aforementioned jingle (1,000 ms long).This was alternately panned from left to right and right to left five times, for a total of 10 panning sessions per trial, to measure the participant's body sway.Two trials of measurements were taken per participant.There was a 5-s pause between panning to avoid influencing the measurement of body sway that appeared after the first-panning and the second panning.

Measuring conditions
In all measurements, the participant's eyes were opened to prevent falling, and a gazing point was placed to prevent swaying of the body due to eye movement.Under the following five conditions, the body sways were measured.
(1) Silent condition: The body sway measurement was performed for 60 s without any signal.(2) Array speakers ONLY: The body sway measurement was performed using only far-field acoustic stimuli from speakers array placed to the right and left without wearing headphones that present near-field acoustic stimuli.(3) Bone-conduction headphone: A combination of boneconduction headphone and array speakers presents each near-directed and far-directed acoustic stimulus.(4) Conventional closed-type headphone: A combination of conventional closed-type headphone and array speakers presents each near-directed and far-directed acoustic stimulus.( 5) Ears-free headphone (auditory AR headset): A combination of ears-free headphone and array speakers presents each neardirected and far-directed acoustic stimulus.From the following, the ears-free headphone refers to the auditory AR headphone developed in our laboratory.

Participants
A total of 30 students from the Faculty of Rehabilitation at Hiroshima International University cooperated in the study.Of them, 22 were men and 8 were women.
We analyzed the data of 29 participants in Section 3.2, excluding the data for 1 woman, which was a statistical outlier.This participant's body swayed significantly even when no stimulation was applied: In the box plot of the lateral sway component, the maximum value was 7.62 mm, but her value was 9.42 mm.

Sound setting
The presentation of near-directed acoustic stimuli is not always the same in terms of the actual volume reproduced, even if the sound source and sound level on the system side are standardized, owing to the differences in the shape of the headphone and the way the vibration is transmitted.Therefore, in this measurement, we asked six participants to cooperate, adjusted the volume of each headphone by subjective sensory volume, and applied the mean value.The set sound pressure value on the system (PC) side that outputs the signal is described in the following as dBSSL (system sound level).The sound pressure level adjusted by the sensory volume was measured with a sound level meter and is denoted as dBSPL (sound pressure level).
Since bone-conduction is a different method of transmitting sound than close-type or ears-free headphone, comparisons are difficult.The array speakers were adjusted by subjective sensory volume to provide the most panning.Table 1 provides the output from the system (PC) side as dBSSL (system sound level) and the measured values from the sound level meter as dBSPL (sound pressure level).
From these measurements, it can be seen that the sound pressure presented by the conventional closed-type headphone (63.7 dBSPL) and the sound pressure presented by our ears-free headset (63.2 dBSPL) are almost the same in the physical sound pressure measurements.In the following, we will evaluate and discuss the results of the participant measurements under these conditions.

Measurement
First, all participants were measured for the body sway in the silent condition and with array speakers only.In the case of bone-conduction headphone, conventional closed-type headphone, and ears-free headphone, six (3!) different orders of measurement were prepared to avoid hysteresis bias, and five participants were measured in each order, following which the results were averaged.In the case of boneconduction headphone, conventional closed-type headphone, and ears-free headphone, the measurements were taken twice each.

Probability elliptic evaluation method
In this measurement, alternating left-right panning sound stimuli were presented with a bone-conduction headphone, a conventional closed-type headphone, and an ears-free headphone to examine their effects on body sway.It is difficult to evaluate the results only by observing the trajectory of body sway.Therefore, we attempted to obtain an error ellipse (probability ellipse) that covers the results of body sway, and to evaluate it quantitatively using three parameters: long diameter, declination, and eccentricity.The error ellipse (probability ellipse) is the result of calculating the elliptic equation from the variancecovariance matrix of a two-dimensional normal distribution from the body sway trajectory.In this article, the cumulative distribution function was set to 0.900, so that the ellipse was fitted to cover 90% of the trajectory range.By obtaining the basis vectors from the mean vectors and covariance vectors of the leftright and front-back oscillations of the body sway measurement results, we can obtain the long and short diameters and the declination along the axis.Declination was evaluated within the range of (−90°, 90°) to 0°midline.If the ellipse is tilted to the left, it is marked within the range (0°-90°), with a positive value, and if tilted to the right, it is marked within the range (−90°-0°), with a negative value. Figure 2 is an example of fitting an error ellipse to the elementary data of body sway COP measurement results.The following graph shows the results of the body sway with the setting to 0 mm of the center.
In the example in Figure 2, when the ears-free headphones are worn, the ratio of the long to short diameters is the largest, resulting in an elongated ellipse.To evaluate the shape of this ellipse, the eccentricity ϵ was determined.The eccentricity 1 of the ellipse can be expressed as follows, using the long diameter a and the short diameter b.
Therefore, the closer the ellipse shape is to a circle, the smaller the value of the eccentricity.In Figure 2, the ears-free headphone has the largest eccentricity.The values are 0.76 for bone-conduction headphone (Figure 2A), 0.93 for closed-type headphone (Figure 2B), and 0.94 for ears-free headphone (Figure 2C), and the eccentricity becomes smaller in the order of ears-free, closed-type, and bone-conduction headphone.In other words, when wearing the ears-free headphone, the fitted ellipse has a large ratio of long to short diameter and an elongated sway range, which can be shown both in the figure and numerically.
The body sway can be evaluated based on the following parameters: (1) Eccentricity ϵ of the ellipse.
We call this "the probability elliptic evaluation method" and proceed with the following analysis.

Evaluation
In the tables presented, "Background" is the condition without sound stimuli, and "Standard Stimuli" is the condition when stimuli were presented only with the array speakers."1st trial" is the first of 10 panning sessions, and "2nd trial" is the second of 10 panning sessions.The "Effect of Stimuli" is the ratio of the "Background" and "Standard Stimuli", or "1st trial" and "2nd trial" in the case of eccentricity and diameter, and the difference in the case of declination.Table 2(d) also shows the Lateral Component Ψ obtained by multiplying the sine of the declination by the radius of motion (half-length diameter).The "Effect of Stimuli" represents the difference between the first and second trials, whose normal distribution is shown below Table 2

(d).
First, the "Eccentricity" ϵ was shown on Table 2(a): 0.89 and 0.85 for the first and second trials when wearing boneconduction headphone, 0.89 and 0.91 with a conventional headphone, and 0.89 and 0.90 with the auditory AR headphone.The "Effect of Stimuli" was not significantly observed, and the ratios between the first and second trials are roughly equal to 1.00.
For "Long Diameter," the bone-conduction headphones were 23.62 and 20.09 mm for the first and second trials, respectively; for the regular headphones, 20.48 and 25.02 mm; and for the auditory AR headset, 20.45 mm and 22.26 mm.As with the eccentricity, the ratio of the first to the second trial was almost equal to 1.00 and no significant change was observed.
The declination θ on Table 2(c): There is no significant difference between ② the "Array speakers only" condition and ③ the "Array speaker + Bone-conduction headphone" condition, and there is also no significant difference between ② the "Array speakers only" condition and ④ the "Array speaker + Closedtype headphone" condition.
There is a significant difference between ② the "Array speakers only" and ⑤ the "Array speaker + Ears-free headphone" condition   Standard stimuli with Array speakers + 1st trial 2nd trial Effect of stimuli ③ Bone-conduction headphone q 1 ¼ 1:22 q 2 ¼ À0:66 % 0 Dq 1!2 ¼ À1:88 ④ Conventional closed-type headphone q 1 ¼ 1:75 q 2 ¼ À0:03 ffi 0 Dq 1!2 ¼ À1:78 ⑤ Ears-free headphone q 1 ¼ 6:32 q 2 ¼ À0:17 ffi 0 Dq 1!2 ¼ À6:49 Declination angle q of Ears-free headphone and Array speakers only condition.We checked the values in Table 2(d) for ① the "Background noise only" condition.The value was −0.04 mm when no standard stimuli were presented.In the case of ② the "Standard stimuli with Array speakers only." in which stimuli were presented only with the array speakers, the value was 0.03 mm.Both values are close to zero, and almost no swaying was observed.Conversely, in the case of ⑤ the "Array speaker + Ears-free headphone" condition, the lateral sway was the largest at 0.92 mm at the first trial, and it was observed to decrease significantly from the second trial.
The absolute value of "Effect of stimuli" is small for ⑤ the "Array speaker + Ears-free headphone" condition, which have the largest lateral component in the first trial, and increase in the order of conventional closed-type headphone and boneconduction headphone.
The normal distribution graph also confirmed that the lateral body sway was largest for the ears-free headphone.The reason for this is explained in Section 4.1.
Without specific sound stimuli, naturally the l human body sways mostly in the longitudinal direction (4).Lateral sways are induced mostly by auditory stimuli and are estimated by the declination angle of the long diameter from the midline.
As described previously, a statistically significant difference in declination θ [Table 2(c)] was identified between ② the "Standard stimuli with Array speakers only" condition and ⑤ the "Array speaker + Ears-free headphone" condition.From this measurement, we have confirmed that the proximal auditory AR system works from the auditory periphery to the autonomic nervous system, and can be applied to vestibular rehabilitation.Normal distribution graphs for the lateral components.

Table 2 (
a-c) present the results for means of Eccentricity ϵ (a), Diameter δ (b), and Declination ϑ (c) when the probability ellipse was fitted.
Lee et al. 10.3389/fresc.2024.1288945Frontiers in Rehabilitation Sciencesin the first trial ["Array speakers only": mean = −0.92°,SD = 11.09;"Ears-free headphone": mean = 6.32,SD = 13.17,mean difference = −7.241363,paired t-test = −2.4917,df = 28, p-value = 0.019 (twosided), 95% confidence interval (−13.19 to −1.29)].The distribution of the difference between the two conditions does not have a significant deviation from the standard distribution [exact one-sample Kolmogorov−Smirnov test: D = 0.09, p-value = 0.94 (two-sided), Shapiro−Wilk normality test: W = 0.96, p-value = 0.33].To evaluate the declination more rigorously, we obtained the lateral sway components to multiply by the sine of the declination and the radius of motion; the values in Table 2(d) show the mean of the lateral sway components and the graphs show the normal distribution on each situation.

TABLE 1 System
Sound Level (SSL) adjusted by subjective sensory volume and its physical Sound Pressure Level (SPL).

TABLE 2
Means and effects of stimuli about eccentricity, long diameter and declination.