Assistive device for orientation and mobility of the visually impaired based on millimeter wave radar technology—Clinical investigation results

We present clinical investigation results of a novel, wearable, orientation and mobility assistive device that uses radar technology. The radar device sends radio signals, detects objects in the environment, and conveys this information to the users by way of sound or vibration feedback. The aim of this investigation is to assess the effectiveness of the device for the visually impaired. During a two-week period of the investigation, 25 visually impaired participants used the device in their everyday activities and kept a diary of their experiences. The investigation included a training session as well as opening and closing interviews, including a standard QUEST 2.0 form to evaluate user satisfaction with assistive technology devices. 92 percent of the participants observed that the device improved their ability to perceive their environment, and 80 percent observed that it increased their confidence in independent mobility. Subjects: Instrumentation, Measurement & Testing; Systems & Controls; Circuits & Devices

ABOUT THE AUTHORS Tero Kiuru, Mikko Metso and Mikko Utriainen are research scientist at VTT Technical Research Centre of Finland Ltd., Espoo, Finland who specialize in radar technology.
Kirsimarja Metsävainio and Hanna-Mari Jauhonen (at the time of writing this paper) are physicians with the Kuopio University Hospital, Kuopio, Finland (KYS).
Riitta Rajala (KYS), Reeta Savenius (Finnish Federation of the Visually Impaired, Helsinki, Finland) and Mirja Ström are orientation and mobility instructors for the visually impaired.
Reijo Juntunen and Juha Sylberg are with the Finnish Federation of the Visually Impaired. They are specialists in the assistive technology devices.
Tiia-Nina Jylhä did her Graduate thesis with the emphasis on the technology discussed in this paper.

PUBLIC INTEREST STATEMENT
There are almost 300 million blind and visually impaired people in the world. Many of them face significant challenges in tasks the sighted people take for granted.
One of these challenges is getting information about people and objects outside the reach of the white cane.
Our radar sensor, which can be worn beneath clothing, solves this problem by giving the visually impaired people this information about the medium-range (0-4 meters).
In this article, we present the results of a twoweek user trial of the radar sensor with 25 visually impaired test users.

Introduction
According to data by World Health Organization (WHO), global prevalence of visual impairment is approximately 285 million, of whom 39 million are blind (Pascolini & Mariotti, 2010). Those visually impaired that require support in their orientation and mobility, use a variety of assistive devices and technologies (Handbook of Resources & Services for Persons Who are Blind or Visually Impaired, 2012), including the white cane, a guide dog, as well as several electronic travelling aids (ETA).
The white cane is by far the most widely used assistive device for visually impaired orientation and mobility today. It is efficient in finding ground level obstacles close to the user (<1 m), but does not help in detecting obstacles that are not standing on the ground (such as an open window), or obstacles farther away. Not being able to detect head level obstacles increases the risk of head level accidents. For example, a study including over 300 legally blind or blind participants (Manduchi & Kurniawan, 2011) reported more than 50 percent suffering a head level accident at least once a year or more often. Furthermore, they reported 23 percent of these incidents having medical consequences.
A variety of ETAs have been developed (Cuturi, Aggius-Vella, Campus, Parmiggiani, & Gori, 2016;Hersh & Johnson, 2008;Roentgen, Gelderblom, Soede, & de Witte, 2008;Roentgen, Gelderblom, Soede, & de Witte, 2009) to give visually impaired a better perception of their environment and help them to avoid hitting obstacles. The main operating principle of most ETAs is the detection of an obstacle in front or around the user by use of some sensor technology and conveying this information to the user by using sound or touch-based (haptic) signals.
Microwave and/or millimeter wave sensors have also been suggested (Pisa, Piuzzi, Pittella, & Affronti, 2015;Scalise et al., 2012) for ETAs, but-to the author's knowledge-they have never been tested with visually impaired users. In this paper, we present clinical investigation results for 25 visually impaired participants who have used a novel, wearable, millimeter wave radar ETA for a twoweek period.
Our radar device differs from the ETAs introduced above in two ways. First, it uses a 24 GHz electromagnetic radar sensor for obstacle detection, which makes it possible to wear the device under heavy clothing, as microwave and millimeter wave signals penetrate typical clothing materials easily. This property is not found in any other mobility aid device used by the visually impaired. Secondly, it features an automatic inclination and orientation sensing and compensation, meaning that it can signal the user if the sensor is worn so that it is pointing either towards the sky or towards the ground. This way it can also automatically compensate for smaller inclination errors, effectively freeing the user from the stress of always wearing the sensor in a "correct" way.
The investigation described in this paper has been organized to make sure that the assistive device under test is not harmful for visually impaired users in any way, and to investigate how useful it is for them. One very important aspect is possible accidents that might occur if users become overconfident when using it and therefore they will be less careful than what they would otherwise be. The device is meant to be a complementary ETA for the white cane, not a replacement.

Background and motivation for the work
As discussed in the previous chapter, there are several ETAs available for the visually impaired. However, our discussions with the Finnish Federation of the Visually Impaired suggested that the current solutions all have some features that are not always wanted by visually impaired users. These include, e.g. needing to keep the device in users hand while using it, not having the possibility to use the device discreetly under outer clothing and poor operation in low lighting conditions (camera technologies). These views were confirmed by a Thesis work at the Haaga-Helia University of Applied Sciences (Jylhä, 2016), in which the use and preferences of assistive devices were evaluated among several other topics. In the Thesis work, a question was posed on the importance on the possibility to hide the assistive device and also on the importance of the assistive device leaving the users hands free.
The results are clear. On the scale from 1 to 5, where 1 is not important and 5 is very important, 59 percent of the respondents answered 4 or 5 for the hiding possibility and an overwhelming 95 percent answered 4 or 5 for the assistive device leaving hands free. A more detailed breakdown of the answers is given in Figure 1.
This means that some of the assistive devices on the market that are handheld, such as the Miniguide, are problematic for some users. In addition, most commercial and research phase ETAs are based on either ultrasonic or camera sensors, and this makes it very difficult or impossible to use them beneath clothing, which is also valued high by the visually impaired users.
A short comparison evaluating some commercial and research phase ultrasonic and camera sensor ETAs is given in Table 1. Based on these findings, we concluded that among the visually impaired, there is a demand for a wearable ETA that can be hidden and used beneath outer clothing. Notes: Scale is between 1 and 5, where 1 = not important and 5 = very important.

Technology and operating principle
The operation of our assistive device is based on the frequency-modulated continuous wave (FMCW) radar principle. In short, the device transmits and receives radio waves, and the distance and direction of possible obstacles in front of the user can be calculated from the received signal properties. The information is then conveyed to the user by means of haptic feedback and/or by sound signals. The operation of the device is illustrated in Figure 2, and a detailed discussion on the miniaturized radar technology is given in Kiuru et al. (2016).
We designed the assistive device to be as simple to use as possible, with only one on-off button, a standard 3.5 mm audio jack and a 6.5 mm power connector for charging the battery. The device is worn as a common heart rate monitor belt. Battery life at full power and with maximum vibration feedback was measured between 4 and 5 h for this prototype, when fully charged. Power consumption is not optimized for the prototypes used in this study, so the maximum usage time is expected to increase significantly in the future.
The range from which this prototype gives vibration or voice feedback is set to 3.5 meters. This distance is further divided to three different vibration frequencies to convey information from range bins of 0.3-1.5, 1.5-2.5 and 2.5-3.5 m. The vibration elements are connected to the device casing. The voice feedback is given in spoken numbers, where "one" signifies the first range bin, "two" the second and "three" the third, in similar way as with the haptic feedback.
The radar sensor detects obstacles in a 25 degree horizontal angle, which means that it detects, for example, a 0.9 m wide area 2 m in front of the user. The vertical angle is approximately 70 degrees, which makes it possible to detect obstacles 1.4 m above the position of the sensor at 2 m distance. With this feature, it is possible to suppress unwanted echoes from objects that would not touch the user even if he or she would walk without the device. Outside of this field of view (25 o × 70 o ) the radar signal does not detect anything.

Investigation setup and procedure
Clinical investigation involving 25 participants was conducted to verify the safety of the technology and its usability in the everyday environment of the visually impaired. Eye doctors took part in the investigation with the VTT Technical Research Centre of Finland Ltd., the Finnish Federation of the Visually Impaired, and with the Kuopio University Hospital. Mobility instructors were present in many of the meetings and training sessions arranged with the test users.
The Research Ethics Committee of the Northern Savo Hospital District evaluated and accepted the research plan for the investigation. After that, The Finnish National Supervisory Authority for Welfare and Health (Valvira) approved the investigation. The title for the research plan that is approved by Valvira is for "Clinical investigation on medical devices" of which we use the short version "Clinical investigation" or "Investigation".
In order to find test users, we put up an open invitation to the web pages of the Finnish Federation of the Visually Impaired. In total, approximately 35 people volunteered for the investigation. We then selected 25 people based on the following inclusion criteria: (1) Level of visual impairment (self-reported). Inclusion of people with blindness or low-vision worse than 0.3 (best corrected).
(2) Ability to independently use the device and move around outdoors.
The number of people in the investigation was ultimately a compromise to get as much statistically meaningful data and on the other hand our ability to manufacture prototype devices and organise meetings with the test users.
The tests were conducted from September to October 2016. The test length for each user was two weeks on average. Each individual test run consisted of three parts: (1) Initial meeting, (2) a twoweek testing period, and (3) a closing meeting. We provided each test user with necessary information regarding the test during the initial meeting, such as instructions on the device and its usage. We also went for a short walk with each test user to make sure that the user has learned device basics to start with. We chose a two-week test period where the users could freely use the device because we wanted to see how the device performs in real environment and in real use cases, instead of just arranging, for example, an obstacle course.
Both parties signed an investigation agreement. In particular, all participants were informed that the assistive device is meant to be a complementary ETA for the white cane, instead of a replacement. This means that if the participant typically uses a white cane, he or she should continue to do so while using the device.
During the second part of the investigation, users could freely decide when and where they used the device. They wrote down short notes of each use, describing their experiences. In addition, these prototype devices automatically recorded their usage time, storing time stamps into internal memory. The users were provided with our contact information for support.
We held a closing meeting with each test user after the two-week test period. We also conducted an interview on experiences of the test period and took the device back to be handed to the next test user, if necessary. The interviews included a standard QUEST form (version 2.0), in order to have comparable results with similar surveys related to other assistive technologies.

Test group
The test group of 25 people was composed of three types of visual impairment. Fourteen of them were blind, seven had low vision, including night blindness or tunnel vision, and the remaining four were deaf-blind.
In order to get more insight from the test results, the test group is further segmented into 14 different groups based on socio-demographics and a few other criteria. In addition to the above mentioned classification based on visual impairment, these are: Young (under 30 years of age), Middle age (30-60 years of age), Seniors (over 60 years of age), Female, Male, Guide dog owners, Active, Born blind, People living alone, People living with family, and People living in rural areas.
The segmentation is not based on any pre-existing standard, but is rather the result of our teams discussions on how to best convey meaningful information about the test group. It is noted here that due to the relatively small size of the segments and the overlap between segments, these results are only indicative, not statistically significant.

Investigation results
The average use time per test user was 7.0 h and the median use time 5.8 h. According to data analysis, QUEST 2.0 user satisfaction overall score is 3.8. More detailed data on these responses is given in the Tables and Figures. The most important finding of the investigation was that using our assistive device did not cause any perilous situations for the test group. The participants reported zero accidents (collisions or otherwise) while wearing the device cumulatively for 175 h (the whole test group).
The individual QUEST 2.0 items are given in Table 2. The Overall score is 3.8 and the average standard deviation 0.4. The best score (4.6) is for the item "ease of use". This is logical as the device only has one on-off button, a 3.5 mm audio jack and a connector for a 6.5 mm power cable. The lowest score (2.9) is for the item "dimensions". This can be expected as the prototype used in this study had not yet been optimized for size and weight. In the future, both size and weight are expected to decrease significantly. The QUEST 2.0 results based on test group segmentation are given in Table 3. Results between different segments are remarkably similar. Difference between the lowest (seniors) and highest score (deaf-blind) was only 0.4 points.
Some non-standard questions related specifically to the assistive device were included in the interviews in addition to the standard QUEST 2.0 questionnaire. Two of these questions are considered highly relevant regarding the device as an ETA for the visually impaired. They are related to the improved environmental perception and to the confidence in independent mobility. These results with test group segmentation are given in Table 4 and answer breakdown to different categories is shown in Figure 3.
The results indicate that people experiencing the greatest benefit in their orientation and mobility while using the device are the deaf blind, people with low vision and ones living in rural areas. On the other hand, people experiencing the least benefit were the blind, people living alone and guide dog owners. However, with such a small sample size, one should not draw conclusions with too much certainty.
A large majority, 92 percent of the test users, answered that the device improved their environmental perception at least to some extent (category 3 or higher) and 36 percent felt that their environmental perception improved significantly or very significantly (category 4 or 5). Regarding the confidence in independent mobility, 80 percent of the test users answered that the device improved their confidence in independent mobility at least to some extent (category 3 or higher) and 44 percent felt that their confidence in independent mobility improved significantly or very significantly (category 4 or 5).
Our assistive device seems to fit well for the blind and visually impaired, although the device for these tests was only the first prototype, with its size and weight not yet thoroughly optimized. Many of the test users were willing to obtain this device if it was commercialized.
We also asked test users about the possible drawbacks of the device in the non-standard question battery. The feedback indicates that two features require further design effort. These are the lack of control in the range from which the vibration or voice feedback is given and the intensity of the haptic feedback.
As discussed earlier, the range from which this prototype version gives vibration or voice feedback was set to 3.5 m. As the test users were of different physical shape and age, and with different walking speeds, it turned out that the device should include the possibility to change the distance from which it detects obstacles. Regarding the haptic feedback, some users reported that they had trouble feeling the haptic feedback. In this prototype version the vibration elements were embedded in the device casing, and this attenuated the vibration in a way, that it became challenging for some users to feel it.
Both of these reported issues are straightforward to fix, as is discussed in the next paragraph of this paper.

Discussion and future work
The most important finding of this investigation was that our assistive device did not cause any dangerous situations for the test group during the cumulative usage time of 175 h. In addition, the general view of the clinical investigation is that the device adds value in the daily life of the visually impaired. The results point to the direction that the deaf blind might get the greatest benefit from using the device, along with people with low vision.
In addition to the quantitative feedback presented above, the test users gave feedback in the form of open written questions and verbally during the interviews. The feedback indicates that the users value the wearability and the possibility to use the device beneath outer clothing highly.
The main reason for this is that wearability leaves hands free, which is a great advantage for the visually impaired, who already have to carry the white cane in their dominant hand. In addition, having the possibility to use the device hidden under clothing gives visually impaired the feeling of control regarding how they are seen by the sighted.
There are very few quantitative research papers on ETA investigations with enough participants who answered a QUEST questionnaire to do comprehensive comparison. The best reference with this information that we could find is Roentgen et al. (2012), where 8 users evaluated the Miniguide and the Ultracane ETAs. The QUEST results in this work are given in a graph and not in a numeric form, but it is possible to estimate that the average is 3.8 ± 0.2, which is the same or at least very close to the evaluation of the assistive device in this work.
The future development work will include the addition of range control for the radar sensor as well as better placement and tunability of the haptic feedback units. Moreover, as hardware components are optimized, the size and weight of the device will decrease significantly at the same time as the maximum usage time increases, making the device more pleasant to wear and use.

Conclusions
We have presented the operation principle and clinical investigation results of a novel, wearable, orientation and mobility assistive device based on millimeter wave radar technology. Our results from a two-week test period with 25 visually impaired test users indicate that the device adds value in the daily lives of the visually impaired. Among the interviews conducted with the test users was the standard QUEST 2.0 questionnaire. In QUEST 2.0 our device scored 3.8 with the standard deviation of 0.4. Notes: The categories are: (1) not at all, (2) not much, (3) to some extent, (4) significantly (5) very significantly. Y-axis indicates the number of answers to each category.