Keywords

1 Introduction

Vision plays a critical role in many people’s lives. People use vision to navigate from place to place. However, many other people who cannot see or who have impaired vision navigate their way through the same places independently on a daily basis as well. There are some existing tools to aid blind navigation such as canes, Global Positioning System (GPS) that provides auditory signals [1], Radio Frequency Identification (RFID) [2], OEM ultrasound positioning system [3] and others. However, there are many technical difficulties in guiding the visually impaired through chaotic urban areas, indoors and complex crosswalks, where GPS signal is affected or blocked by the Urban Canyon Effect and walls. A conventional single navigational system becomes inaccurate and would drift farther and farther from the correct route if not corrected [4]. The dangerous traffic and loud ambient noise can also be a distraction and barrier for the blind users to receive the navigational instructions. There is a need for a more accurate and interactive navigation system for the blind and visually impaired community that is relatively easy to operate, light to carry, and provides users with concise and easy-to-interpret directional guidance.

2 System Components

This system incorporates three redundant positioning systems: visual odometry, pedometry, and GPS (Fig. 1) to increase the accuracy of navigation in different environments. The global positioning system (GPS) used in this study was augmented with and enhanced by visual odometry and pedometry. The visual odometry system was mounted to the chest, consisting of a stereo camera and an inertial measurement unit (IMU). The pedometry system was mounted to the foot using an IMU to detect footsteps to provide heading information and distance traveled. These components were combined with a Kalman filter for more accurate positioning than any single component can provide.

Fig. 1.
figure 1

System hardware components

Full size image

To find alternative delivery methods of the navigational systems than visual cues, the created system uses auditory and haptic cues instead to overcome the complicated ambient environment for the blind. An iPhone application (app) was created that is capable of reading its contents out loud as users hover over text. The iPhone was chosen because more than 60 % of the blind population in the US use iPhone and its Voice Over function, according to National Federation of the Blind (NFB) [5]. The app allows users to input origin and destination, provide estimated travel time and landmarks around the user in real time, as well as guidance to find access to destinations (ex. restaurants or shops). The user would also wear a tactile belt, which is wirelessly controlled by the iPhone app, to receive directional haptic cues. The app also allows volume adjustment and provided options to turn on or off the abovementioned features. In addition, Dedicated Short Range Communication (DSRC) radios are used to provide information on the location of crosswalks, crosswalk signals, and traffic light signals to users.

Redundant Positioning Systems.

For assisting the visually impaired as they commute unfamiliar environments, a confident global navigation solution must be attained. Such a system should provide ubiquitous positioning with the ability to handle various environmental conditions. For tracking the user’s global position and orientation, an array of navigation devices are fused into one overall solution. The sensor suite consists of inertial measurement units (IMU), a stereo camera, and GPS. An IMU is capable of measuring accelerations and rotation rates in all directions. Provided perfect measurements from the device, a 3D position and orientation can be attained. The IMU is very common for navigation purposes because of its high sample rate and ability to operate in virtually all environmental conditions. For a MEMS grade IMU, like the one used in this project, the size, weight, power consumption, and cost are relatively low making it ideal for consumer products. In general, a MEMS grade IMU cannot be used for stand-alone navigation due to noisy, biased measurements. Coupling an IMU with other navigation systems can counteract such errors while maintaining the attributes of the inertial device. A stereo camera is used to perceive motion of the user. By comparing consecutive images from the camera, a “visual odometry” algorithm is capable of producing measurements of rotation and translation time period between frames. Such measurements are redundant to those produced from an IMU. By fusing redundant measurements, the accuracy and reliability of the navigation solution can be improved. Visual odometry does, however, rely on a certain number of stationary, visible objects in the camera frame. Such system suffers in highly dynamic environments (such as crowded streets, malls, subway station etc.) and low lighting scenarios. For this reason, complementary fusion with an IMU proves beneficial. For relating the navigation solution to points of interest throughout the user’s commute, GPS provides positioning in a global reference frame. GPS is extremely common in navigation systems although it suffers in many scenarios. When the GPS receiver is surrounded by tall buildings or foliage, the signal’s path from the satellite can be obstructed, causing erroneous position calculations. A GPS solution is also unavailable in virtually all indoor environments. Because of this, the IMU and camera systems are used to propagate the global navigation solution through sections of GPS unavailability.

Mapping.

The bridge between the tactile belt and the navigation solution from the visual odometry, pedometry, and GPS systems is the mapping system for guiding the blind user from one location to another. The map consists of a network of nodes and linear links known as a graph. Each link has an associated weight determined by the length of that link between the two nodes at the ends of that link. Each node is a potential destination or point where the user must turn, while each link is a possible path for the blind user. For example, a city intersection could consist of four nodes with connecting links to show the possible paths around the block, as shown in Fig. 2. In this figure, the right intersection has four crosswalks for the corresponding four nodes and links. In the left intersection, a crosswalk does not exist, and as such, no link is present for that crosswalk. When the blind user enters a destination through the iPhone, a route is calculated using Dijkstra’s algorithm, which is a path planning algorithm for finding the shortest path between two nodes in a graph [6]. Figure 2 shows two calculated paths for two desired goals. One goal is to reach a business meeting directly, and the other goal is to go to breakfast followed by the meeting. The calculated path from the hotel to the meeting (orange) takes the shortest route by crossing the road through the right intersection, traveling along the city block, and finally to the business meeting. The calculated path to breakfast and then to the meeting (purple) crosses the intersection, through a park, and then to breakfast. From the breakfast destination, the blind user travels to the intersection but is safely directed all around the intersection due to the lack of a crosswalk at that crossing. The tactile belt can guide the user to the next desired node with the knowledge of the user’s current position from the redundant positioning systems explaining in earlier sections. The next section describes how the tactile belt guides the user to the next desired node along the linear link through notifications of either slight corrections to heading or a hard turn.

Fig. 2.
figure 2

Example of two calculated routes for two desired goals

Full size image

Vibro-tactile Belt.

The tactile belt’s computer hardware is based from an Arduino Uno microcontroller. An Arduino was selected over alternatives due to its very low power consumption, Pulse-Width Modulation (PWM) output capabilities, and relative ease of use. The Uno’s main chipset is an ATmega328 with a 16 MHz crystal and communicates over serial connection at 9600 baud rate. For the tactile belt unit, only four of the six PWM outputs are used, as the tactile belt has four vibrating motors. To power the DC motors, a separate circuit is needed outside the Arduino base. By using a transistor circuit, the high current loads needed to power the DC motors are isolated from the sensitive microcontroller. The Arduino is simply used to send a small pulse signal to a TIP120 NPN transistor. This specific transistor was chosen for its high current handling capabilities and overall robustness. Transistors typically have three pins – a base, emitter, and collector. The base of the transistor is connected to the Arduino’s output, which is protected against surge currents by the use of a 1 k-Ohm resistor. Since the DC motors used for the belt have internal brushes, a standard 1N4004 diode is integrated into the circuit to only allow current to flow to the motor. DC brush motors have a propensity to send extremely high reverse voltage spikes when the motor comes to a stop, which is what the integration of diodes prevents. By oscillating the motor’s ground and not the positive source, the transistor stays much cooler.

The vibrating motors were chosen based on overall vibrational intensity. Since the Arduino has a 5 V DC output, all the motors tested were 5 V motors of different size, weight, and rotational speed. The motors currently in use are low-torque, 3200- RMP motors that weigh 20 grams each. A larger motor was chosen for the user to sense feedback through a thicker jacket, if necessary. By using PWM, the motor’s rotational speed can be adjusted to four distinct rotational speeds, each of which allows for a progressively more intense vibration. Four vibrating motors are integrated into the belt to guide the user. In order to be as unobtrusive and not overwhelming as possible, the belt is made with only four tactile user inputs. These four motors will be positioned 45 degrees off center from the standard cardinal directions, shown in Fig. 3.

Fig. 3.
figure 3

Tactile belt motor layout

Full size image

The tactile belt is hard coded to accept different heading angles and use this information to delegate the direction of travel to the user. The four variables being passed from the laptop to the belt are heading angle, range, volume, and an emergency stop command. There are four different vibrating patterns for the UX (Fig. 4).

Fig. 4.
figure 4

Heading decision map

Full size image
  1. 1.

    When heading is between −15 and 15 degrees from perpendicular, the belt will not send a vibrating response. This is a design aspect to make the UX less cumbersome and of an annoyance. For now the 15-degree variations have been arbitrarily chosen, but further testing will prove a more definitive cutoff.

  2. 2.

    Between −60 and −30 degrees, and 30 to 60 degrees from perpendicular, the area denoted by (2), is a path correction vibration. When the user’s heading angle is between these cutoffs, a single vibratory pulse of 500 ms on each side will alert user to correct their path. If the user deviates to the right side of the defined path, a left side vibratory pulse will be issued. If the user deviates to the left side of the defined path, a right side vibratory pulse will be issued.

  3. 3.

    The region between −60 and −90 degrees, and 60 and 90 degrees is defined as the ‘hard turn’ region. When heading angle to the next node is in this region, the belt will pulse on the side of the body that turn occurs. (i.e. a left turn results in a tactile pulse of the left side front and back motors). The pulse will be of the pattern pulse-delay-pulse, where the pulses last for 500 ms apiece. This hard turn is differentiated from a path correction by the range to the next waypoint.

There is an emergency stop function that can override any command with a simple Boolean switch from the laptop. This e-stop pattern is pulse-delay-pulse-delay-pulse. The pulses are one second in length, and regardless of the volume setting, and pulsed at the maximum vibration volume. All four motors will also pulse during an e-stop command.

iPhone Application.

The iPhone, recommended by NFB for its accessibility and wide use among the blind community, relays information with the navigational computer through Low Energy Bluetooth, interface with a designed graphical user interface and blind user through the iPhone’s built-in accessibility capabilities, and communicate with the other components in the navigational system. Several communication protocols were explored for passing messages between the iPhone and the rest of the system. Wifi was not feasible due to a lack of support in many areas. While researching the applicability and usability of Bluetooth for this project, a fairly new technology called Bluetooth low-energy was also investigated. Upon further research, it was found that Bluetooth low-energy was the best choice. Software from another team on the project could only be implemented in Linux. In order to handle both the iPhone development and passing messages from the iPhone to the navigation system, a MacBook Pro runs Linux and Parallels, which is a platform that allows a user to run more than one operating system on a single computer. The OSX application Pro running Linux in Parallels (Parallels is a platform to run more than one operating system on one computer.) was built to send and receive messages to and from the iPhone opens a Transmission Control Protocol (TCP) port to send messages to a listening Robot Operating System (ROS) node, which then publishes the messages for the rest of the system to access. The OSX application also listens for messages and passes them to the iPhone when received. The iPhone’s accessibility functions include voice over, zoom (for visually impaired), color inversion, larger or bolded text, as well as phone noise cancellation and LED flashing for the hearing impaired. Of particular interest to our project and the blind community is voice over. With voice over enabled on the iPhone, a voice will read the title of each page aloud, along with any selected text and the title of any labels or buttons highlighted. In addition to selected text and titles, various hints and suggestions, which the programmer can add or alter as needed, will be read aloud. Also, while voice over is enabled, navigating the iPhone becomes just as easy for the visually impaired as it is for the sighted. Simply swiping one finger to the right will progress forward across options and swiping left will progress back. When the desired option is found a double tap will select it.

3 Conclusion and Discussion

The three redundant positioning systems used in this study better assures positioning accuracy. The combination of auditory and haptic cues with volume adjust abilities provide users with multiple options of receiving directional guidance. And the iPhone application serves as the main user interface where visually impaired users take advantage of its Voice Over function to send command to the rest of the system and receive feedbacks accordingly.

This study is currently in the process of acquiring Institutional Review Board (IRB) approval for human subject testing. Upon approval, 12 visually impaired subjects that meet physical and other requirements will be recruited to test the system performance. Subjects will be asked to have the designed system mounted on them and walk on designated routes. Data will be recorded in real-time and will also be used post-experiment to simulate the subjects’ gaits and routes in order to analyze the system performance.