HeadsUp: Keeping Pedestrian Phone Addicts from Dangers Using Mobile Phone Sensors

Walking while staring at the mobile phone is dangerous, and the danger mainly arises from distraction. While watching the mobile phone, one could fall into a deep well without noticing the manhole cover was missing, one could be hit by a rushing car without observing the traffic light, and so forth. Some mobile phone users are already aware of the crisis, and they keep looking up and down to allocate some focus to danger spying; however, the statistics data revealed by US government make such efforts frustrating: about 1,152 pedestrians are injured in US in the year 2010, while they were using mobile phones, and the number doubled in the year 2012. This paper identified the possibility of using mobile phone sensors to develop a walk pattern recognition system. By sampling from embedded sensor, such as accelerometer and gyroscope, the movement pattern of mobile phone users can be computed. We design and implement HeadsUp, a system that warns pedestrian and locks the screen when one looks at the mobile phone while walking. Evaluation results from experiments of 20 testers in real life situation show that, on average, the false negative rate is less than 3%.


Introduction
The rapid growth of needs for smart mobile terminals expands the mobile phone industry; each 100 persons own 82.6 mobile phones in China by the end of the year 2012, according to MIIT (the Ministry of Industry and Information Technology) of China [1]. People carry around their phones anywhere and anytime, not only for communication, but also for entertaining. People read news using mobile phones while they are on subways and buses, people watch movies using mobile phones while they are lying on couches and beds, people chat with friends using mobile phones while they are walking and driving. We call these people as "phone addicts" or "phubbings" [2]. Some researches [3] report that "phone addiction" makes people indifferent with each other, and traditional social behaviours are breaking down. In fact, other than social problems, "phone addicts" are suffering safety crisis and some of them are not even aware of that.
The US Consumer Product Safety Commission reports that 1,152 pedestrians were treated in emergency rooms after being injured while using a mobile phone or some other electronic devices in 2010, and the number had doubled since the year before [4]. The number of pedestrian deaths and injuries has risen dramatically as a result of texting while walking, the number of pedestrians who died rose from 4,109 in 2009 to 4,432 in 2011 and an additional 69,000 injuries, according to a study published by the National Highway Traffic Safety Administration [5].
Recent research works have already seen the potential of mobile phone to enhance the safety of people. Several works focus on the safety of drivers: to forbid a driver to use the mobile phone while driving, regulation is not always effective, how to recognize the phone belongs to the driver, and blocking the incoming call and message is a research challenge. Reference [6] use car speakers, which are fixed, to periodically send beacon messages to detect the location of every mobile phone inside the car; therefore, the one that belongs to the driver is recognized. To infer whether a user enters the vehicle from left side or right side of the vehicle or sits in front or rear seats [7] exploit the unique patterns mined from the rich dynamics in the accelerometer and magnetometer data observed during respective actions and make cognitive decision based on machine learning techniques. Some research works tend to protect elderly people from certain accidents [8][9][10][11][12], like accidental falls. Reference [13] use accelerometer to constantly monitor the fall event, and when such event is triggered, the event message will be routed [14] to the nearby cameras which can be added to assist to find and locate the fallen elder; [15] designs type-2 fuzzy models [16][17][18][19][20] that utilize smart phone triaxial accelerometer signals to detect fall incidents and identify abnormal gaits among elderly. Once a fall incident is detected, an alarm is sent to notify the medical staff for taking any necessary treatment. However, only a few researches try to enhance pedestrian safety [21][22][23] in the perspective of pedestrian itself through carry-on mobile phones. Reference [24] points out that mobile phone conversations distract users, presenting a significant impact to pedestrian safety. It uses the back camera of the mobile phone to detect vehicles approaching the user while he is making a phone call, alerting the user of a potentially unsafe situation.
In this paper, we introduce and evaluate HeadsUp, a system that warns pedestrian and lock the screen when one looks at the mobile phone while walking. Different from existing research works which warn users of upcoming rushing cars only while they are making phone calls, HeadsUp detects the behaviours of mobile phone user. Two key behaviours are Walking and Watching. HeadsUp only gives the mobile phone user a warning when a combination of behaviours of walking and watching occurs. Mobile phones are equipped with various of sensors, like accelerometer, gyroscope, magnetometer, and so forth [25], and the sensors provide the ability of sensing the physical world; therefore, the mobile phone has the possibility to sense the behaviour of user. Although some issues arise from sensing with mobile phones, such as noise filtering, a number of research works underpin the design of HeadsUp. Reference [26] gives a detailed understanding of how well pedometry works when applied to mobile phones and evaluates common walk detection and step counting algorithms applied to mobile phone sensor data. However, the design of HeadsUp still faces several challenges. (1) Unlike most research works that keep wearable devices and mobile phones fixed, we assume that users will place their mobile phones randomly.
(2) A mobile phone could be in different space for different usage, it could be in a pocket for carrying, could be held in hand for watching, could be lifted for calling. It's a challenge to judge which space the mobile phone is in and tell if the user is watching the phone.
We solve these challenges by taking advantage of gravity. The data generated by accelerometer embedded in a mobile phone is three-dimensional; take android system for example, see Figure 1. Due to the noise and accuracy of device, the movement of the mobile phone can be approximately depicted, meanwhile constant gravity will also impact the sensing of accelerometer. In a common sense, gravity will be treated as noise and be cancelled; however, it could be a hint to discover the state of mobile phone. For instance, a mobile phone is placed as Figure 1 shows; the reading of the accelerometer is " -axis: −0.22984336; -axis: 9.423578; -axis: 1.7238252", the -axis reading reflects gravity; hence, the placement of mobile phone can be deduced via gravity readings.
We note that HeadsUp needs not to keep the mobile phone fixed which is in accord with users' daily usage habits, and it will not block any incoming calls and messages; it only give warning and locks the screen as the user is walking and watching the phone, which keeps the inconvenient experience to the minimum. Our contributions may be summarized as follows.
(i) We design HeadsUp, a pedestrian phone addicts warning system to prevent phubbings from dangers. Mobile phone sensors, including accelerometer and gyroscope, are used to detect the gait pattern.
(ii) We identified the possibility to detect users' gait pattern without keeping the mobile phone fixed. It makes HeadsUp practical.
(iii) We implement HeadsUp on Android platform.
The rest of the paper is organized as follows. Section 2 presents a high level system overview, followed by the challenges and solutions in Section 3. The system evaluation is presented in Section 4, while summary is given in Section 5.

System Overview
HeadsUp periodically measures the accelerometer and gyroscope readings from user-carried mobile phones. However, gait recognition is somehow a computational work, and mobile phone battery power is very limited. As Figure 2 shows, HeadsUp divides detection mode into two stages: (1) phone usage detection; (2) gait recognition. Only when a mobile phone is being held and watched, the gait recognition will be triggered, which makes HeadsUp more energy efficient.
HeadsUp starts with a Sampling stage, both accelerometer and gyroscope will be constantly read to get enough sample data. The data will be sent to a module called Usage Detection; this module will determine of which state the mobile phone is. There are three states to be determined as Figure 3 shows: Pocket state as Figure 3(a) shows is a random state of mobile phone, it could be rolling, could be lying flat, could be in an erect position. Watching as Figure 3(b) shows is that the mobile phone may be held by the user or lying flat on a table, and the user is using it which also means the user is watching it. Calling is that the mobile phone is lifted for calling and it is in an erect position as Figure 3(c) shows. When Watching state is detected, the system will progress to Gait Recognition. In this stage, there are two steps. The first step is to determine and transform the coordinate system, because the mobile phone could be in any position, a universe coordinate system is important for gait recognition. The second step is walk pattern recognition, the sampled data will be compared with a walking pattern data set to recognize the gait style of the user.
After Gait Recognition, the pedestrian phone addict will receive a warning and the mobile phone screen will be locked. The user could choose to stop walking for a few seconds to unlock the screen or abruptly abort HeadsUp.

Challenges and System Design
The design of HeadsUp in real environments presents practical challenges. This section characterizes the nature of these challenges and presents our approaches in addressing them. shows; Calling state is always vertical or a few degrees off vertical as Figure 4(c) shows. Distinguishing Watching from Calling is simple. Using gravity as a reference, it is easy to determine the two states through coordination system readings. However, it would be a little difficult to differentiate Watching from Pocket. As Pocket could be in any position, it may be placed horizontally which is the same position as Watching. We noticed that when a user is using and watching the mobile phone, the screen will be tapped and touched intermittently; although these actions maybe subtle to other actions like walking, gyroscope is sensitive enough to capture these slight actions, also it is observed that the gyroscope is not affected by the walking movement, mainly because there is no spin or rotation of mobile phone during the walking.

Watching versus Calling.
The direction of gravity is constant, we can determine the state of Watching and Calling by accelerometer readings. Suppose the reading of the accelerometer is a vector, which is defined as = ( , , ), where is the reading of axis , is the reading of axis , and is the reading of axis . The reference vector, which refers to an ideal reading of the accelerometer when the mobile phone is placed perfectly horizontal, is defined as = (0, 0, ). Normally, equals 9.8 m/s 2 ; however, it varies in different areas. To eliminate the variance of , we normalize as = /| | = (0, 0, 1), as = /| |. Now we give the equation for determining the Walking state: In (1), equals the inner product of and , and it refers to how close the mobile phone approaches horizontal position, the smaller the closer. If < , where is an adjustable parameter used as a threshold to indicate the state of Watching, we set the current state of the mobile phone as Watching. In HeadsUp, we usually set to 45 ∘ .

Watching versus Pocket.
If the mobile phone is occasionally placed horizontally in a pocket or bag, it may be confused as in the Watching state in HeadsUp. A gyroscope is a device for measuring or maintaining orientation, based on the principles of angular momentum [27]. The gyroscope is very sensitive that it can capture subtle movements which refer to rotation and spin. When a user is using the mobile phone, the user may tap the screen to open a web page, slide the screen to unlock it, or scroll a news page. Although these actions are very subtle, they can still be captured by the gyroscope. Also, the movement of walking involves few or very slow spin or rotation action, it hardly affects the reading of gyroscope. Therefore, we can use gyroscope to distinguish Watching from Pocket. Figure 5 gives readings of gyroscope in two situations: one is that a user held the mobile phone horizontally and kept walking; the other one is similar except that the user was using the mobile phone. We can see from Figures 5(a) to 5(c) that the readings of gyroscope are hardly fluctuating due to are the component readings of coordination system of gyroscope, and is a threshold which indicates the Watching state. (

Watching versus
to distinguish Walking from Calling; (1) needs to be expanded to matrix operations as (3) shows: where = ( ) . To distinguish Walking from Pocket, the underlying condition needs to be satisfied: Therefore, we can give (5) to determine whether the mobile phone is in a Watching state:

Coordinate System Transformation.
For the convenience of Pattern Recognition, we need to make coordinate transformation to a universal coordinate system. We choose an ideal situation where a mobile phone is placed horizontally and the Earth's acceleration due to gravity equals the standard one which is 9.8 m/s 2 , and we set the coordinate system as the universal one. As shown in Figure 6, the coordinate system of the mobile phone needs to rotate to fit the universal one. Moreover, there exists measurement errors in every mobile phone which needs to be eliminated. We assume the coordinate system transformation is in a affine space; then, we get the underlying equation as follows: where is the base vectors of the universal coordinate system; it is equal to [ ]; is the ideal reading of accelerometer in the universal coordinate system; is the base vectors of the mobile phone coordinate system which is unknown; is the reading of accelerometer which equals to ; is the measure error of the mobile phone. Now we give a theorem for coordinate system transformation when the user is standing still.
because lim → 0 + + = 0 then = because and are base vectors, they are orthogonal matrixes, then = = : For Walking Pattern Recognition, the component 3 of is enough; however, when the user starts to move, 1 and 2 will no longer be zero which makes Theorem 1 not fit. From the proof we know that | | = | | and | | 2 = 2 1 + 2 2 + 2 3 ; then, | 3 | = √| | 2 − 2 1 − 2 2 . What we need to get is the sum of 2 1 + 2 2 . We all know the underlying famous equation which reveals the relation between distance and acceleration, where is distance traveled in a certain amount of time ( ), V 0 is starting velocity, is acceleration, and is time: If we can get the value of and V 0 , we can get the value of 2 . Now that the accelerometer is not available for measuring, we turned to another common sensor, GPS. GPS gives the exact location of the mobile phone; suppose the sampling interval is , then we can get the value of by = +1 − and get the value of V 0 by V 0 = Δ /Δ . In the universe coordinate system, 2 equals 2 1 + 2 2 ; then, we have (10) for coordinate transformation: Figure 7 gives the readings of accelerometer in five typical situations, including walking, running, jumping, sitting, and riding. We can observe that the fluctuation of readings of each axis component is different; axis-is dominant. To verify the observation, we depict all the sampled data on a 3D figure as Figure 8 shows. It is obvious that the data points expand along axis-. PCA (Principal Component Analysis) is a statistical procedure that uses orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components [28]. Using PCA, we can get the principal component of the data set. Suppose the data set is , the underlying equation is how to get eigenvectors of the covariance of ; then, the first eigenvector of coefficient is the principal component which is (−0.0613, 0.1338, 0.9891). Therefore, we choose axis-as the feature to perform Walking Pattern Recognition:

Walking Pattern Recognition.
Now that we leave only one feature, we can then use a simple and effective way to perform the walking pattern recognition. In signal processing, cross-correlation is a measure of similarity of two waveforms as a function of a timelag applied to one of them. This is also known as a sliding dot product or sliding inner-product. It is commonly used for searching a long signal for a shorter, known feature [29]. We establish a data set which contains several readings of accelerometer related to different users while they are walking. Once the Walking Pattern Recognition module is triggered, the system will compare current readings of accelerometer with each record in the data set using cross correlation.
Suppose the known symbol pattern of walking is of length , ( ) is the complex number representing the th received symbol, then the cross-correlation at a shift position is as (12) shows: (12) * is the complex conjugate of and ( , , ) is the correlation coefficient. Once the [ ] is aligned with , there will be a sudden spike in the correlation as Figure 9 shows.
Once ( , , ) > , where is a threshold which indicates the walking pattern, we assume the user is in a walking state.

Evaluation
We implement HeadsUp on an Android platform and it is compatible with 2.2 and up versions of Android operation system. The test devices include 20 mobile phones with android 4.2, such as HTC one, Sumsung S4, and Lenovo S2. All the devices are embedded with accelerometer, gyroscope, and GPS. There are 20 users involved in the test, each one is in charge of one mobile phone. The test performs in four different scenes: (1) walking and watching; (2) walking and not watching; (3) sitting and watching; (4) riding and watching. Each scene tests for 100 times in random places.

Scene 1: Walking and Watching.
Twenty users randomly choose walking paths and walk in a usual way with the intermittently stop; each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure 10 shows.

Scene 2: Walking and Not
Watching. Twenty users randomly choose walking paths and walk in an usual way with intermittently holding up the mobile phone and watching; each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure 11 shows. x-axis readings of running situation 10

Scene 3: Sitting and
x-axis readings of jumping situation 5 x-axis readings of sitting situation 5 x-axis readings of bus riding situation   it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure 12 shows.

Scene 4: Riding and
Watching. Twenty users randomly choose a riding tool, including bus, taxi, and subway, with intermittently leaving the car or train and walking away while using the mobile phone. Each single test lasts 30 seconds and it repeats for 100 times. Through the result, we get the False Positive and False Negative rate as Figure 13 shows. Through the test results from four-scene test, we notice that the false positive rate is lower that 2% on average, which shows that HeadsUp seldom interferes users' normal usage of the mobile phone. The false negative rate is around 3% on average which guarantees that HeadsUp will duly warn and keep the pedestrian phone addicts from watching the screen then make them stop and use the mobile phone, therefore, reducing the occurrence rate of potential dangers.

Summary
In this paper, we design and implement HeadsUp, a practical system that warns pedestrian and locks the screen when one looks at the mobile phone while walking. Evaluation results show that, on average, the false negative rate is less than 3%, which guarantees that HeadsUp will duly warn and keep the pedestrian phone addicts from watching the screen then make them stop and use the mobile phone, therefore, reducing the occurrence rate of potential dangers.