Sleep monitor: A tool for monitoring and categorical scoring of lying position using 3D camera data

Wepresent a software package for analysing body positions of a subjectwhen they are lying or sleeping in their bed. The software is designed to interface to inexpensive sensors, such as the Microsoft Kinect, and is thus suitable for monitoring at the subjects own home, rather than a dedicated sleep lab. The system is invariant to bed clothing and levels of ambient lighting. Analysis time for a single night session is under five minutes, a significant improvement over the 30–60 min analysis time reported in the literature. © 2018 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).


a b s t r a c t
We present a software package for analysing body positions of a subject when they are lying or sleeping in their bed. The software is designed to interface to inexpensive sensors, such as the Microsoft Kinect, and is thus suitable for monitoring at the subjects own home, rather than a dedicated sleep lab. The system is invariant to bed clothing and levels of ambient lighting. Analysis time for a single night session is under five minutes, a significant improvement over the 30-60 min analysis time reported in the literature.  so as to answer a number of health-related questions. The lying positions of a person in bed and the duration of these positions have been associated with various medical conditions. For example, sleep apnea is more likely to present itself when lying supine (on the back) [1]. Lying prone (on the stomach) on the other hand has been associated with sudden unexpected death in epilepsy [2] and sudden infant death syndrome [3]. Lying in the same position for an extended time increases the risk of developing pressure ulcers, a significant problem for people with impaired mobility, with guidelines recommending a change in position at least every two hours [4]. Links between lying position and other health conditions have also been postulated, with researchers seeking to discover a connection between lying position and shoulder pain [5], function and development of the brain [6] and quality of sleep [7]. The concept of 24-hour management of body posture is a key concept in occupational therapy in order to prevent body deformities [8], however one research gap is the lack of evidence for night-time positioning equipment [9].

Problems and background
A barrier to research on the effects of lying position is the intrusiveness, inconvenience and expense of clinic-based monitoring. A polysomnogram is typically acquired by having the subject attend a sleep clinic for a number of nights, where they are monitored over each night using a variety of attached sensors. This introduces a number of differences to a subject's normal environment (for example, unfamiliar location, different bed mattress, intrusivity of attached sensors), which may affect their lying position [10].
Several home-based solutions for monitoring lying position are found in the literature or as commercial products. Many of these rely on sensors being attached to the body, which, due to their intrusive nature, may affect sleep or are targeted towards a specific purpose. For example, the NightBalance (NightBalance B.V., The Hague, The Netherlands) is a torso mounted device targeted at the prevention of sleep apnea by vibrating when the wearer is in the supine position. It also tracks and provides graphs for sleep position duration, however the configurability of the tracked positions is limited and visual verification of the body lying position is not provided as part of the output.
The least intrusive monitoring systems rely on cameras and image interpretation rather than sensors physically attached to the subject's body. Some of these offer fully automated recording and analysis [11,12]. Fully automated systems however suffer from a number of weaknesses. Firstly, automated algorithms are easily affected by occlusion that can arise from a blanket. More importantly, they need to be trained and tuned to the task at hand. For a particular approach to be re-used with a different lying position classification protocol, the algorithm would have to be retrained. This is a non-trivial task, as typically a large training data set is required. In [12] for example, the authors used data from 78 subjects, recorded over a period of six months. Thus, current fully automated approaches do not offer a 'ready to use' tool for exploring arbitrary lying positions, apart from those they have been trained to detect.
A more generic lying position analysis tool can be created through a semi-automated approach, where computer-based processing of the data is used to support a human expert, rather than to replace them, however there is little published work in this area. An example of such a system was developed by Cary et al. [13] in order to investigate the relationship between sleep lying position and morning spinal symptoms. They developed a protocol utilising two infrared cameras coupled with a commercial, securityoriented, digital video recorder. One camera provided a continuous recording, whilst the other was triggered by motion. The article does not specifically state whether blankets were present during the recording, though some type of covering is shown in the figures. Using the recording from the motion detection camera, a subsequent manual categorisation into four postures was reported to take 30 min, as opposed to 60 min when using the continuous camera recording. One limitation of systems that rely on commercial motion detection hardware is the limited flexibility and reporting that these systems offer, as they are typically designed to detect rather than categorise motion, leading to extensive time having to be spent on manually categorising the resulting data.
In this paper we describe a custom developed program designed for the inexpensive, unobtrusive monitoring and analysis of lying position of a subject in their own bed. Our system is designed to use a depth camera sensor, such as the Microsoft Kinect (Microsoft Corporation, Seattle WA), Intel R ⃝ RealSense TM (Intel Corporation, Santa Clara CA) or Orbbec Astra (Orbbec 3D Technology International Inc. Troy MI). In our evaluation we used the Microsoft Kinect camera version 2 sensor, utilising both infrared and depth images, making recording in a completely dark environment possible. Though the Kinect was released primarily to act as an input controller for games on the Microsoft Xbox platform, it has found a number of non-game applications. Existing literature suggest this sensor is suitable for detecting movement with high precision, for example Silverstein et al. [14] found that the Kinect version 2 camera can be used to track respiratory motion comparably to existing commercial hardware. Using depth in addition to infrared images provides more sensitivity in detecting movement when the subject is occluded by a covering. When a subject moves under a covering the pixel intensities in an infrared image show minimal change as they depend on the optical properties of the covering. In the depth image however, where pixel intensities correspond to distance from the camera, changes are easily observed, with reported accuracy of under 2 mm [15]. The analysis component of the system allows the analyst to set a movement threshold which then automatically partitions the night into segments spent in a particular position. The analyst is then able to manually adjust the segmentation, classify the segments into particular postures and add notes. The analysis system produces a report with statistics on durations of various lying position calculated.
Human subjects who appear in the figures contained in this paper and whose data has been used to generate the graphs have given their written consent to the inclusion of this material in the paper.

Software architecture
There are two parts to the overall system, provided as separate executable programs for monitoring and analysis. The monitoring component program connects to a Microsoft Kinect V2 camera and records a subject's body positions in their normal bed environment. The monitoring program is designed for minimal interaction and produces a folder of captured images. The analysis program allows for the interactive categorisation of the captured images to determine the length of time the subject spent in each position. The software is divided into these two programs as typically the person being monitored will not be doing the analysis, rather a researcher or assistant will be using the analysis program on a separate computer to the one in the field gathering data to process many runs of the logging program. A further advantage to separating the monitoring and analysis program is to make the analysis program independent of the monitoring hardware used. The role of interfacing with the monitoring hardware (Kinect or otherwise) and producing data in a format suitable for the analysis software is the responsibility of the monitoring program. Therefore, the analysis program can be used on data produced by other sensors without code recompilation, as long as a version of the logging program is written for the new sensor.

Monitoring program
The monitoring program has an interface that shows the current view from the camera and, for maximum simplicity to avoid user error, contains a single button to initiate recording. The setup procedure consists of mounting the camera on a secure stand with a good view of the bed. Details and illustrations of the monitoring setup are provided in the monitoring program manual. The monitoring software saves images from the colour, infrared and depth cameras at intervals of seven seconds. This interval was determined to achieve a balance in minimising storage space required whilst still being able to provide insight into movements that take place. For each hour of monitoring time, approximately 250MB of storage space is required. For convenience, the software creates a new folder for each recording.
Out of the three types of images, only the infrared and depth images are useable when lights are turned off. We still store the colour images for verification on how much light was actually in the room. The Kinect provides a source of infrared light, allowing the infrared images to be captured in a dark environment. The depth camera produces an image where the brightness of each pixel represents the distance of that object to the camera, objects closer to the camera appear brighter. The file for each screenshot follows a defined naming convention which is then interpreted by the position analysis program when loading data. The file name is in the format of:

Position analysis program
The analysis program takes as its input a folder of images captured during a monitoring session by the monitoring component software. The analysis software allows for the image sequence to be viewed as a video with variable speed and provides a timeline, indicating the amount of movement that occurred at each point in the monitoring session. A screen capture of the analysis software is shown in Fig. 1.
The movement indication graph is constructed from the depth camera image sequence. The y-coordinate of a point at a particular x-coordinate time instance is derived by first taking the absolute differences of pixel intensities between two captured images, one at that particular time instance and the other from the previous instance. These absolute pixel differences are summed to produce a raw movement value for that time instance. Once raw movement values for the entire monitoring session have been produced, they are processed for display. This processing involves shifting the series of values so as to put the median at 0, then truncating between zero and four times the standard deviation of the series. For display purposes, the values are then scaled to a range from 0 to 100. For periods where there has been no movement, the movement indication graph shows values close to zero. In situations such sleep or prolonged bed rest, where the subject is usually not moving, any changes of position show as distinct peaks in the movement graph and are used to draw the analyst's attention to these movement changes. The height of the peaks corresponds to the amount of movement.
The analyst can examine points along the timeline by moving the 'current time' indicator to get an understanding of what movements were occurring. The horizontal red threshold line can be positioned to segment the graph. In addition further segment boundaries can be added manually, or existing ones deleted.
Each segment can be classified by the analyst. As the actual classification coding scheme used will vary by the hypothesis being investigated, a generic set of 16 numerical categories are provided by the software with the analyst using a sub-set based on the number of categories they are interested in. For example, in the case of monitoring a person who relies on external supports during sleep, a two-class system of (1) supported and (2) unsupported might be used to determine the percentage of their sleep during which these external supports are effective. For a standard lying position classification, a four-class system of (1) supine, (2) prone, (3) left and (4) right may be used (or any other numerical coding of these). To classify a particular segment, the analyst selects anywhere within the segment and then selects the class number they wish to assign to it. Once body position segments have been assigned classes, a report summarising the monitoring session is produced in comma separated value (csv) format. An extract from such a report is shown in Fig. 2. This indicates, in the top left corner that the total monitoring time was six hours, seventeen minutes and eleven seconds, with 2754 sets of image samples taken during that time. Following this is a breakdown of the monitoring session, with one row per body position segment. Each of these rows lists the image number that starts the sequence, the position classifier number assigned by the analyst, the duration of the segment, and the time that the segment starts in terms of time of day and time from the start of the monitoring session. For the example given in Fig. 2 the subject starts the session (segment 1) in position 3 and stays in this position for 59.4 min.
Below the monitoring session breakdown is a set of summary data for the positions. This indicates for the example that the subject spent 52.8% of the night in position 3 (lying on their left side), 37.2% of the night in position 4 (lying on their right side) and 10% lying in position 1 (supine). The total duration of each of these is given in minutes, along with the number of segments where the subject was in each of these positions.

Implementation and empirical results
The software was developed in C# under the Microsoft . Net framework using the Microsoft Kinect V2 SDK. The ability of the analysis software to detect movement was tested using a variety of bed clothing types being used to cover the subject: onesie, light blanket and a heavier blanket. Extracts of movement graphs for the three types of covering (a-c) are shown in Fig. 3(a-c). Superimposed on each graph is an indication of body position segments where the subject is in one of the four positions (prone, supine, on left side, on right side). It can be seen from the sharp distinct peaks in these graphs that major body movements are detected even under the heavier blanket.
To investigate the impact of using the software on analysis time, four monitoring sessions were conducted with a subject sleeping overnight. The monitoring computer hardware consisted of an Asus N3530 laptop with 4 GB of RAM, 500 GB hard disk, running Microsoft Windows 8.1. The analysis of each of these monitoring sessions into the four basic lying positions (supine, prone, on left side, on right side) using the software was timed. The protocol for this timing was to start the analysis software and then start the timer, with the timer stopped after the analysis report for that monitoring session was saved. Thus, the time taken for analysis includes: • Time to navigate to and open the monitoring session directory using the analysis software.
• Time for the captured images to be read in and the movement graph to be generated.
• The time to do the analysis, segmenting the session and giving each segment a class number. This includes setting the threshold, performing the automatic segmentation, then inserting and deleting segment boundaries so that only boundaries corresponding to changes in lying position remain.
• Time for saving the report.
Analysis data for the four position monitoring sessions are shown in Table 1. Each of the four sessions was analysed in a time between four and five minutes. This is a substantially lower time than reported for the system described in [13], which lists an analysis time of 60 min for analysing from raw video footage and 30 min when a motion-detection triggered recording was used.

Illustrative example
To give an indication of how peak height corresponds to movement amount, Fig. 4 shows two adjacent images that correspond to a position change that would result in a major peak, such as the ones used to partition the sessions in Fig. 3. For comparison, Fig. 5 shows a slight movement, examples of which cause the less significant peaks found in the graphs of Fig. 3.

Impact
The software has been in active use since 2015 at Disability Services, a Western Australian government agency with one of its aims being to improve quality of life for people with disability. There it is used to analyse the sleeping position patterns of clients with movement restrictions, such as those affected by cerebral palsy. The analysis provides insight into the effectiveness of physical sleep support systems, and used to investigate hypotheses regarding the source of ailments such as body pains.

Conclusion and future work
The analysis of body lying position can be insightful in a number of applications, such as postural care and sleep apnea. However, current means of obtaining and analysing the data are either expensive, inconvenient or tuned for a single particular application. Our software offers an open-source solution that minimises the intrusiveness to the subject and can be used to differentiate a configurable number of body lying positions. The software aids the analyst to identify changes during a monitoring session, allowing them to rapidly analyse, interpret and report on the data. The software is generic, providing 16 numeric class labels for the analyst. Unlike most existing solutions, the system is capable of detecting changes in lying position with various bed clothing in complete Future work can include building support for a larger range of camera technologies, as well as enhancement to aid in the answering of specific research questions related to the medical conditions highlighted in the introduction, such as shoulder pain, sleep quality and brain development. Though the software has been developed specifically with lying position during sleep in mind, it can also find application in other areas where categorical scoring of scene changes is desired. These may include the analysis of behaviour patterns of individuals, for example to monitor activity when regaining mobility after recovering from surgery. Other applications could be in the monitoring of physical spaces, such as shopping centres and carparks where different activity levels could be identified. Some modification may be required to the program to fit these other applications.

Declaration of Interest
The authors report no conflict of interest.