Keywords

1 Introduction

Smartwatches are well suited for monitoring health and/or security hazards for elderly people. They can support a self-determined and safe life in the familiar home until the very high age. Software assistance apps running on standard forefront smartwatches (e.g. LG Urbane 2™, LG Sport™, Samsung Gear S3 Frontier™) provide much more functionality far beyond the capabilities of traditional emergency push buttons [1]. The traditional solution typically relies on the wearer of the device to initiate an alert in case of an emergency. The smartwatch assistance app additionally will continuously monitor the activity patterns and vital parameters of the smartwatch wearer. The app will recognize from the smartwatch sensor signals the events of daily living (EDLs, like tumbles, leaving or returning at home), the excessive duration of activities of daily livingFootnote 1 (ADLs like naps, bedtimes, outdoor walks), substantial deviations of the usual daily routine (indicating potential illness, depressions or mental disorder) and abnormal heart rates (indicating bradycardia, tachycardia) [2].

The app will conclude the presence of health and/or security hazards based on recognized EDLs, ADLs and their sequencing in the course of time. The smartwatch app then will autonomously decide when and how to inform about the hazardous situation. Pre-alerting the smartwatch wearer, who is the user of the health assistance app, is always the first step of handling the hazard. Our chosen smartwatch devices include an integrated 3G/LTE cellular phone. The essence of our hazard handling includes transmitting data about the acute hazards to distant family members and/or a home emergency call center (e.g. current geographic location of the user, vital data). If situationally appropriate and agreed with the user, a speech connection will be established additionally via the 3G/LTE phone to clarify the situation on the spot.

By utilizing 3G/LTE communication, the smartwatch assistance app does not depend on any susceptible infrastructure like Bluetooth or Wi-Fi connections. The assistance app can transmit data and establish a speech connection from practically anywhere. Such smartwatch solutions constitute the next generation of home emergency call systems – extending also the reach of their services beyond the boundaries of the familiar home.

But, the plenitude of health hazard monitoring tasks performed by the assistance app and the very small display of the smartwatch (about 1.4’’ diameter for today’s smartwatches) raise substantial challenges on how the interaction between the app and the smartwatch wearer can be designed cognitive efficiently and at eye’s level. The basic idea behind the smartwatch assistance app is an aid for realizing a self-determined and safe life of the smartwatch wearer. It must not end up as a surveillance agent jeopardizing the privacy or dismantling the decision-making ability of the user. In the following we describe a model based dialog control approach and a supporting application framework for achieving this principal task.

2 Requirements Analysis

The bundle of requirements for dialogue design between the smartwatch assistance app and the smartwatch wearer, user, can be differentiated in human smartwatch interaction as well as software engineering aspects.

2.1 Human Smartwatch Interaction

The interaction with the user is based on the core privacy by design principles [4, 5]: The user should be always kept in control of interaction and data transmission as long as he conscious and remains capable to act. Very tangible privacy relevant data about his location, activity patterns, vital parameters, … shall only be exchanged based on a prior approval of the user. This requirement will be fulfilled via a “pre-alert”, by which the reasons for the intended data transmission will be explained to the user. A conscious user still capable to act shall be empowered to stop the intended data transmission at any time. Otherwise, in a real hazardous situation where those preconditions on the user will not be fulfilled, it is important that the data transmission resp. the alerting really takes place automatically without the necessity of a manual intervention of the user.

The situation is further complicated by the fact that, if the health assistance app has concluded the presence of several hazards, these must be jointly handled in one combined dialogue. Only one speech connection to distant family members or a call center has to handle all pending matters, in order to enable a potential external reaction/intervention as fast as possible.

The interaction between the user, the smartwatch app and potentially distant family members or the call center typically includes the following six consecutive steps (ids for these steps in […] brackets for later reference):

  1. 1.

    [Inform] informing the user about the concluded acute hazards (“pre-alerting”) and explaining, why the assistance app has concluded (and is assuming) the presence of the hazardous situation,

  2. 2.

    [Propose Decision] asking the user for a (unilateral or bilateral) decision: if for example distant family member resp. the call center shall be called (alerted) or if everything is OK (and no data transmission and speech connection shall take place),

  3. 3.

    [Wait] waiting for a reaction from the user. Within this period (about 30 s) the user needs to perceive the information presented to him and to make up his mind about the appropriate reaction in the current situation,

  4. 4.

    [Confirm] confirming the immediate consequences of a decision taken the user resp. indicating the immediate consequences of an omitted decision, if not obvious,

  5. 5.

    [Transmit] transmitting relevant location and health data to distant family members resp. a call center, if this is situationally appropriate and the user has indicated his consent with this step or does not respond at all in the assumed hazardous situation.

  6. 6.

    [Clarify] placing a clarification call, if this is situationally appropriate and the user has indicated his consent with this step or does not respond at all in the assumed hazardous situation. The prior transmitted data will be typically presented to the responding person and will help to understand the user’s momentary situation on the spot.

The described interaction sequence constitutes a fixed dialogue pattern for our application domain. Patterns are a well-established approach in software engineering (see [6], Sects. 6.3, 6.4) esp. in the creation of reusable and adaptable software, solving a given recurring problem and presenting a solution for it. We name this new pattern “critical dialogue section” (CDS). Based on [6] a CDS can be classified as “behavioral pattern”.

A “critical dialogue section” (CDS) must not be interrupted by other activities, dialogues and messages in order not to mentally overload the user. For example, incoming phone calls or potentials pre-alerts about additional concluded hazards shortly afterwards need to excluded during the execution of the CDS. In congruence with the well-known operating system concept of a “critical section” (see [7], Sect. 6.2), also the CDS shall be either executed completely or not at all and only by a single process. There are in fact situations when the section should not be executed at all, for example for safety reasons when the user is driving at high speed on a highway. Or, if the smartwatch is not worn at that moment. This situation is also the one exception to the mandatory complete execution of the CDS. If the wearer decides to drop the smartwatch during the execution of the section, which will be signalized by the heart rate sensor, its continuation obviously does not make sense.

The sketched inherent complexity of a smartwatch dialogue design and implementation quests for a systematic model and supporting framework for handling this complexity. Based on our experience, this is one of the most crucial productivity factors for the realization of comprehensive smartwatch apps.

2.2 Software Engineering Factors

The knowledge about an appropriate handling of health and/or security hazards is empirical, volatile and based on best practices. This knowledge will grow and must be optimized daily. Any implementation must take this into account. We are still dealing with the first generation of programmable wearable devices with a complete new spectrum of sensors, interpretation and interaction possibilities. Therefore, a declarative representation of the interaction with the smartwatch user offers decisive advantages with respect to maintainability and changeability of the software [6].

Another software engineering aspect aims at the separation of the health resp. security hazard conclusion and handling knowledge into separate knowledge chunks. Although the monitoring of all hazards must be executed by the smartwatch app simultaneously, concluding the presence of independent hazards and describing their handling shall be represented separately in different modules. This alleviates the comprehensibility and maintainability of the software (»divide et impera« principle). Each module shares as a common secret [8] the specific knowledge for best practice reasoning about related hazards and their appropriate handling. For example, Fig. 3 describes the combined hazard handling of four related health and security hazards: leaving_home, reentering_home, a runaway_situation or excessive_absence_from_home, which might occur during monitoring the absence of a user from his home. Handling these hazards is completely independent from handling the health hazard of insufficient_drinking (fluid ingestion) or the health hazard handling for abnormal_heart_rates and thus these latter hazard handlings should be represented independently.

2.3 Aspects of Practice

Today, there is no single, universal technology known for the detection of health and/or security hazards. Many sudden EDLs (e.g. tumbles) can be recognized from the motion sensors of the smartwatch via artificial neural networks (ANNs) with a very high precision [11]. This is also true for short term activities like drinking, combing or tooth brushing. Other activities, which have a longer duration (e.g. falling asleep for a nap) can best be detected by monitoring the user’s behavior via a finite state automaton. The progress in concluding such activities will be modelled by state transitions of the automaton following the different stages of the recognition process. The state transitions of the automaton will be triggered by EDLs, other ADLs or timing constraints. For example, the health hazard of insufficient_drinking (fluid ingestion) can be concluded, if no drinking ADLs will be observed over a longer time. Figure 2 describes a simple two state automaton for handling this health hazard. Each drinking ADL recognized by an underlying ANN will reset the timer controlling the state transition of the automaton.

3 System Architecture

The application software architecture [6] of our smartwatch solution is based on a hierarchical, multi-layer structure [9, 10], where – by definition – an upper layer uses functionality from the lower layers. The structure is depicted by the UML component diagram in Fig. 1.

Fig. 1.
figure 1

Hierarchical structure of the smartwatch assistance app as UML component diagram

Full size image
Fig. 2.
figure 2

UML finite state machine for concluding the health hazard insufficient_drinking

Full size image

The bottom layer of our software architecture consists of two four components: the motion analysis, location analysis, the WAN communication control and the local interaction control component. The motion analysis component contains an artificial neural network (ANN). It condenses 39 statistical parameters calculated from a sliding time windows of 10 s applied to the sensor signals of the smartwatch (accelerometer, gyroscopes, barometer, heart rate, …) into the set of recognized EDLs, ADLs (cf. [11] for details). The neural network combines a good training capability by samples sets of movement patterns with a good recognition rate for the EDLs, ADLs. (cf. [12, 13] for example for details of the recognition of ADL drinking, fluid ingestion). The location analysis component recognizes EDLs like leaving home or leaving an agreed vicinity of the home (geofencing) by analyzing and interpreting the WiFi and GPS sensor information. If the individual rooms of the home have been equipped with bluetooth beacons [14], or the varying home Wi-Fi signal is scrutinized for similarity with reference patterns [15], also an indoor localization will be possible. The WAN communication control component will be used for transmitting relevant data in the [transmit] step of CDS execution and for placing a 3G/LTE phone call in the [Clarify] step of CDS execution. Similar, the first four [Inform], [Propose Decision], [Wait] and [Confirm] steps of CDS execution use the local interaction control component for displaying their output on the smartwatch screen and getting to know the user’s reaction on the presented information.

The medium layer consists at its core of two components. First, a multitude of UML finite s tate m achines, USMs. Each USM describes the reasoning about a single or a group of related health and/or security hazards in a declarative way (see Fig. 3, Sect. 4 for details). Each USM typically contains a least one concluded hazard. These concluded hazards are distinguished states of the USMs which will be written or deleted from the blackboard, as soon as the USMs enters or leaves such states as their current state. All finite state machines are executed simultaneously, and the state transitions of the machines are triggered by the detected ADLs, EDLs and their occurrence, duration or absence in real time. The second component of this medium layer is the dialogue manager, which executes the specific CDSs. A CDS contains the dialogue tactical knowledge for handling a specific (group of) health or security hazard. The dialogue manager is controlled and used by the blackboard based, central scheduler on the top layer.

Fig. 3.
figure 3

UML finite state machine for concluding several health and security hazardsin the scope of a user’s absence from home

Full size image

The top layer consists of a blackboard [16] as central synchronization mechanism between the different finite state machines and their concluded hazards on the medium layer. The central, priority based scheduling mechanism selects one of those hazards currently on the blackboard for execution. The execution of the selected health or security hazard is delegated to the dialogue manager containing the specific CDSs. During the execution of the respective [Inform], … [Clarify] steps of the selected CDS the I/O and communication devices of the smartwatch are exclusively reserved and attached to the section. The scheduler uses the entries on the blackboard as its input data for the scheduling algorithm and passes the set of acute hazards to the dialogue manager for instantiation of the corresponding CDS pattern.

The functionality of the middle and top layer can best be compared to Fowler’s »model, passive view and presenter« pattern, elaborating from the well-known MVC pattern [17]. This pattern has decisive advantages with respect to testing the health assistance app.

4 Towards a Formal Dialogue Model

4.1 Health Hazard Conclusion and Handling via USMs

For concluding the presence of a health hazard from the sequencing of recognized EDL and ADLs occurring over the course of time, the domain specific knowledge is described by a UML finite state machine (USM) (see also [9]). A single USM models an individual hazard and its associated handling process including the execution of the CDS. It describes the domain handling knowledge in a declarative and maintainable manner. All defined USMs within the health assistance app will be executed/interpreted simultaneously.

The progress of the reasoning process about the hazard is modelled via different states of an USM, whereby the handling of a finally concluded hazard is represented by a CDS as a distinguished state of the machine. The state transitions of the USM depend (a) on its current state, and, (b) as “input”, on:

  • elapsed (real) time periods for which the USM is already in the current state,

  • the recognized EDLs, ADLs as momentary output of the location analysis or motion analysis component.

4.2 Synchronized Health Hazard Handling via a Blackboard

The synchronization of the different USMs is done via a backboard [16]. In the scope of a state transition from an old state sold to current state snew, a USM posts a new current state snew, which is of class «Hazard» , as a new entry on the blackboard. Posting a CDS on the blackboard, pragmatically represents the request for execution, dialogue handling, of this hazard. If, for any reason, an USM changes its current state snew of class «Hazard» to a different successor state, the specific hazard is erased from the blackboard, denoting a withdrawal of the prior execution request. Pragmatically for a moment in time, the set of all specific hazards on the blackboard at that time denote the set of all concluded and assumed, acute health and security hazards.

The blackboard is organized by different priority levels, so the priority attached to the health or security hazard by the corresponding USM state will determine on which blackboard level the hazard will be posted Table 1.

Table 1. Blackboard priority levels
Full size table

The decision about the requested execution and their prioritizing and sequencing will be done by the central scheduler of the blackboard.

4.3 Priority Based Scheduling

The central scheduling algorithm of the blackboard controls the dialogue with the user by selecting one of the specific hazards on the blackboard with highest priority for execution. The execution of the selected health or security hazard is delegated to the dialogue manager containing the specific CDS for this hazard. The selected CDS will be granted exclusive access to the smartwatch I/O and communication devices for the time period of their execution. Because the execution of the CDS typically may include a call, speech connection, to a distant family member or call center, this execution time may be of unforeseeable duration. During this interval, the CDS can present its explanation, confirmation, … texts on the smartwatch screen, use microphone and loudspeaker for the speech connection and can accept input from the mic, touchscreen, buttons or bezel of the smartwatch. The scheduling, i.e. selection of a posted hazard for execution, will be governed by the following scheduler rules:

  1. (a)

    Inhibition rule: only select a CDS when there are no inhibitory facts presents: (1) an ongoing execution on another CDS or (2) a user moving too fast.

  2. (b)

    Eligibility rule: only select a CDS with unique name p for execution, for which in the [Inform], [Confirm] steps for dynamic text configuration, a text is listed and associated with {p} or the exact set of all acute health hazards on the blackboard.

  3. (c)

    Highest priority rule: unambiguously select the specific CDS with a blackboard entry of highest priority, if applicable.

  4. (d)

    Most comprehensive selection rule: if there is more than one CDS with highest priority, select the specific CDS which handles (in its action blocks by the corresponding unique names) the exact set of all acute health hazards on the blackboard.

  5. (e)

    Most recently added rule: if rule c) still selects more than one CDS, select the most recent added CDS for execution.

4.4 The Execution of CDS for Dialogue Handling of Health and Security Hazards

Controlling the execution of health or security hazard dialogue handling by CDS combines two main advantages:

  • It supports a conceptually complete and systematic user dialogue, in which:

    1. a.

      The dialogue is performed on eye’s level and the user is always informed about the conclusions of the system. In the [Inform] step the user gets to know all concluded acute health hazards. A dynamic text configuration mechanism can produce different explanation texts tailored to the specific configuration of acute health hazards (see below). In the later [Confirm] steps the user gets to know the immediate consequences of his taken decision.

    2. b.

      in the [Propose Decision] step, choices are restricted to (unilateral) affirmation or (bilateral) YES/NO decisions. This restricts the cognitive complexity for the targeted user group of elderly citizens.

    3. c.

      all potential reactions of the user to the presented information – including an omitted reaction - will be considered in the [Confirm] step.

  • The implemented framework for the CDS behavioral pattern involves many comfort features:

    1. a.

      deferring the execution of a CDS – for safety reasons - if the user is moving (potentially steering a car) at high speed. This will be automatically determined from analyzing the location information by the scheduler (via GPS). Similar, a CDS execution will not start as long as the smartwatch is not worn.

    2. b.

      blocking incoming calls during the execution of the section.

    3. c.

      stopping/fading down a running MP3 player for the execution time of the CDS.

    4. d.

      blocking the disturbance of the section by overlapping starts of other CDSs or other alternative usage of the smartwatch I/O devices.

    5. e.

      catching the attention of the user by putting the assistance app in the foreground of the smartwatch display for a presented pre-alert and catching the attention of the user by haptic and acoustic signals.

    6. f.

      checking for a rising of the arm/wrist rotation in order to detect when the user starts to actively perceive the presented pre-alert information about the health hazard (and thus calculating a fair waiting time for the user’s reaction),

    7. g.

      aborting the CDS resp. cancelling a (scheduled or ongoing) call when the user drops the smartwatch during execution of the section.

The dialogue manager component will typically contain a set of different CDS stereotypes for the specific hazard classes.

In their [Inform] and [Confirm] action blocks these CDS stereotypes have rules for dynamic text configuration based on the set of acute health or security hazards on the blackboard. Those rules mention the set of hazards as preconditions, which must be present, when the specific text will be used for an explanation or confirmation text. Accordingly, different data may be transmitted in the [Transmit] phase of CDS execution, based on the set of acute hazards on the blackboard. This can be actively utilized for controlling privacy. For example, when the security hazard manual_call_request will be executed. Heart rate data will only be transmitted to a distant communication end point if a concluded hazard abnormal_vital_parameters is also present on the blackboard. The geographic location data of the user will only be transmitted, if an excessive_absence or runaway_situation is also present on the blackboard at this moment. If none of these hazards is present at the moment when the user requests the manual call, based on the configured rules no data will be transmitted just before the call.

In order to perform this dynamic text configuration resp. configuration of data transmission, the dialogue manager will instantiate the utilized CDS with the set of all present hazards on the blackboard. This set will be passed to the dialogue manager from the central scheduler, when the scheduler delegates the execution of the dialogue management for a selected hazard to the dialogue manager.

The hazards on the blackboard, which will be used in the preconditions of applied dynamic text configuration rules or data transmission rules will be denoted as “referenced hazards” by the execution of the CDS. Pragmatically, the execution of the CDS is assumed to jointly handle the dialogue for all “referenced hazards”. Therefore, the later completion of the CDS execution will be signaled by the dialogue manager component as well to the specific USM, which has posted the selected and executed hazard on the blackboard and to all the USMs of the “referenced hazards”. This is done by sending the event “CDS executed” to the USMs. This event will cause a corresponding state transition of those USMs. In the course of this state transition, the corresponding USMs will erase their handled hazards from the blackboard (Figs. 4 and 5).

Fig. 4.
figure 4

Behavioral pattern for Critical Dialogue Section (CDs) as UML activity diagram

Full size image
Fig. 5.
figure 5

Health Assistance App for Samsung Gear S™. Left: Default display when no health hazard has been detected. Middle: Result of [Inform], [Propose Decision] steps for tumble CDS (pre-alert). Right: Result of [Confirm] step for tumble CDS, if no user decision has been taken

Full size image

5 Discussion

A field test of the developed assistance app for the Samsung Gear S™ (cf. [18] for detailed results) showed that most flaws appear in the design and implementation of the dialogue with the user, and are not caused by missing or malfunctioning functional features. For example, we originally had not considered that a dialogue sequence should be postponed for safety reasons, when the user is driving at high speed on a highway. Or, that the start or continuation of a CDS does not make sense if the user drops the smartwatch and retires to bed. These aspects now have been integrated in the application framework and will be automatically checked and considered as a comfort feature of dialogue implementation for all future applications based on the framework.

More situational awareness for inhibiting CDS execution would be possible by additionally considering digital maps and mapping the current user position on relevant geographic objects (“map matching”). Thus, it could be identified whether the user is driving on a street or by railroad. Or, if the wearer is moving in a condensed town center or on a rural highway. In practice, the computational effort and the energy consumption for implementing these features must be accounted against the gain in comfort. The described functionality increase remains only of secondary importance as long as the topmost, mandatory goal of running the health assistance app over a full 18 h day with at least a 10% battery reserve for emergency calls is not fully achieved.

6 Conclusions

Battery capacity resp. energy consumption is the most scare and valuable resource for a smartwatch. Therefore, one might ask: Does the use of the model based application framework [16] for a formal dialogue management and the resulting computational overhead really pays off for such a device? Our – experience based – answer is yes, in that the application framework allows to focus on domain specific issues of the application and the application logic. The proposed model and its implementing framework relieves the software developer from most aspects the concise dialogue implementation. Moreover, the presented model ensures a consistency and completeness of the dialogue design and implementation for the application, which otherwise would be difficult to achieve and would bind substantial development resources.