1 Introduction

Every year 15 million people suffer a stroke, worldwide. From these, 5 million die and another 5 million are left with permanent disability, becoming a burden to the family, to the National Health Service and, in general, to the community [1]. In 2010, the absolute numbers of people with first stroke (16.9 million), stroke survivors (33 million), stroke-related deaths (5.9 million), and Disability-Adjusted Life Year (DALY) lost (102 million) were high and had significantly increased since 1990 (68 %, 84 %, 26 %, and 12 % increase, respectively), with most of the burden (68.6 % incident strokes, 52.2 % prevalent strokes, 70.9 % stroke deaths, and 77.7 % DALYs lost) in low-income and middle-income countries [37].

The central hospitals of rehabilitation medicine have multiple valences, providing special rehabilitation and functional reeducation cares, with emphasis on stroke patient treatment and spinal cord injury treatments, due to the various aspects and constraints related to their limitations. In the initial phase of the physical rehabilitation, patients require a strong monitoring and therapists support with close proximity, in most cases due to their limitations and low autonomy. During their recovery, the patients’ progress evaluation is carried out by medical doctors and specialized therapists, albeit with some degree of subjectivity, using scales and other assessment instruments to characterize the evolution of these patients.

The application of technology in health care has attracted, from a long time, the engineering attention and support to assist the therapy process, such as the case of gait analysis. Due in large part to the great changes in technology, particularly in optical motion capture systems and information extraction through digital image analysis. These systems have been created to support health professionals in the evaluation recovery process of patients, and to help to improve the level of the physical recovery of people. Some research demonstrates that exergames (video games which integrate exercise and gaming entertainment) can generate some general health benefits and also positive effects in increasing the recovery process in some diseases. The level of motor and functional recovery after the occurrence of a stroke is characterized by a great individual variability.

Some motor functions recover quickly, while others may remain indefinitely with permanent deficits. A large part of the motor function recovery occurs within the first three months after the occurrence of the vascular event, and after six months only small improvements are expectable, but these improvements can be functionally significant. For some patients recovery of motor function can continue for a long period.

After the initial recovery process in central specialized hospitals, patients have medical discharge, but they have to continue the recovery process. Usually, a set of exercises is prescribed, and can be performed in proximity physiotherapy centers, or by themselves in their own homes, in attempt to continue to improve their recovery. These periods that patients spend at home are essential both for economic and resource management reasons, and, or by social and emotional personal reasons.

After this recovery period in ambulatory, patients return to the central hospital facilities for evaluation, and since medical discharge there is no interaction or feedback to the medical doctors or central hospital. According to recent studies, in most cases, either for economic reasons or because of family and social low support, the patients do not fulfil the recovery plan. Consequently, setbacks occur in the recovery process, worsening the quality of life and, in some situations, patients need to return to the hospital for another period of time.

In this context, we propose a set of research and development activities, with the following main objectives:

  • Application of information and communication technologies in the development of a prototype of a system focused on the upper limb rehabilitation of post stroke patients using exergames and natural interaction, in attempt to establish a link between central level (i.e. specialized hospitals) and patients recovery process at home or in proximity centres.

  • Design and implement an intervention program to a group of post stroke patients randomly divided into experimental group (technology aided care) and control group (usual care).

  • Validation of the results of the previously mentioned prototype and program with the results of medical assessment of the post-stroke patient.

The system development is associated with the specific objectives of procedural generation of the levels of exergames based on medical prescriptions and the capabilities to be exercised, dynamically adapted to the state and progression of the patient; capture of body movements and extraction of gesture information; nonintrusive acquisition of complementary biomedical signals; and biomedical and biomechanical metrics generation.

The system will be implemented using an evolutionary development model at the Rovisco Pais Medical Rehabilitation Center (CMRRC-Rovisco Pais), which will have the support of the Research Unit of the centre during the first eighteen months of the project. After the development phase, the system will be set up at the patients homes appointed by CMRRC-Rovisco Pais and will be tested and evaluated by patients and by health experts of this centre over the following months under project NanoStima (NORTE-01-0145-FEDER-000016). At the same time, relevant changes or adjustments will be introduced.

2 State of the Art

In the last two decades there has been an increase in the absolute number of people with new stroke (68 %), post-stroke survivors (84 %) and DALY (12 %) 23 [4]. Although the ratios incidence/mortality came to decline in the last two decades, there has been an increasing overhead in terms of the annual absolute number of stroke victims, the number of post-stroke survivors and DALY, especially in countries with low and middle incomes. Estimates from the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD 2010) ranked stroke as third most common cause of disability-adjusted life-years (DALYs)2 worldwide in 2010 [38].

After a stroke various capacities can be affected, inter alia, language, personality and memory. The classic description of motor deficit in the acute phase of a stroke often points the predominant involvement of the upper limb, and its less favourable recovery comparatively to the leg. Spontaneous motor recovery of upper limb after stroke is generally limited to the first six months of injury, during which the rehabilitation medicine may have an active role, in a facilitating sense.

Traditional methods require intensive rehabilitation after a stroke, accompanied by physical therapists, therapies and specialized equipment. In recent years there has been an increase in the costs associated with these treatments [5], which has led to a decrease in demand for treatment sessions, which, in turn, has reflects in the rehabilitation process. One potential solution to the problem is based on the use of home-based exercise in virtual reality systems. Virtual reality has been shown as an effective means when used in the rehabilitation training. This approach, combined with technology of serious games contributes to the increase of the stimuli and the patient’s motivation indices to meet the rehabilitation programs [6].

Research has shown that digital games improve elders’ physical health, in aspects as diverse as physical balance, balance confidence, functional mobility, executive function and processing speed [7]. The last decade has witnessed the increasing use of games that employ the motor skills of players, including elders, for physical rehabilitation, treatment and diagnosis [8,9]. The use of such games, dubbed “exergames” [10] has been shown to have a positive effect on the physical outcome of elders 11. In particular, stroke rehabilitation using low-cost motion detection is a recent focus of research, with applications both on rehabilitation methods and serious games [12]. Several research efforts focused on the evaluation of Kinect’s spatial accuracy specifically for stroke rehabilitation purposes. Limitations of current research are its focus on sets of gross movements, with more specific diagnostic movements sets needed; and the need to complement Kinect’s accuracy limitations, on detection of internal shoulder joints rotation and fine motor skills (ibid.). A recent survey comparing therapy outcomes for post-stroke adults between motion-detection virtual reality (VR) and conventional therapy has shown that VR rehabilitation yields moderately improved outcomes, and called for better control of participation measures, and motivational components of therapy [13]. The motivational aspects can be tackled by the entertainment component brought about by exergames, rather than plain VR therapy. This motivational factor, or engagement, can address a major barrier to rehabilitation: patient nonadherence. Research points out that game design, in aspects such as choices, rewards, and goals, leads to increase in motivation/engagement [14].

This design aspects add on the basic design guides and should address the principles of motor recovery and learning: meaningful task; intensive and repetitive practice; close-to-normal movements (including bilateral exercises); muscle activation driving practice of movement; focused attention/motivation; and training specificity, variability and progression [31–34].

Procedural content generation is a commonly used term to describe a methodology that seeks to produce automatically, rather than manually, any kind of media content (models, textures, sounds, objects, etc.) through a set of techniques, computer algorithms and a certain degree of randomness. The concept of procedural content generation is not new in the gaming domain. One of the first use cases emerged in the 80’s, called Elite.

There is a set of distinct techniques for procedural content generation for digital games [15]. Some of these techniques can be used to generate game levels [16] in a way to adapt the game design, automatically, to the distinct needs of a specific problem, namely it can be used to adapt to a specific therapy. The success in engaging the player is also connected to the way the gameplay adapts her/him. Therefore, some recent research is focused on adapting the gameplay, dynamically, to the players’ needs [17]. The game design should enforce the training shaping (specificity, variability and progression) assuring: immediate feedback concerning movements; individualized tasks; prompting and cueing; and progressive increase in the difficulty of the tasks [35, 36].

Motion tracking or Motion Capture (MoCap) records human body’s kinematic data with high accuracy and reliability. It has been started as analysis tool in biomechanics research in the 1970 s, however as the technology matured, it was expanded in several fields: robotics, human computer interaction (HCI), cinema, and video games, and virtual reality. Nowadays MoCap is very useful in medical science and sports applications to analyse human movement and gait.

Optical motion capture systems utilize data captured from image sensors to obtain the 3D position of subject’s body joints. Traditionally data acquisition was implemented using special markers attached to an actor; however, emerging techniques and research in computer vision leaded to the development of the markerless approach. Because in vision-based markerless human motion capture technology, users are not required to wear a special costume and no markers need to be attached to the human body, this new technology provides a very attractive solution for the movement analysis in patients with stroke.

In markerless systems, the image features such as colors, edges, shapes, and/or depth are used to track human skeletal and estimate accurately the 3D body joints positions. Microsoft Kinect is one of such capture systems, which uses computer vision algorithms to detect and track human body from the sequences of depth images. It enables users to control and interact with electronic devices, through a natural user interface using gestures and spoken commands. Several studies identify the Kinect’s potential for use in rehabilitation [19,20].

3 Methodology

The methodology aligns the expected positive effects, correlating the key research challenges with specific research activities. The results of these research activities may provide new insights into the distance monitoring of rehabilitation patients and the ability of technology to support patients to fulfil their rehabilitation plan.

We expect several positive effects of the use of exergames for rehabilitation:

  1. (1)

    Patients motivation: exploring the entertaining aspect of rehabilitation, as literature suggests that the age group, of stroke patients, falls within the strategy of usage of serious games for exercise;

  2. (2)

    Socialization: a social network of stroke patients with matching motor skills enhancing the group capability to pursuit their objectives, following gamification patterns;

  3. (3)

    Personalized exergames with adaptation to user capabilities: the exergames are generated to each user from the clinical prescription and adapted to the users’ attitude and execution of the exercises correcting their positions and movements.

The System will mimic the exercises used in specific Physical therapy interventions, improving the motor function (fine and gross), speed and grip strength related to arm-hand activities focused in intensive high repetitive task-oriented and task-specific training in all post stroke phases [22]. Therefore, we expect to contribute positively to the recuperation of patients due to an increase of the number of hours of specialized therapy thanks to its availability at every moment at their homes.

3.1 Objectives and Research Activities

The specific objectives are described in the following list, as well as the research activities designed to assure their fulfillment. Specific objectives:

  • (O1) the development of a prototype of a system focused on the upper limb rehabilitation of post stroke patients using exergames and natural interaction, establishing a link between central level centres and patients when they are recovering at home or in proximity centres;

  • (O1.1) procedural generation of the levels of exergames based on medical prescriptions and the capabilities to be exercised, dynamically adapted to the state and progression of the patient;

  • (O1.2) capture of body movements and extraction of gesture information;

  • (O1.3) non-intrusive acquisition of complementary biomedical signals;

  • (O1.4) generation of biomedical and biomechanical metrics.

  • (O2) application of an intervention program to a group of post-stroke patients randomly divided into experimental group (SelfTherapy) and control group (usual care);

  • (O3) validation of the results of SelfTherapy with the results of medical assessment of the post-stroke patient.

The research activities with a focus on testing and evaluation:

  • (R1) Exercise motivation and Serious games;

  • (R2) Evaluating smart clothes for physiological monitoring with embedded electronics;

  • (R3) Body motion capture and biomechanics analysis;

  • (R4) Integration and visualization platform;

  • (R5) Tests and evaluation.

The objective (O1.1) is primarily supported by research work to be developed on research activity 1 and the objectives (O1.2) and (O1.3), are respectively addressed by research activities 2 and 3. Research activity 4 comprises the integration of work carried on research activities 1, 2 and 3, in order to attain the objective (O1.4) and consequently the aggregation of the work to achieve the overall objective (O1). The consolidation of objectives O2 and O3 are addressed in research activity 5, also ensuring a continuum of evaluation by users and experts’ feedback crosswise involved in research activities 1, 2, 3 and 4.

3.2 Development

Methodologically a user-centred development will be followed, based on the ISO 9241-210: 2010, with strong relations between the research team and the potential project stakeholders, linking the identified requirements with the user needs. This design approach also aims to improve patients’ motivation and engagement, without compromising the goals of therapy. We will combine the Lukosch et al.’s method [18] with low-tech prototype methods of participatory design, which enable faster feedback loops and diminish risk [21]. Functional prototype generation and testing will be performed in each cycle in two steps: first using fast prototyping tools for a quick feedback loop with the participation of experts. After experts approve the prototype, different system actors will be involved in the testing, to further refine the prototype iteratively. Four combinatory cycles are defined for the development process:

(C1) Core features: analysis, where typical methods (therapist interviews, literature review) are combined with therapist participation to establish design criteria, via brainstorming sessions known as Group Decision Room and role-playing sessions to enhance context awareness; conceptualization of objectives O1.1, O1.2 and O1.3 and low-fidelity prototyping of concepts; demonstration of the technology possibilities to the users; testing and requirements gathering of multimodal interfaces for achieving the stated features.

In research activities 1, 2 and 3 single low fidelity demonstrators will be created. In such demonstrators users should perceive the capital gains and individual impacts of serious games, body motion capture, gesture recognition and textiles with embedded electronic that the project aims to develop and integrate. In research activity 4 the system usage storyboard will be developed, considering the various use cases and players of the system. This storyboard will be validated in tests set within research activity 5, which will be consequently mapped into use cases diagrams. The activities outlined which involve interaction with the system actors are framed in research activity 5 and will be conducted in the laboratory at CMRRC-Rovisco Pais. During this cycle there is interdependence between research activity 1 and research activity 5, overlooking the definition of the principles and metrics to be used for exercises and therapies classification that can be mutually accepted by therapists, clinicians, and data captured during the game execution (game analytics).

(C2) Multimodal interfaces: Synthesis of the criteria, yielding a provisional design; Simulation via generation of a low-tech prototype, which will enable experts and developers to test the designs, detect misunderstandings, and refine concepts, yielding a refined system design; first interactions in the laboratory, enhancing the assessment of user needs and their capabilities to the use of technology overlooking the refinement of the interaction potential and user experience.

Functional prototypes will be implemented, developing the demonstrators established in the previous cycle and reflecting the needs and expectations stated in the tests. In research activity 1 serious game prototypes will be developed integrating game mechanics adapted to the rehabilitation plan prescribed by clinicians, a generation module and dynamic adaptation of the levels of these games given the capacity to exercise the patients according to the good practice defined a set of standardized exercises established and validated for rehabilitation of stroke patients. An application prototype of a t-shirt with embedded sensors to capture biological data will be developed at this stage in research activity 2. In research activity 3 libraries to capture body movements and gestures extraction will be developed and exploited standards for representation of the collected data. Note that all demonstrators and prototypes developed during this cycle in research activities 1, 2 and 3 are standalone, without any kind of integration between them. The definition of integration API and low-fidelity prototypes of different user interfaces (mobile, web and set top device) will be developed during this cycle on research activity 4. In research activity 5 different tests on the developed functional demonstrators will be coordinated and a database of therapies and exercises classified according to the metrics defined in the previous cycle will be developed.

(C3) User interaction and user experience: development of a full-system; laboratory testing with users for real-scenario test preparation. This cycle is characterized by the integration of the various components developed in different research activities, with a strong interdependence between all research activities. Thereby in the research activities 1, 2 and 3, the specified integration API will be used under the coordination of the research activity 4 to connect the all system. In these research activities usability tests and developed mechanisms will also be defined to improve the user experience through an internal module (in-game assessment) to capture the game data (game analytics). In research activity 4, the integration process will be coordinated and defined the orchestration model following storyboard set on cycles (C1) and (C2). As in previous cycles, the interaction tests with system actors are set at research activity 5 and will be conducted in lab environment at CMRRC-Rovisco Pais.

(C4) End user testing in real scenario: tests involving a small group of users that during a three months period will have the system installed in their homes, simulating an actual usage environment and allowing the team to understand the system impacts. Two groups of patients will be considered: an intervention group that will use the system during the period of 3 months; a control group that will not use the system. Patients will be observed before and after the period in ambulatory by physicians who follow them, as detailed in the planning description of research activity 5.

The coordination of the tests with users in a real scenario will be performed on research activity 5, setting the overall system evaluation policy. Sectorially, for each of the research activities and according to the established objectives metrics analysis and appropriate quantifiers to the specific needs assessment will be defined.

A group of patients classified with mild/moderate stroke after hospital admission and baseline tests will receive 3 months of a “routine” physiotherapy by a specialized hospital therapist. After hospital discharge and new testing, patients will be randomly assigned in a control group (CG) or intervention group (IG). The control group will continue physiotherapy on ambulatory conditions (e.g. rehabilitation centers), and the intervention group will continue physiotherapy on ambulatory conditions plus the same exercise in the system platform in a home-based condition during more 3 months.

The recovery process assessment will be tested with the following tools:

  1. 1.

    Brunnstrom recovery stage [24];

  2. 2.

    Enjalbert recovery stage;

  3. 3.

    Wolf Motor Function Test [25];

  4. 4.

    Box and Block Test [26];

  5. 5.

    Stroke Impact Scale [27];

  6. 6.

    Action Research Arm Test [28];

  7. 7.

    Fugl-Meyer Scale [29];

  8. 8.

    Short form (36) health survey (SF-36) [31].

The methodology presented and the defined plan are articulated in the following cycle/research matrix, which summarizes the entire process.

4 Conclusion

The development of a system following this methodology should assure a low cost, user friendly and motivational/behavioral solution that aggregates three elements in the rehabilitation of post-stroke patients: recognition of body movements and gestures; instrumented textiles; and adaptive serious games and gamification strategies.

Regarding the overall benefits of the system, we expect to demonstrate that the passive monitoring strategy abdicates of clinical-patient interactions in the ambulatory period, creates an indirect relationship between the two, based on notifications and allowing early intervention in situations of default and/or reversal of the patient’s condition. The gamification approach should ensure an easy adoption of the system as well as a user readiness by the patients as end-users.

A key aspect for the success of this methodology is the consortium which will develop the various research activities, including the software prototype. It is such an interdisciplinary proposal, that we came to the conclusion that this consortium should have deep insight in the areas of technology, health, rehabilitation and post-stroke treatment.