1 Introduction

Japan is currently facing hyper-aging society, which causes many social issues. According to a report by the Government of Japan, the number of elderly people over the age of 65 is 35.15 million in 2017, which is 27.7% of the total population [2]. Due to the long average life and the low birthrate, the number of elderly over 65 is increasing in Japan and will reach 40% of the total in 2050. There are many challenges, for example, the shortage of human resources and facilities in the nursing care field. Therefore, the government is focusing on supporting home care. However, there is a possibility that the burden on family caregivers increases in home care. Furthermore, the elderly at home has various difficulties in daily life due to the decline of cognitive function. In particular, as elderly people are getting forgetful, they can not do what they had done until then, so that they feel strong anxiety, confusion, and deterioration of self-esteem. In order to support these elderly people, many research on assistive technology has been carried out in various organizations in recent years.

Our research group has been developing the activity recognition service based on the environmental change [6]. This service read the variations of environmental sensors in elderly’s home and detects their activities by machine learning technology. The detected activities are notified to the mobile terminals of elderly, and they respond to the notifications based on the their current situation. This service recognizes elderly’s activity and at the same time realizes the creation of communication opportunities for them.

On the other hand, the previous research is limited to the activity recognition that can be observed with traditional sensors. To execute more personalized care for elderly, it is important to obtain not only such external events but the internal states such as human emotions and physical condition. The internal state is directly linked to human health, and it is important information for monitoring [1]. However, the internal states are usually acquired through inquiries and counseling by experts that can not be done permanently at home. Moreover, it is difficult to measure the changes of internal states by sensors.

The purpose of this research is to support elderly people who feel uneasy about forgetfulness. In this paper, especially, we propose a new memory-aid service using virtual agents, which are 3D humanoid robots. The proposed service mainly has the following two functions.

  • Agent-Assisted Mind Sensing: A system that an agent asks users about their minds, to require and record the internal states of them

  • Memory-Aid Retrospective: A system that an user and an agent look back the day using the data recorded by mind sensing

In order to solve the limitation of the previous research, we propose mind sensing that a virtual agent directly asks the elderly about their internal states. With various events as a trigger, the agent sends a message to the elderly mobile terminal asking for the current physical condition and emotion. After the user answers the question, the detailed information of the answer is automatically stored in the database. As a result, we can grasp the internal state of the elderly and use it for more sophisticated care.

Retrospective is that the elderly and the agent review the day at an arbitrary sense of time utilizing the accumulated internal states of them. While viewing the list of messages entered by elderly by mind sensing, they calibrates and classifies these messages one by one. Thus, elderly people can prevent forgetfulness by looking back at what happened recently. Furthermore, they can search the internal states later on the basis of classifications, words, period, and can retrieve them whenever necessary.

In this paper, we also implement a prototype system that is equipped with the function of mind sensing and retrospective. Through the preliminary experiments, we confirm that the proposed system can be expected to be extremely effective forgetfulness.

2 Preliminary

2.1 Elderly Care at Home

Along with the aging society, single households whose head is over 65 years old is increasing in Japan. However, it is difficult to accept and carry out the care for all elderly at medical facilities. On the other hand, there are a lot of elderly that prefer to receive care at their own home or their family’s home rather than to receive at nursing home or care house. Against this background, in recent years, many attempts to care for elderly people at home have been made in various fields. Japanese Government is focusing on training human resources for home health care and promoting cooperation in each local government, and aims to realize Community-based integrated care systems [3] in which residences, medical care, and nursing care are integrally provided so that elderly can continue their own living in a familiar area even if they become severely in need of nursing care. The general companies are starting to work hard on developing new services that relieve elderly of loneliness using a communication robot. These effort aims to improve the quality of life (QOL) of the elderly.

2.2 Virtual Agent

In elderly care, it is expected to use care robots that utilize various assistive technologies to assist daily living. Among these technologies, virtual agent is a 3D friendly humanoid robot displayed on the PC screen, and can use speech recognition and speech synthesis technology to interact with users. By introducing such an agent to the medical field, it is expected that instead of an actual care provider, it becomes a communication partner of the elderly and reduce the burden of caregiver. In our previous research, we have developed a system, called Virtual Care Giver (VCG), using the VA [7]. VCG was designed to be able to cooperate with Web services, to integrate IoT, smart home and cloud. However, there is also a limitation caused by the looking of VA. The care and advices from unfamiliar avatar do not always motivate the elderly very well. In a recent study, to solve such a limitaion, we has been studying MPAgent system using virtual agent technology [5]. Figure 1 shows a screen shot of MPAgent. This service creates a 3D model of a face and dynamically generates an agent based on the feature points of the face acquired from the facial photograph. It also realizes a realistic dialogue by lips movement and changing the expression by the utterance. Delegating the communication care with MPAgent, a human caregiver can concentrate on human-centric tasks that cannot be done with ICT.

Fig. 1.
figure 1

Screen shot of MPAgent

Full size image

2.3 Elderly Monitoring Service

As one of the technologies to support home care for the elderly, a monitoring system at home using ICT is considered promising. In the general process of monitoring, first, the system senses real-world data, and grasp the situation of the elderly and their environment. The data to be sensed includes biometric data such as pulse and acceleration of arm or leg, and environmental data such as temperature, humidity, and illuminance. Second, using the obtained data, the system recognizes the elderly’s activities of daily living. Finally, based on the recognized activity, the system leads to actual care. Such a monitoring system reduces the efforts of nursing care and aims at improving the quality of care that only human beings can. However, in the conventional monitoring system, there were issues such as introduction cost, invasiveness problem, failure to achieve personal adaptation using the collected data, and lack of communication to the elderly. In order to solve these problems, in previous research, our research group has proposed and developed an elderly state notification service [6] using activity recognition based on environmental change. This service consists of an activity recognition service that predicts elderly’s activity by using environment sensor data, and a care execution service that makes notifications based on predicted activity using SMS application. With this system, it is possible to record activity by non-invasive sensing and to create communication opportunities for the elderly. On the other hand, even by utilizing various sensors and devices, it is only detecting external events such as environmental data and user’s expressions and behaviors, and it is impossible to observe the internal states within the user’s mind. The internal state is directly related to human health, and it is important information for monitoring. Obtaining internal states helps to grasp the more detailed status of the elderly and helps more sophisticated care. In addition to acquiring conventional external events, it is important for the monitoring system to continue to monitor the internal state.

3 Proposed Service

3.1 Requirement

In this research, we propose a service to acquire and accumulate the internal state of the elderly who could not be obtained by the conventional monitoring system. Also, it contributes to the forgetfulness of elderly based on the accumulated internal condition. The proposed service realizes the following two system requirements.

  • R1: Externalize the mind of the elderly into words

  • R2: Accumulate the words and retrieve them at any time later

We implement a mind sensing service with the function of R1, that acquire the internal states of elderly by the virtual agent. We also implement a retrospective service depending on the function of R2, that have elderly to look back on the acquired internal state using a virtual agent and lead to prevent forgetfulness.

3.2 Use Case Scenario

Here, we explain how the user actually uses the proposed system. When the user starts the mind sensing service, first of all, the system registers information of sensors installed in the house in order to link with sensor data. By doing this, various notification is sent to the user triggered by user’s activity recognition, so the user returns the current feeling, what happened on that day, future schedule, and so on.

When doing retrospective service, while interacting with the agent, the user corrects and classifies the data input so far. By classifying the contents of the data into classes defined by the users themselves, they are used for later retrieval. If the user wishes to remember what happened in the past, he can use various queries to search and confirm the data he entered by mind sensing, at any time. In addition, if the user wants to change personal settings, the user can make settings such as the agent’s questioning timing and user information change.

3.3 System Architecture

In this research, to support memory aid of elderly people, we build an architecture that can manage important information for the elderly’s life. Figure 2 shows the system architecture of the proposed service. The proposed system is made up of two services. One is Agent-Assisted Mind Sensing Service that acquires and records the feelings of the user. The agent triggers various events such as the result of the user’s activity detection presumed by the existing system and asks the user about his mind using a smartphone. Then, the service obtains the answer from the user and stores it in the database. By managing such internal state for users, we will use it for other services. For instance, the doctor and the caregiver refer to the accumulated information, they can grasp the state of the elderly and make appropriate care plan. The second is Memory-Aid Retrospective Service based on the data recorded by mind sensing. In this service, while interacting with the agent, the user looks back inside the mind that the user himself inputs on one day. As a result, the user can remember what happened on the day and what he thought. Furthermore, the user can search the data of his mind based on various queries and check them at any time. Since elderly people can retrieve important information anytime in their daily lives, it is possible to ease the anxiety of forgetfulness. It is expected these retrospective views will lead to memory aid of elderly. We discuss these two services in detail in Sects. 3.4 and 3.5 below.

Fig. 2.
figure 2

The proposed system architecture

Full size image

3.4 Agent-Assisted Mind Sensing

This system promotes the expression of the inner mind of each elderly person, and record it for high-quality care and communication to be done later. Especially, we use an external agent service to understand the internal conditions such as the elderly’s psychology and physical conditions and at the same time create richer communication. The agent inquires of the user about various internal conditions, then, stores the obtained answer in the database.

3.4.1 User Interaction

The agent gives questions according to the situation of the user. For example, at a fixed time every day, the agent asks the user about the physical condition at morning wake-up and about what happened today before sleeping at night. Furthermore, in cooperation with the existing system, the proposed system consider the observable user situation. Specifically, based on the user’s activity of daily living recognized by the existing system, the agent inquires accordingly at that time. The proposed system sends messages to the user using the agent service existing as an external service, triggered by the detection from the previous research.

3.4.2 Record of Internal States

This service asks the user to input his self-talk using a smartphone in a way that he answer questions of the agent, or spontaneously speak what he think at the moment the user likes. The user input messages by text or voice using a smartphone, and the message is automatically collected through the Web-API at the time when transmitted. The data structure of the message we manage is as follows.

figure a

Category is a schema for users to classify their messages in the retrospective system, and it is empty when each message is first stored. Accordingly, we organize information from the viewpoint of when, who, and what, so that, it is possible to grasp them based on various queries.

3.5 Memory-Aid Retrospective

This system helps the user to remember the memory by looking back on the messages of the mind entered by the user. While interacting with the agent, the user can perform the following operations.

3.5.1 Data Cleansing

The user calibrates and classifies the most recently entered messages. Initially, a list of messages for one day is displayed on the screen, and the user selects one by one. This is to correct unexpected conversion errors and sentence delimiters caused by voice input. If there is nothing to change, the user continues the retrospective without doing anything here. When calibration is completed, the user then sorts the messages into the categories defined by the user himself. It is not classified at the time when the user first inputs the message. The information of the messages on the database is updated when the user classifies them. By arranging and adding information to the original data in this way, they become richer contents, and it is useful for later searching under various conditions.

3.5.2 Search

When the user wants to remember necessary information, the user can search for data by three conditions: category, word and period. It is possible to confirm the details of messages from each search result, and the user can acquire necessary information at any time.

  • Search by classification

    The user can search tweets according to the classification of them. When the user selects a classification, the user can retrieve a list of the relevant tweets. The user can confirm.

  • Search by word

    The user can get a list of tweets based on the words of which the user is interested.

  • Search by date

    When the user designates a period, he can acquire the tweet inputted within the period. Therefore, the user can remember what happened at that time.

3.5.3 Personal Settings

The proposed system manages the setting information of each user in the database. The user can change various system settings on the screen. In this section, we will describe the following three functions.

  • Classification settings

    The user can define the types of categories as described in Sect. 3.5.1. This change is reflected when the user classifies the data.

  • Agent settings

    In the proposed system, we use the previous research MPAgent as a virtual agent. Since the MPAgent system generates an agent from an arbitrary facial picture, users can customize to their preferred agent. It is possible to realize a conversation with a person familiar with the user, such as relatives, children, close friends, etc.

  • Sensing settings

    The user can set a trigger for the agent to interact with the user. By setting the time and the place that the user wishes to notify, and what kind of content he wishes to be notified. Since notifications such as simple greetings, forgotten records, and important schedules arrive when the user needs it, the user will not have to worry about forgetting. By intervention in life using an agent, we can expect that elderly people can easily reconsider their lives.

4 Prototype Development

In this paper, we implement a prototype system that is equipped with the function of mind sensing and retrospective consisting of server-client system. In particular, we build a mind sensing service with the LINE Messaging API [4] for acquiring the self-talk of the elderly. To accumulate the internal state of elderly people obtained by mind sensing, we prepare a database and implement the Web-API for inserting and retrieving data. Furthermore, in collaboration with an agent which is an external service, we design a memory-aid service for a daily retrospective to help elderly memory. The following sections describe the details of the prototype.

4.1 Agent-Assisted Mind Sensing

In previous research, we have been developed the communication agent service using LINE application. This service is specialized in the function of interactive interaction and implemented as a web service. When an event such as message transmission/reception occurs, a POST request can be sent to the specified URL. In this paper, we develop a database which manages the internal states of users, and APIs to store and retrieve data. In mind sensing, when various events such as the result of activity recognition occur, or at a specific time, a message asking the user’s mind is sent to the user on LINE. When a message is replied from the user, a POST request is sent to the created API and the content of the message, the time when it was sent and so on, is accumulated in the database.

4.2 Memory-Aid Retrospective

When the user starts using the retrospective service, a menu screen is displayed, and the user can select any one of buttons “Daily Retrospective”, “Search”, and “Settings”.

When the user begins “Daily Retrospective”, the messages accumulated in the database are acquired by the API and displayed as a list. When the user selects any one of the messages, the list screen changes to the modification/classification screen. If the message is changed here, the data on the database is updated by the API. After this work is finished, the screen returns to the list screen again and the list of the messages to be displayed is also dynamically changed. The user can visually see the message and prevent forgetfulness by remembering what happened recently. Figure 3 shows the screenshot when the user experiences the retrospective function. We deploy MPAgent as an agent that interacts with the user on the left of the screen. On the right of the screen, we prepare a user interface for the user to perform various operations. In this figure, the contents the agent is talking about and a list of messages entered by the user are displayed.

Fig. 3.
figure 3

Screen shot of the proposed system

Full size image

When the user starts “search”, the list screen changes to the search screen. The user here can search past messages by category, word, and period, and easily switch search type by switching the tabs. In the search by category, it is possible to acquire the messages based on the category decided by the user at the time of daily retrospective. In the search by word, it is possible to acquire the message including the specified word, and in the case of the search by the period, the message within the specified period can be acquired. The user can retrieve necessary information at any time, so there is no worry about forgetfulness.

4.3 Development Environment

The development environment is as follows.

  • Development language: Java, JavaScript, HTML, CSS

  • Database: MongoDB-3.4.17

  • Web server: Apache Tomcat 7.0.77

  • Web service framework: Jersey framework

5 Preliminary Experiment

We conducted preliminary experiments targeting actual elderly people using prototypes. The purposes of the experiment are to evaluate the quality at the time of use for each function by having the user use the system. In this section, we explain the outline of the experiment using the proposed system, and consider the results based on the two requirement shown in ??.

5.1 Outline

The target of the experiment is a male in his 60s living with a couple. When the mind sensing system detects waking up of the tester, it asks the physical condition at that time. Similarly, when the system detects the activity of the tester, it makes a question depending on the situation. At the end of the evening, the tester use a retrospective system to look back on the day’s actions. The experiment period is 38 days from December 25, 2018 to January 31, 2019.

5.2 Result and Discussion

We confirm that there were messages about the pain of a body and physical condition of the tester. Such information cannot be obtained by traditional sensors, and we fulfill the requirement of mind sensing. In other words, it is possible to obtain internal states by mind sensing. On the other hand, the tester recorded measured values such as blood pressure and weight. It is expected to be able to develop further services by analyzing the information.

From the viewpoint of usability, the tester said to input by speech recognition is more convenient than that by flick operation. However, due to many operations until entering his internal states, it may be somewhat difficult for elderly people with advanced dementia to deal with the system.

In mind sensing service, the number of messages sent from the tester to the service was 337, and 331 of them were text messages, and 6 were images. In addition, 229 out of 337 messages were classified by the tester in retrospective service. In this preliminary experiment, the tester defined the six classifications, and the number of messages classified into each is as shown in the table below (Table 1).

Table 1. The number of messages in each category
Full size table

Here, messages classified into multiple categories are counted for each category. This result indicates that we can grasp what the tester is interested in. The reason why all the messages are not classified is that the user forgot to look back on the day or that the classification was not performed correctly due to systematic error. We should consider the need to prompt users to do retrospective and, of course, we must increase the reliability of the system.

The tester said that the function of short-term retrospective was effective in that when he was relaxing, it was able to correct typos of messages and realized what they did. On the other hand, there were opinions on the difficulty in using the user interface, such as many operations.

Regarding the search function, the search most frequently used by the tester was to search messages classified as “Memorandum”. This was why he accumulated what he might forget in advance, and used it as a way to prevent forgetfulness by checking later. In this way, by the search function, the tester analyzed the number and contents of the classified messages.

Through the preliminary experiments, we confirm that the proposed system can be expected to be extremely effective forgetfulness.

6 Conclusion

In this paper, we proposed a new memory-aid service supporting elderly who feel uneasy about the decline in cognitive function. Especially, the proposed service obtains and records the internal states of the user by mind sensing. Then, users and agents will look back on the day based on the accumulated data. We also implement prototyping and confirm the effect of the proposed service by preliminary experiments on an actual elderly person. For future research, we will extend the proposed system to support multiple users and evaluate in detail through experiments on many more people. In addition, we will develop a service that acquires new insights by analyzing accumulated internal states of elderly.