1 Introduction

Dementia affects more than 35 million people worldwide, and the number is still growing dramatically. Dementia includes the symptoms of declining in cognitive capabilities including memory and thinking skill due to various reasons. Dementia damages people’s ability to live independently, thus many dementia patients face the prospect of being institutionalized. As home care is often costly, we are investigating the development of an affordable socially assistive robot to support cognitively impaired older adults with daily living activities, which will potentially extend their time living at home.

To facilitate effective personalized treatment, a caregiving robot should be able to communicate effectively with people who it interacts with, including both the patients and their family members. To communicate effectively, a caregiving robot should tailor its interactions to its interlocutors, taking into account the nature and degree of cognitive impairment, other health issues, and individual preferences.

2 Approach

We equipped the robot with a knowledge base embodying “best practices” for communicating with different classes of individuals [1, 2, 3, 4, 5, 6]. Upon delivery, the knowledge base can be augmented with (some) information about the people with whom the robot is expected to interact. When communicating with an individual, the robot uses the knowledge base to determine a customized strategy.

People with dementia often have difficulty completing daily tasks. In this work, we customized the interactions of the robot with users of the COACH system, which provides assistance to complete the task of handwashing [7]. COACH tracks the user’s hand positions to ensure that he or she is following a proper sequence to wash their hands. If COACH determines that the user is having difficulty, it will provide audio or video instructions, called prompts. These prompts are then customized by our system.

3 Implementation

COACH uses a POMDP-based Task Manager to estimate the user’s progress and determine the next prompt. The prompt is then processed by the Executor which plays a pre-recorded audio or video clip (see Fig. 1).

To customize the behaviour of COACH, we deployed a Prompter which reasons using features stored in the knowledge base to adapt the prompt for the user. These facts of the user are input into the knowledge base when the robot is delivered to the home by family members. We also modified the Executor so that it could use the output of the Prompter to dynamically generate an appropriate audio clip, rather than simply using a pre-recorded one (see Fig. 2).

Figure 1: The original COACH system.

Figure 2: COACH system with customization

Universal parameters — such as prompt detail level, volume, voice type, and speed — are determined using the knowledge base. Using these parameters, the prompt from the Task Manager, and other relevant information (e.g., how the user likes to be addressed, what kind of soap is in the environment), the Prompter generates a customized prompt and mode of delivery. This allows for different types of prompts for different users and situations. The Executor then uses IBM’s Watson to synthesize the audio clip (see Fig. 3).

Figure 3: The architecture of the Prompter

4 Illustrative Examples

The original COACH system does not take the individual condition of the user and the environment into consideration. Consider the following example prompt, in which the Task Manager decides the user should turn on the tap.

Original COACH Pre-recorded Prompt

“Try pulling the silver lever towards you.”

Now, we show three examples of prompts that our system generates instead, depending on individual circumstances, via the knowledge base. Note that both the content and prosodic features vary according to the individual.

Customized Prompts Based on Context entered by user

Scenario 1: The user is an elderly woman that has severe dementia and hearing difficulty.

“Ms. Marth, now turn on the tap.”

The user is addressed formally (as desired) using a short sentence. This prompt is generated using a low-pitched voice, high volume, and slow speed.

Scenario 2: The user is an elderly Spanish-speaking woman with mild dementia and mild hearing difficulty.

“Señora Abar, saque un poco de agua.” (“Ms. Abar, get some water.”)

The user is addressed formally in Spanish (as desired) using a short sentence. This prompt is generated using normal volume and normal speed.

Scenario 3: The user is a 4 year-old girl.

“Vivian, now turn on the tap. Be careful with the hot water.”

The user is addressed by first name using multiple sentences, with a high-pitched voice, normal volume, and normal speed. Being a child, she was also reminded of potential danger.

5 Evaluation

To evaluate the effectiveness of the customizations, we propose using the following criteria.

Effectiveness: Effectiveness measures how successful our approach is at helping users complete the handwashing task. We plan to evaluate effectiveness in trials by measuring how often patients can complete the handwashing task successfully when assisted by our system. We also plan to measure the number of prompts used per handwashing session.

Satisfaction: Satisfaction measures how satisfied users feel with the help provided by the robot. We would use a questionnaire completed by the user(s) and/or expert observers to provide feedback on individual satisfaction with the robot’s behaviour.

These kinds of evaluations require human trials. Due to the need for ethics approval, the evaluation is left for future work.

6 Conclusion

We customized the COACH’s prompts for assisting people with handwashing to demonstrate how human-robot interaction can be tailored. The prompts are customized using a knowledge base embodying best practices, taking into account individual circumstances and preferences. Our customizations adapt the robot’s interactions to the nature and degree of cognitive impairment, other health issues, as well as personal preferences, such as language choice. Ultimately, these customizations aim to help dementia patients to complete the daily living tasks and hence improve their quality of living.

7 Future Work

We plan to explore the use of Reinforcement Learning to adapt customizations through interaction with the user. Since people’s preferences and impairments can change over time, and the initial knowledge base may require some tuning, the robot should be able to learn about its users and incorporate the new knowledge into its interactions.

As our goal is to create a socially assistive robot to support cognitively impaired older adults with daily living activities, the robot may need to help with different kinds of activities, such as tea or meal preparation. The system can also be expanded to use other languages than English or Spanish. Finally, our customization techniques should undergo evaluation. We have discussed some strategies for evaluation above, and would use them to evaluate the system.

Acknowledgements

We would like to thank our collaborators: CrossWing Inc.; Professor Alex Mihailidis and team at the Toronto Rehabilitation Institute; Professor Goldie Nejat and team in Mechanical and Industrial Engineering at the University of Toronto; and Professor Franc ̧ois Michaud and team at the University of Sherbrooke. We also wish to thank Dr. Elizabeth Rochon from the Toronto Rehabilitation Institute for her expert advice on suitable strategies for communicating with persons with dementia. Computations were performed on the SOSCIP Consortium’s Cloud Data Analytics computing platform. SOSCIP is funded by the Federal Economic Development Agency of Southern Ontario, the Province of Ontario, IBM Canada Ltd., Ontario Centres of Excellence, Mitacs and 15 Ontario academic member institutions.

References

[1] Rozanne Wilson, Elizabeth Rochon, Alex Mihailidis, and Carol Leonard. Examining success of communication strategies used by formal caregivers assisting individuals with alzheimer’s disease during an activity of daily living. Journal of Speech Language and Hearing research, 55(2):328–341, 2012.

[2] Rozanne Wilson, Elizabeth Rochon, Alex Mihailidis, and Carol Leonard. Quantitative analysis of formal caregivers use of communication strategies while assisting individuals with moderate and severe alzheimer’s disease during oral care. Journal of Communication Disorders, 46(3):249–263, 2013.

[3] Elizabeth Rochon. Personal communication, 2017.

[4] Alzheimer’s Association. Communication and alzheimer’s, 2017.

[5] Alzheimer Society Canada. Ways to communicate, 2017.

[6] Family Caregiver Alliance. Caregiver’s guide to understanding dementia behaviours, 2017.

[7] Jesse Hoey, Axel von Bertoldi, Pascal Poupart, and Alex Mihailidis. Assisting persons with dementia during handwashing using a partially observable markov decision process. The 5th International Conference on Computer Vision Systems, 2007.