--- title: "Digitalized Cognitive Assessment mediated by a Virtual Caregiver" authors: - "Matteo Luperto" - "Marta Romeo" - "Francesca Lunardini" - "Nicola Basilico" - "Ray Jones" - "Angelo Cangelosi" - "Simona Ferrante" - "N. Alberto Borghese" doi: "10.24963/ijcai.2018/856" url: "https://ijcai.org/proceedings/2018/856" pdf: "https://ijcai.org/proceedings/2018/0856.pdf" venue: "Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence" year: 2018 pages: "5841-5843" track: "Demos" keywords: - "Multidisciplinary Topics and Applications: Human-Computer Interaction" - "Multidisciplinary Topics and Applications: Intelligent User Interfaces" --- # Digitalized Cognitive Assessment mediated by a Virtual Caregiver Matteo Luperto, Marta Romeo, Francesca Lunardini, Nicola Basilico, Ray Jones, Angelo Cangelosi, Simona Ferrante, N. Alberto Borghese *Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence*, pages 5841-5843. DOI: [10.24963/ijcai.2018/856](https://doi.org/10.24963/ijcai.2018/856) ## Abstract The ageing of the population deeply impacts on the social costs relative to health care. The use of modern technologies is one of the most promising approaches, under current study, to reduce such impact. In this demonstration, we propose a framework that can be employed for at-home assessment of Mild Cognitive Impairment (MCI). It is composed by a set of digitalized cognitive tests, developed from their paper-and-pencil counterparts, and by a Virtual Caregiver, which oversees the test execution and provides instructions. ## Full text ## 1 Introduction The growth in number and proportion of the older persons in each country's population is a well observed and established phenomenon [Nations, 2002], which has a remarkable influence on the social costs relative to health care. Among others, cognitive changes induced by ageing have an impact on the overall life quality of elders and on the efforts required to assist them. Such changes can develop into dementia by crossing an intermediate phase called Mild Cognitive Impairment (MCI). At present, neuropsychological assessment is the most useful tool for identifying patients with MCI, which is usually performed by means of a set of standardized cognitive tests, performed by a clinician, in a controlled environment, during a medical examination. Early assessment that anticipates and detects the need for a thorough clinical evaluation is a difficult task for which an established solution is still largely missing. Digital technologies are considered a valuable tool in order to bridge this gap. In this demonstration, we present a scenario in which cognitive assessment is provided by means of a set of digitalized cognitive tests, inspired by their paper-and-pencil counterparts, and performed under the supervision and guidance of an AI embodied into a multimodal interface. State of the art paper-and-pencil cognitive assessment involves two actors: the subject who performs the test and a caregiver who oversees the test, interacts with the subject and provides indications. The caregiver role within the test is complex and multifaceted, and the instruction he/she has to provide are numerous. Among them we can identify the following: 1. provide the instructions on how to perform the tests 2. assess that the user has understood such instructions 3. oversee the execution of the test. Then, by following a standardized procedure, 4. detect errors made during the test 5. provide indications (if needed) 6. end the test 7. evaluate the test. In this demo, we describe a set of digitalized cognitive tests and a virtual caregiver embodied by a multimodal interface. The digitalized cognitive tests (Section 3) are provided as a web application and inspired by state-of-the-art cognitive tests. Besides the actual execution of the tests this component is responsible for (4,6,7) and for data collection and analysis. The Virtual Caregiver (VC) is an AI which interacts through speech with the subject, listens to his/hers replies, and acts accordingly. Its role is to perform steps (1-2-3-5). A graphical explanation of the interaction between the two components can be seen in Figure 1. These two agents represent building blocks of a wider multi-actor platform developed within the scope of the Movecare project 1 , which pursues the development of a multi-actor system to provide assistance, activities and transparent monitoring to the elder at home. In this framework, the VC centralizes the knowledge obtained through a set of different sources (sensor embedded in the environment, social and cognitive activities performed on virtual community) and proposes a direct intervention by using an assistive Giraff robot, who lives with the elder, thorough a multimodal interface. We want to stress that our system does not intend to replace cognitive assessment performed by a clinician with a test performed by a virtual caregiver. We intend to provide an instrument that, if used at home, could make a step towards early cognitive monitoring, by identifying patterns that may correlate with an early phase of MCI in order to trigger a timely response by a subsequent clinical cognitive assessment. ## 2 Cognitive Tests Computerized testing has several advantages over the standard paper-and-pencil one, such as automatic data collection and analysis [Costa et al., 2017; Valladares-Rodr´ıguez et al., 2016]. The first attempts to introduce digital versions of classical paper-and-pencil neuropsychological tests have been reported in [Valladares-Rodr´ıguez et al., 2016; Canini et al., 2014; Fellows et al., 2017; Dahmen et al., 2017]. We have developed a digitized version of three neuropsychological tests commonly used to screen for MCI, Trail Making Test of type A and type B (TMT-A, TMT-B) [Reitan, 1958] and Bells Test [Gauthier et al., 1989]. 1 http://www.movecare-project.eu/ The digital versions of the neuropsychological tests were implemented as a web application using HTML5 canvas and JavaScript and were developed to work on any touch interface in full screen, to avoid distractions. For the proposed demonstration, we use a Samsung Galaxy Tab A6 with an S-Pen, whose stylus mimics the paper-and-pencil approach and increases the test acceptability. During the Trail Making Test the user has to connect, in sequence, a list of dots (numbered 1 to 25 for TMT-A and using both letter and numbers, thus forming the sequence {1-A-2-B-3-C-. . . } for TMT-B). During the Bells test, the user has to identify a set of identical icons (icons of a bell) among a set of similar icons. The digital TMT was designed to be structurally similar to the original paper-based version but, given the reduced dimensions of the tablet w.r.t. an A4 paper sheet, the number of targets was decreased to 20 for both versions [Fellows et al., 2017], following the layout proposed by [Giovagnoli et al., 1996]. The layout of the digital Bells test, on the other hand, was identical to the one proposed by [Vallar et al., 1994] (35 targets and 280 distractors in the same position) but the graphical elements were scaled to fit the tablet size. If the test protocol requires that an error made by the subject is corrected (as in the case of TMT but not in the case of Bells) the error is signaled to the VC who acts accordingly. Test results are stored in a cloud database. These data are used to allow caregivers and clinicians to watch a deferred session of the test without the need of physically attending it. ## 3 Virtual Caregiver The Virtual Caregiver (VC) is an AI agent, deployed in the cloud but embodied in a multimodal interface (on a screen for this demonstration, on an assistive Giraff robot in the Movecare project), with the main functionality of initiating the test procedure and guiding the user towards its completion, by mimicking the role of an actual caregiver. A multimodal interface is a well-established requirement for all those applications that are explicitly aiming at being accepted, and effective, for the older population. Its immediate advantage, perceived by the users, is the possibility to choose the input modality they want to use [Oviatt, 2003]. This is fundamental for the elders that often have difficulties, due to poor eyesight or movement impairments, to use classical graphical interfaces, or, due to hearing impairments, to use a speech interface. Thus, multimodal interfaces result more advantageous for older users, by offering them the choice of the interaction channel that is most suitable to their capabilities [Wechsung and Naumann, 2009]. In this demonstration, whose steps are described in Figure 1, the VC guides the users through a multimodal interface that combines both a speech and a graphical interface (the actual cognitive test) embedded in a monitor and in a tablet. The speech interface has two distinct duties: it has to recognize all the answers or requests arriving from the users and produce the audio response. Both the speech recognition and the speech synthesis modules of the interface are implemented relying on cloud services. The speech recognition is done in real time using Google Cloud Platform Speech API. The speech synthesis is done exploiting Acapela Voice As a Service that, after receiving a text over HTTP request, returns an audio file. The speech interface exchanges (ROS) messages with the digitalized cognitive test during their execution (on the tablet screen) in order to properly react to every user action. The speech interface is always listening for utterances, once it captures an input from the user, it performs a keyword search on the text returned by the Google API, based on a set of dictionaries. Each dictionary is related to the state in which the interface is, the phase of the interaction, in order to perform a specific and fast identification of the next step to trigger. Once the next step is detected, a message is sent to the graphical interface/cognitive test, so that it can update itself to work in parallel with the speech module. If the graphical interface detects the change of state before the speech interface, the exchange of message is performed in reverse. Figure 1: The flowchart of this demonstration. ![Image](https://ijcai.org/proceedings/2018/fulltext/856/image_000000_ad41e5d0eb65188502e8d102c667ee19eb137d5715fe7cb623064d28d24fdd93.png) An explanatory example is given by the test completion phase, as in Figure 1. The completion of the test can be notified to the system by the speech interface capturing the user saying "I have finished" (or a semantically similar sentence) or by the graphical interface. In both cases a ROS message is sent, from the speech interface to the test or vice versa, containing the next phase to be implemented: the ending of the test and the delivery of the feedback to the user. ## References 1. [Canini et al., 2014] Matteo Canini, Petronilla Battista, Pasquale Anthony Della Rosa, Eleonora Catricalà, Chris- tian Salvatore, Maria Carla Gilardi, and Isabella Cas- tiglioni. Computerized neuropsychological assessment in aging: testing efficacy and clinical ecology of different interfaces. Computational and mathematical methods in medicine, 2014. 2. [Costa et al., 2017] Carlos Rivas Costa, Manuel José Fernández Iglesias, Luis Eulogio Anido Rifón, Miguel Gómez Carballa, and Sonia Valladares Rodríguez. The acceptability of tv-based game platforms as an instrument to support the cognitive evaluation of senior adults at home. PeerJ, 5:e2845, 2017. 3. [Dahmen et al., 2017] Jessamyn Dahmen, Diane Cook, Robert Fellows, and Maureen Schmitter-Edgecombe. An analysis of a digital variant of the trail making test us- ing machine learning techniques. Technology and Health Care, 25(2):251-264, 2017. 4. [Fellows et al., 2017] Robert P Fellows, Jessamyn Dahmen, Diane Cook, and Maureen Schmitter-Edgecombe. Mul- ticomponent analysis of a digital trail making test. The Clinical Neuropsychologist, 31(1):154-167, 2017. 5. [Gauthier et al., 1989] Louise Gauthier, Francois Dehaut, and Yves Joanette. The bells test: a quantitative and quali- tative test for visual neglect. International journal of clin- ical neuropsychology, 1989. 6. [Giovagnoli et al., 1996] Anna Rita Giovagnoli, M Del Pesce, S Mascheroni, M Simoncelli, M Laia- cona, and E o Capitani. Trail making test: normative values from 287 normal adult controls. The Italian journal of neurological sciences, 17(4):305-309, 1996. 7. [Nations, 2002] United Nations. World population ageing: 1950-2050. New York: Department of Economic and So- cial Affairs, 2002. 8. [Oviatt, 2003] Sharon Oviatt. Multimodal interfaces. In The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, page 286304. 9. L. Erlbaum Associates Inc. Hillsdale, NJ, USA, 2003. 10. [Reitan, 1958] Ralph M Reitan. Validity of the trail making test as an indicator of organic brain damage. Perceptual and motor skills, 8(3):271-276, 1958. 11. [Valladares-Rodríguez et al., 2016] Sonia Valladares- Rodríguez, Roberto Pérez-Rodríguez, Luis Anido-Rifón, and Manuel Fernández-Iglesias. Trends on the application of serious games to neuropsychological evaluation: a scoping review. Journal of biomedical informatics, 64:296-319, 2016. 12. [Vallar et al., 1994] G Vallar, ML Rusconi, S Fontana, and M Musicco. Tre test di esplorazione visuo-spaziale: taratura su 212 soggetti normali. Archivio di Psicologia, Neurologia e Psichiatria, 1994. 13. [Wechsung and Naumann, 2009] Ina Wechsung and Anja B. Naumann. Evaluating a multimodal remote control: The interplay between user experience and usability. In 2009 International Workshop on Quality of Multimedia Experi- ence, pages 19-22, July 2009.