Towards Human-Engaged AI

1. [Cattle and Ma, 2018] Andrew Cattle and Xiaojuan Ma. Rec- ognizing humour using word associations and humour anchor extraction. In Proceedings of the 27th Interna- tional Conference on Computational Linguistics (COL- ING), 2018.
2. [Chakraborti et al., 2017] Tathagata Chakraborti, Subbarao Kambhampati, Matthias Scheutz, and Yu Zhang. Ai chal- lenges in human-robot cognitive teaming. arXiv preprint arXiv:1707.04775, 2017.
3. [Chapman, 1997] Peter McFaul Chapman. Models of engage- ment: Intrinsically motivated interaction with multimedia learning software. PhD thesis, University of Waterloo, 1997.
4. [Chen et al., 2016] Longbiao Chen, Daqing Zhang, Leye Wang, Dingqi Yang, Xiaojuan Ma, Shijian Li, Zhaohui Wu, Gang Pan, Thi-Mai-Trang Nguyen, and Jérémie Jakubow- icz. Dynamic cluster-based over-demand prediction in bike sharing systems. In Proceedings of the 2016 ACM Inter- national Joint Conference on Pervasive and Ubiquitous Computing, pages 841-852. ACM, 2016.
5. [De-Marcos et al., 2014] Luis De-Marcos, Adrián Domínguez, Joseba Saenz-de Navarrete, and Car- men Pagés. An empirical study comparing gamification and social networking on e-learning. Computers & Education, 75:82-91, 2014.
6. [Gajos and Weld, 2008] Krzysztof Z Gajos and Daniel S Weld. Usable ai: Experience and reflections. In Work- shop on Usable Artificial Intelligence (at CHI'08, 2008.
7. [Grudin, 2009] Jonathan Grudin. Ai and hci: Two fields di- vided by a common focus. AI Magazine, 30(4):48, 2009.
8. [Gunning, 2017] David Gunning. Explainable artificial intel- ligence (xai). Defense Advanced Research Projects Agency (DARPA), nd Web, 2017.
9. [Lau, 2009] Tessa Lau. Why programming-by-demonstration systems fail: Lessons learned for usable ai. AI Magazine, 30(4):65, 2009.
10. [Laurel, 1991] Brenda Laurel. Computer as Theatre: A dra- matic theory of interactive experience. Addison-Wesley, 1991.
11. [Li et al., 2014] Wei Li, Tovi Grossman, and George Fitzmau- rice. Cadament: a gamified multiplayer software tutorial system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 3369-3378. ACM, 2014.
12. [Li et al., 2017] Quan Li, Peng Xu, Yeuk Yin Chan, Yun Wang, Zhipeng Wang, Huamin Qu, and Xiaojuan Ma. A visual analytics approach for understanding reasons behind snowballing and comeback in moba games. IEEE transac- tions on visualization and computer graphics, 23(1):211- 220, 2017.
13. [Li et al., 2018] Quan Li, Ziming Wu, Peng Xu, Huamin Qu, and Xiaojuan Ma. A multi-phased co-design of an inter- active analytics system for moba game occurrences. In Proceedings of the SIGCHI Designing Interactive Systems (DIS'18). ACM, 2018.
14. [Ma and Cao, 2017] Xiaojuan Ma and Nan Cao. Video-based evanescent, anonymous, asynchronous social interaction: Motivation and adaption to medium. In Proceedings of the 2017 ACM Conference on Computer Supported Coopera- tive Work and Social Computing, pages 770-782. ACM, 2017.
15. [Ma, 2014] Xiaojuan Ma. Data-driven techniques to assist de- sign of perceptually effective visual displays. Information Design Journal (IDJ), 21(3), 2014.
16. [McCarthy and Wright, 2007] John McCarthy and Peter Wright. Technology as experience. MIT press, 2007.
17. [Nass et al., 1993] Clifford Nass, Jonathan Steuer, Ellen Tauber, and Heidi Reeder. Anthropomorphism, agency, and ethopoeia: computers as social actors. In INTERACT'93 and CHI'93 conference companion on Human factors in computing systems, pages 111-112. ACM, 1993.
18. [Nass et al., 1994] Clifford Nass, Jonathan Steuer, and Ellen R Tauber. Computers are social actors. In Pro- ceedings of the SIGCHI conference on Human factors in computing systems, pages 72-78. ACM, 1994.
19. [Peng et al., 2018] Zhenhui Peng, Jeehoon Yoo, Meng Xia, Sunghun Kim, and Xiaojuan Ma. Exploring how software developers work with mention bot in github. In Extended Abstracts of Chinese CHI workshop of the 2018 CHI Con- ference on Human Factors in Computing Systems, 2018.
20. [Ren, 2016] Xiangshi Ren. Rethinking the relationship be- tween humans and computers. IEEE Computer, 49(8):104- 108, 2016.
21. [Salehzadeh Niksirat et al., 2018] Kavous Sale- hzadeh Niksirat, Sayan Sarcar, Huatong Sun, Effie L-C. Law, Torkil Clemmensen, Jeffrey Bardzell, Antti Oulasvirta, Chaklam Silpasuwanchai, Ann Light, and Xiangshi Ren. Approaching engagement towards human- engaged computing. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 2018.
22. [Shi et al., 2018] Yang Shi, Xin Yan, Xiaojuan Ma, Yongqi Lou, and Nan Cao. Designing emotional expressions of conversational states for voice assistants: Modality and engagement. In Extended Abstracts of the 2018 CHI Con- ference on Human Factors in Computing Systems, page LBW557. ACM, 2018.
23. [Sidner et al., 2005] Candace L Sidner, Christopher Lee, Cory D Kidd, Neal Lesh, and Charles Rich. Explorations in engagement for humans and robots. Artificial Intelligence, 166(1-2):140-164, 2005.
24. [Silpasuwanchai et al., 2016] Chaklam Silpasuwanchai, Xi- aojuan Ma, Hiroaki Shigemasu, and Xiangshi Ren. Devel- oping a comprehensive engagement framework of gamifi- cation for reflective learning. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems, pages 459-472. ACM, 2016.
25. [Sun et al., 2016] Zhida Sun, Mingfei Sun, Nan Cao, and Xi- aojuan Ma. Videoforest: interactive visual summarization of video streams based on danmu data. In SIGGRAPH ASIA 2016 Symposium on Visualization, page 10. ACM, 2016.
26. [Sun et al., 2017a] Mingfei Sun, Zhenjie Zhao, and Xiaojuan Ma. Sensing and handling engagement dynamics in human- robot interaction involving peripheral computing devices. In Proceedings of the 2017 CHI Conference on Human Fac- tors in Computing Systems, pages 556-567. ACM, 2017.
27. [Sun et al., 2017b] Zhida Sun, Nan Cao, and Xiaojuan Ma. Attention, comprehension, execution: Effects of different designs of biofeedback display. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 2132-2139. ACM, 2017.
28. [Vinciarelli et al., 2012] Alessandro Vinciarelli, Maja Pan- tic, Dirk Heylen, Catherine Pelachaud, Isabella Poggi, Francesca D'Errico, and Marc Schroeder. Bridging the gap between social animal and unsocial machine: A survey of social signal processing. IEEE Transactions on Affective Computing, 3(1):69-87, 2012.
29. [Vinson et al., 2018] David W. Vinson, Leila Takayama, Jodi Forlizzi, Wendy Ju, Maya Cakmak, and Hideaki Kuzuoka. Human-robot teaming. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, CHI EA '18. ACM, 2018.
30. [Wu and Ma, 2017] Ziming Wu and Xiaojuan Ma. Money as a social currency to manage group dynamics: Red packet gifting in chinese online communities. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 2240-2247. ACM, 2017.
31. [Yang et al., 2017] Yang Yang, Xiaojuan Ma, and Pascale Fung. Perceived emotional intelligence in virtual agents. In Proceedings of the 2017 CHI Conference Extended Ab- stracts on Human Factors in Computing Systems, pages 2255-2262. ACM, 2017.
32. [Yue et al., 2016] Yanzhen Yue, Xiaojuan Ma, and Zhenhui Jiang. Influence of content layout and motivation on users' herd behavior in social discovery. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 5715-5719. ACM, 2016.
33. [Zhu et al., 2015] Bin Tina Zhu, Yanqing Zhang, Xiaojuan Ma, and Haibo Li. Bringing chinese aesthetics into design- ing the experience of personal informatics for wellbeing. Design and semantics of form and movement, 289, 2015.
34. [Zhu et al., 2017] Fengyuan Zhu, Ke Fang, and Xiaojuan Ma. Exploring the effects of strategy and arousal of cueing in computer-human persuasion. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 2276-2283. ACM, 2017.

References were extracted automatically from the PDF and may contain errors.

Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.

1 Introduction

Researches on Artificial Intelligence (AI) and HumanComputer Interaction (HCI) have more crossovers in recent years, with AI techniques empowering HCI innovations while receiving data and feedback in return [Grudin, 2009]. At an application level, emerging robots, virtual agents, and voice (natural-language) interfaces for social and/or end-user service purposes are good examples of synthesis of the two fields. At a conceptual level, the rising "usable AI" (e.g., [Gajos andWeld, 2008; Lau, 2009]), "explainable AI (XAI)" (e.g., [Gunning, 2017]), and "human-aware AI (HAAI)" (e.g., [Chakraborti et al., 2017]) paradigms acknowledge the co-existence of AI technologies and human users, and have attracted attention of practitioners and researchers from both fields. One may envision more in-depth convergence of AI and HCI in certain domains in the future, with the focus being designing better use of (as opposed to merely better) AI.

To achieve mutual enhancement between human and AI, both parties need to honor and realize each other's full potential. In this process, the notion of "engagement" offers a lens into the synergistic relationship between human users and AI technologies [Salehzadeh Niksirat et al., 2018]. McCarthy and Wright suggest in their book "Technology as Experience" that it is the "meaningful engagement" that "transforms people and systems" [McCarthy and Wright, 2007]. For example, our past research on reviewer-recommendation bot deployed in online software development platforms e.g., GitHub showed that, automatic reviewer recommendation services would not improve the collaborative coding review process if they overlook potential code reviewers' (behavioral, cognitive, and emotional) engagement with the target project [Peng et al., 2018]. In the CHI 2018 Panel & Roundtable Discussion on HumanRobot Teaming (HRT), the panelists who are top scholars from academia and industry listed engagement and usability as the first key topic area of HRT [Vinson et al., 2018].

In this paper, we present an overview of the basic concepts and research landscape of human-engaged AI (HEAI), and our data-driven HEAI framework.

2 Human-Engaged AI (HEAI)

Adapting the notion of human-engaged computing proposed in [Ren, 2016], human-engaged AI (HEAI) aims to achieve a state of optimal, balanced synergism between engaged humans and engaging AI systems.

2.1 Definition of Engagement

Engagement is "a state of consciousness in which one is fully immersed in and aligned with the activity at hand" [Ren, 2016] and "the process by which interactors start, maintain, and end their perceived connections to each other during an interaction" [Sidner et al., 2005]. Engagement is not a basic action; rather, it is a process of communicating "what has been understood so far, what the interactors are each (or together) attending to, evidence of their waning connectedness, and evidence of their desire to disengage" [Sidner et al., 2005]. In the scope of this paper, interactors refer to human users and AI systems such as robots and virtual agents that form a 'symbiotic mutualism' relationship on the path towards a (shared) goal. Engagement in a way reflects the 'affordance' between the interactors [Salehzadeh Niksirat et al., 2018].

Human Engagement

Identifying when and why the human interactors' engagement cues conflict with the original intent to establish human-AI synergism is critical. The actual definition and measurement of human engagement may change along with the context, use case, and user group characteristics. Generally, we can classify human engagement into four types [Silpasuwanchai et al., 2016].

• Attentional Engagement: (selective) attentional attribution to a task, such as attention allocation and redistribution (e.g., [Chapman, 1997]).

• Cognitive Engagement: psychological devotion to a task, such as active thinking and reflection (e.g., [Li et al., 2014]), etc.
• Behavioral Engagement: physical participation and involvement (e.g., [De-Marcos et al., 2014]).
• Emotional Engagement: affective responses towards the task at hand, such as interest, excitement, frustration, and boredom (e.g., [Laurel, 1991]).

Generally, human users' engagement can serve as a sign of their conscious application of mental and/or physical capabilities during the interaction with AI technologies. How to infer one or more types of human engagement under a specific scenario is one of the key research problems in HEAI.

AI Engagement

People have the tendency to treat computers as social actors [Nass et al., 1994]. In particular, human users often make social attributions toward computing systems with even a minimal set of anthropomorphism and agency characteristics (e.g., robots and agents), and behave as if they are in front of other humans [Nass et al., 1993]. That is to say, people may also seek cues of AI engagement during the interaction, so as to adjust their behaviors and decisions accordingly.

To users, the 'engagement' of a technology can include but not limited to the availability and allocation of resources, internal states such as uncertainty and error while progressing in task, contextual and situational awareness, etc. Communicating the 'engagement' of an AI system via human interpretable cues makes its capacity and potential more transparent to the users. How to design the expression of AI engagement in a given task is another essential question in HEAI.

2.2 Engaged Human Users

In human-human interaction, people can take a holistic (sensory) and/or an analytical approach to inferring other individuals' engagement. This process consists of three levels: perception, comprehension, and projection. Perception means sensing all possible cues that can be indicative of one's engagement status. Comprehension means leveraging pre-established engagement model and backward looking engagement signals [Sidner et al., 2005] to further analyze the engagement dynamicity - fluctuation over time - and the underlying reasons. Projection means exploiting forward looking engagement signals [Sidner et al., 2005] to predict future engagement, with current actions, if any, taken into consideration.

2.3 Engaging AI Systems

If both human and AI are engaged in the interactive process, it is likely to result in more productive performance and relationship. In other words, the benefits of HEAI extend beyond utility, transforming the prosaic experience between human and AI to an aesthetic experience that is particularly fulfilling, satisfying, and creative [McCarthy and Wright, 2007].

The question arises as to what strategies AI systems can employ to engage users in an appropriate manner. In humanhuman interaction, such scenarios are usually handled with social intelligence and emotional intelligence. We propose to equip AI with similar capabilities, which, as Subbarao Kambhampati pointed out in his AAAI 2018 presidential address, is largely overlooked in current AI research.

3 Research Landscape of HEAI

Figure 1 illustrates our data-driven human-engaged AI framework. It consists of five key components: 1) construction of computational model of human engagement; 2) real-time holistic and analytical inference of human engagement; 3) management of human engagement; 4) expression of AI engagement; and 5) engagement-based applications.

3.1 Engagement Inference andModeling

Our group emphasizes on the use of the following three types of signals for inferring and modeling human engagement.

Social signals are "communicative or informative signals that, either directly or indirectly, provide information about social facts, namely social interactions, social emotions, social attitudes, or social relations" [Vinciarelli et al., 2012]. That is to say, social signals can convey attentional, emotional, and even cognitive engagement. Common social signals include gaze, facial expressions, vocal behaviors, proxemics, gesture, posture, and other body languages. Our group looks into non-intrusive methods to capture social signals during an interaction, applying sensors like camera, depth sensor, microphone and eye tracker (e.g., [Sun et al., 2017a; Zhu et al., 2017]; Figure 2).

Physiological signals are readings produced by physiological processes of human beings, including but not limited to heartbeat rate (ECG/EKG signal), pulse, respiratory rate and content (capnogram), skin conductance (EDA signal), muscle current (EMG signal), and brain electrical activity (EEG signal). We have been using physiological signals as indicators of attentional, cognitive and emotional engagement (e.g., [Sun et al., 2017b; Zhu et al., 2015]; Figure 3).

Behavioral signals are actions and activities performed during an interaction, as inputs or reactions towards output generated by the other party. Behavioral signals can be captured by interface logs, suggesting behavioral, cognitive, and sometimes emotional engagement. The actual types of data are often task-specific, ranging from conversational acts, text input, clicks, scrolling, page switch, emoji usage, likes, check-ins, to aggregated time on task and number of attempts. We have been mining behavioral signals and patterns in different application domains, such as online learning [Silpasuwanchai et al., 2016], digital games [Li et al., 2017; 2018] (Figure ??), social media and multimedia services [Ma and Cao, 2017; Wu and Ma, 2017], e-commerce [Yue et al., 2016], and smart transportation [Chen et al., 2016].

On the one hand, we use real-time social, physiological, and behavioral signals as backward and forward looking engagement cues to recognize users' ongoing engagement dynamics. On the other hand, by mining patterns (e.g., temporal or contextual) of aggregated social, physiological, and behavioral signals, we are able to construct general engagement model for future inference and prediction.

3.2 Management of Human User Engagement

We have been exploring two mechanisms - user initiative and system initiative - for managing human engagement.

User Initiative: Awareness and Reflection

The user initiative mechanism for engagement management requires AI system to evoke users' awareness and reflection of their own engagement status. Through techniques such as data visualization (e.g., Figure 3 and 4), users can gain a better understanding of their attention, emotion, and activeness on the fly and apply self-adjustment consciously. They can also conduct retrospective analysis of their past engagement dynamicity to improve future performance.

System (AI) Initiative: Intervention

The system initiative mechanism has AI proactively direct users' engagement (see Figure 5 for the different ingredients of this mechanism we have investigated). For example, we employ the proxemics theory (social intelligence) to design a persuasive screen that can engage users in posture adjustment during prolonged sitting [Zhu et al., 2017] (Figure 6 bottom). We exploit personality theory to design virtual agent [Yang et al., 2017] (Figure 7) and robot [Sun et al., 2017a] (Figure 6 top) to handle user challenges with emotional intelligence.

3.3 Expression of AI Engagement

When humans and AI systems are in close collaboration, keeping humans informed of the systems' 'engagement' can help mitigate miscommunication, boost coordination, and enhance satisfaction. In particular, we take an ethopoeia approach [Nass et al., 1993] to designing AI engagement expression. For instance, we experiment with the use of non-verbal emotional expressions to indicate a voice assistant's conversational states, to increase its connectedness and promote user engagement [Shi et al., 2018] (Figure 8).

3.4 Other Applications of Engagement

Additionally, we leverage the notion of engagement to design and improve intelligent services driven by AI algorithms. For example, we model users' attentional and cognitive engagement on different interfaces using eye-tracking data, and propose algorithms to automatically optimize the interface design (e.g., [Yue et al., 2016; Ma, 2014]). We mine player behavioral engagement patterns that can lead to different game occurrences, and develop a learning system based on the findings to identify game configurations that are critical for balancing different gameplay outcomes [Li et al., 2018; 2017]. We analyze viewer emotional engagement while watching (live) streaming videos according to time-sync commenting behaviors, and design a video synopsis, navigation, and recommendation system [Ma and Cao, 2017; Sun et al., 2016]. We have also been working on building computational humor recognizers in social media [Cattle and Ma, 2018].

4 Conclusion

This paper presents an overview of the conceptual framework and research landscape of data-driven human-engaged AI (HEAI), which aims to achieve optimally synergized interactions between humans and AI technologies. We summarize our exploratory works related to two essential components of HEAI, i.e., engaged humans and engaging AI systems. Our researches demonstrate the potential to reach mutual enhancement and aesthetic experience between the two parties.

Acknowledgments

This work was partially supported by the HKUST Initiation Grant No.IGN15EG02, the WHAT LAB grant No.1516144-0 and No.1617170-0, and ITF No.ITS/319/16FP grant.

Machine-readable versions of this paper, for tools and AI agents:

• Markdown — metadata, abstract, full text and references
• JSON — metadata and structured references
• BibTeX