UniMF: A Unified Framework to Incorporate Multimodal Knowledge Bases intoEnd-to-End Task-Oriented Dialogue Systems

Yang, Shiquan; Zhang, Rui; Erfani, Sarah M.; Lau, Jey Han

doi:10.24963/ijcai.2021/548

UniMF: A Unified Framework to Incorporate Multimodal Knowledge Bases intoEnd-to-End Task-Oriented Dialogue Systems

Shiquan Yang, Rui Zhang, Sarah M. Erfani, Jey Han Lau

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 3978-3984.

https://doi.org/10.24963/ijcai.2021/548

Keywords

Natural Language Processing: Dialogue
Natural Language Processing: Information Retrieval
Natural Language Processing: Knowledge Extraction

Abstract

Knowledge bases (KBs) are usually essential for building practical dialogue systems. Recently we have seen rapidly growing interest in integrating knowledge bases into dialogue systems. However, existing approaches mostly deal with knowledge bases of a single modality, typically textual information. As today's knowledge bases become abundant with multimodal information such as images, audios and videos, the limitation of existing approaches greatly hinders the development of dialogue systems. In this paper, we focus on task-oriented dialogue systems and address this limitation by proposing a novel model that integrates external multimodal KB reasoning with pre-trained language models. We further enhance the model via a novel multi-granularity fusion mechanism to capture multi-grained semantics in the dialogue history. To validate the effectiveness of the proposed model, we collect a new large-scale (14K) dialogue dataset MMDialKB, built upon multimodal KB. Both automatic and human evaluation results on MMDialKB demonstrate the superiority of our proposed framework over strong baselines.

Shiquan Yang, Rui Zhang, Sarah M. Erfani, and Jey Han Lau. 2021. UniMF: A Unified Framework to Incorporate Multimodal Knowledge Bases intoEnd-to-End Task-Oriented Dialogue Systems. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 3978–3984. https://doi.org/10.24963/ijcai.2021/548

@inproceedings{ijcai2021p548,
  title     = {UniMF: A Unified Framework to Incorporate Multimodal Knowledge Bases intoEnd-to-End Task-Oriented Dialogue Systems},
  author    = {Yang, Shiquan and Zhang, Rui and Erfani, Sarah M. and Lau, Jey Han},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {3978--3984},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/548},
  url       = {https://doi.org/10.24963/ijcai.2021/548},
}

TY  - CPAPER
TI  - UniMF: A Unified Framework to Incorporate Multimodal Knowledge Bases intoEnd-to-End Task-Oriented Dialogue Systems
AU  - Yang, Shiquan
AU  - Zhang, Rui
AU  - Erfani, Sarah M.
AU  - Lau, Jey Han
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 3978
EP  - 3984
DO  - 10.24963/ijcai.2021/548
UR  - https://doi.org/10.24963/ijcai.2021/548
ER  -