ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning

1 Introduction

Recent advances in large language models (LLMs) exemplified by GPT-3 [Brown et al., 2020] and LLaMA [Touvron et al., 2023] have demonstrated significant potential in the domain of zero-shot learning and logical reasoning. By aligning pre-trained LLMs to follow human language instructions through Reinforcement Learning with Human Feedback (RLHF) [Christiano et al., 2017], InstructGPT [Ouyang et al., 2022] and ChatGPT [OpenAI, 2023a] have showcased powerful capabilities for human-AI interaction, leading to a new paradigm shift towards the realization of artificial general intelligence (AGI).

Inspired by the remarkable success of GPT series [Brown et al., 2020; OpenAI, 2023a; OpenAI, 2023b], researchers attempt to incorporate more modalities into LLMs for multimodal human-AI interaction, with vision-language interaction being an important topic of focus. In order to incorporate visual modality into LLM, significant processes have been made to bridge the gap between LLMs and vision foundation models. There are two mainstream paradigms for building multimodal large language models (MLLMs). One is plugin-based MLLM [Wu et al., 2023; Yang et al., 2023b; Shen et al., 2023] that utilizes off-the-shell LLMs [OpenAI, 2023a; Touvron et al., 2023; Chiang et al., 2023] as central controllers to schedule different visual expert models as plugins. In this way, the users can interact with LLMs to achieve diverse visual functions. Another paradigm is endto-end MLLM [Alayrac et al., 2022; Huang et al., 2023; Liu et al., 2023b] that employs various techniques to align visual signals obtained from the vision encoder to the language semantic space and input vision tokens and language tokens together into the large language decoder.

Despite existing end-to-end MLLMs have achieved remarkable progress in vision-language human-AI interaction, the mode of interactive instruction is still limited in natural language. When meeting complex scenes as illustrated in Fig. 1 (a), it is difficult to only use the language to accurately describe the requirement of the user. However, if we can add some referring prompts, e.g., reference points, bounding boxes, etc, to MLLMs, the model can focus on the region of interest (RoI) and achieve finer-grained interaction, which is more flexible and user-friendly.

Motivated by this, we present ChatSpot, a fully end-toend unified multimodal language model designed to empower special region vision-language interaction. As illustrated in Fig. 1 (b), ChatSpot extends the LLMs' power to incorporate diverse multimodal inputs, and it can support a range of interaction forms. Users can communicate with the system using their native language, as well as gestures such as clicking and drawing boxes that we call Precise Referring, to obtain the desired information about the entire image or the region of interest (RoI). When a specific region is selected, ChatSpot can follow the precise referring instructions to perform various fine-grained applications, such as identifying jersey numbers or analyzing facial expressions in the given task, which is illustrated in Fig. 4. Furthermore, the precise referring can be regarded as a link of the chain-of-thought (CoT) to enhance the special logical reasoning ability of MLLMs. When an intelligent agent (robot or expert model) locates a target or region of interest based on user demands, ChatSpot can further analyze the details of this region and provide more specific suggestions and refined instructions, enabling the agent to interact with the physical world more effectively and precisely.

The success of ChatSpot hinges on three components: (1) We design a simple but effective precise referring instruction tuning method for MLLMs to support fine-grained and flexible human-AI interaction.

1. We construct a high-quality Multi-Grained VisionLanguage Instruction-following Dataset (MGVLID) including image-level and region-level multimodal SFT data with around 1.2M images and 3M query-answer pairs by collecting from existing datasets and generated based on GPT-4.
2. We design a series of evaluation tasks and metrics to assess the effectiveness of the proposed MLLM.

Extensive experiments have been conducted on a wide of vision-centric and vision-language benchmarks, and our ChatSpot shows excellent performance.

2 RelatedWorks

2.1 Large Language Models

In recent years, large language models (LLMs) have garnered considerable attention in the domains of natural language processing (NLP) and artificial general intelligence (AGI) owing to their remarkable performance in language generation, in-context learning, world knowledge, and logical reasoning. Early works, e.g., BERT [Devlin et al., 2018], GPT-2 [Radford et al., 2019] and T5 [Raffel et al., 2020] established the foundation architecture of LLMs. Then, with the release of GPT-3 [Brown et al., 2020], the first-ever language model to reach the parameter size of 175 billion, LLM achieved impressive zero-shot performance on various language benchmarks. Furthermore, researchers discovered emergent ability [Wei et al., 2022] in LLMs. That is when the model size of language models scales up to a certain level, there is a qualitative leap in the capabilities of language models. Sequentially, by aligning pre-trained GPT-3 to follow human instructions through Reinforcement Learning with Human Feedback

(RLHF) [Christiano et al., 2017], InstructGPT [Ouyang et al., 2022] and ChatGPT [OpenAI, 2023a] showcased powerful capabilities for human-AI interaction, which make LLMs reach its "iPhone moment". Inspired by the great success of GPT series, many other open-sourced LLMs, such as OPT [Zhang et al., 2022b], LLaMA [Touvron et al., 2023], and GLM [Zeng et al., 2022], have been proposed, which achieve similar performance to GPT-3. Based on these opensourced LLMs, several specific fine-tuned models are proposed to construct LLMs for various applications. For instance, Alpaca [Taori et al., 2023] proposes a self-instruct framework based on LLaMA [Touvron et al., 2023] and employs 52K instructions generated by ChatGPT [OpenAI, 2023a] to construct an exceptional dialogue model.

2.2 Human-AI Interactivity

Once we have sufficiently powerful LLMs, the next step is to figure out how to make these large models better meet human needs, with human-AI interaction being the core issue, that is, how to enable large models to more efficiently receive and understand human instructions. The powerful zero-shot and logical reasoning ability of LLM makes it the central controller of the interactive system to schedule various application tools for different modality tasks, such as VQA, image editing, and image captioning. There are two mainstream interactive styles, i.e., plugin-based and end-toend interaction. Plugin-based methods [Wu et al., 2023; Yang et al., 2023b; Shen et al., 2023; Liang et al., 2023; Yang et al., 2023a] usually prompt LLM (ChatGPT [OpenAI, 2023a], GPT-4 [OpenAI, 2023b] or LLaMA [Touvron et al., 2023]) to invoke different plugins from other foundation or expert models to perform specific functions according to human instructions. However, despite plugin-based methods that enable diverse applications, they are limited in the effectiveness of plugin invocation and the performance of the plugin model. On the contrary, end-to-end interactive systems usually use a single large multimodal model to accomplish interaction. This approach takes advantage of crossmodal transfer, aligning multimodal domains to a common language semantic space and then using autoregressive language models as decoders to output the language. Follow- Full image: "<System prompt> /n <Human>: Given an image <image>. Describe the whole image in details." Special Region: "<System prompt> /n <Human>: Given an image <image>. What is the object in this region?<x1><x2><y1><y2>"

ing this pipeline, Flamingo [Alayrac et al., 2022] developed a gated cross-attention trained on billions of image-text pairs to align vision and language modality, which shows strong performance in few-shot learning. BLIP-2 [Li et al., 2023] introduced Q-Former to align visual features with language space more effectively. More recently, LLaVA [Liu et al., 2023b] and LLaVA-like methods [Yu et al., 2023; Wei et al., 2024; Dong et al., 2023] proposed to use a linear layer to replace Q-Former and design a two-stage instruction-tuning procedure. Although existing end-to-end methods achieve remarkable performance in high efficiency, they are all limited to the interaction form of the full image and language-only instruction, which can not satisfy the demand for the specific region interaction. In this work, we build an end-to-end unified multimodal language model that supports a range of interaction forms that supports both full images and specific region.

3 Methods

3.1 Overall Architecture

As illustrated in Fig. 2, ChatSpot consists of an image encoder, and a decoder-only LLM, and a modality alignment block. Inspired by LLaVA [Liu et al., 2023b], ChatSpot incorporates a simple multilayer perceptron (MLP) to align the visual tokens with the space of language. The overall architecture is simple and does not use any extra AI models or post-processing operations. Different from previous end-toend interactive systems [Liu et al., 2023b; Zhu et al., 2023] that only support full image interaction, ChatSpot presents a more flexible interaction that supports users in further selecting the region of interest (RoI) to issue finer-grained instructions.

When an image I is uploaded by users to the ChatSpot system, users can use a mouse to select the RoI (points or boxes) Rt through a series of gestures, such as clicking and drawing boxes, and give some language instructions Xinstruct about these RoIs. The system then converts the position of Rt into the region prompt and connects it with the Xinstruct to generate the precise referring instructions. Afterward, the image I is inputted into the visual encoder to extract visual tokens. And then the modality-align projector transforms the visual tokens to the language semantic space. After obtaining the refined visual tokens and precise referring instructions. LLM decoder F takes them as inputs and generates the response language sequence Y autoregressively. Formally,

where V are aligned visual tokens. G is the visual encoder and ζ denotes the vision-language alignment projection. F is the large language decoder. Φ(·) is the normalization operation.

3.2 Precise Referring Instruction

Due to the inherent semantic unit mismatch between images and texts, it is ineffective to directly use the whole image and language sentences to describe the vision-language task. To this end, we propose precise referring instruction that enables the unification of multi-grained vision-language task descriptions and supports proxy interaction forms. Specifically, we divide the instructions into two types, i.e., image-level instructions and region-level instructions.

Image-level Instructions. The image-level instructions are usually used to describe the task of the whole image, and existing multimodal instructions mostly adopt this form. For instance, given an image and we want to know what the content of the image is. Then the instruction can be like "Given an image <image>. Describe the whole image in detail", or "Given an image <image>, please tell me: <question>", where <image> is the input image and <question> denote some relative questions about images. Afterward, the LLM ingests the whole sentence and outputs the response.

Region-level Instructions. Compared to the global information about the whole image, we often pay more attention to the information in specific regions. Therefore, it is valuable to design effective region-level instructions for MLLMs. The key challenge of region-level instructions is how to make LLM aware of the specific location of the region of interest. Here, we provide a simple but effective instruction format to achieve it. Specifically, we first define a unified region representation format as a tuple Rt = {xk, yk}Nk=1 that represents N points located in the selected region. Then the coordinates of selected points are normalized to [0, 1] and transferred to the text tokens as Φ(Rt). Finally, the region coordinate tokens are connected with the language instructions to generate the final region-level instructions. ( [Liu et al., 2023b] and [Bubeck et al., 2023] have served as evidence that LLM possesses the capability to comprehend spatial relationships and coordinates based on textual descriptions.) A simple example of region-level instruction is as follows: "Given an image <image>, What is the object doing in the region? <region>" where <region> = <box>Φ(Rt)</box>, <box> and </box> are special tokens to tell the LLM that this is a set of coordinates of the RoI. Notably, the number of selected points N is set freely so that we can achieve multigrained interaction, such as points, boxes, and polygons.

3.3 Multi-grained Vision-language Instruction-following Dataset

In order to empower ChatSpot with the precise referring instruction following ability, we design a data collection pipeline with the assistance of GPT-4 [OpenAI, 2023b], as shown in Fig. 3. Inspired by LLaVA [Liu et al., 2023b], we use the captions and bounding boxes of the target as the prompts and leverage GPT-4 to refine these captions and generate more relative and diverse conversation data. The difference is that our approach distinguishes itself by enforcing the alignment of every generated dialogue with precise regional coordinates. To achieve this, we leverage the VisualGenome dataset [Krishna et al., 2017], which provides comprehensive annotations of objects, attributes, and relationships within each image, enabling us to construct region-level instruction following datasets. These datasets consist of dense regionwise captions organized alongside carefully curated seed examples, which are used to query GPT-4 in an in-contextlearning fashion. Through this pipeline, we have successfully gathered a total of 108K region-level instruction following samples. Based on this data generation pipeline, we build a high-quality Multi-Grained Vision-Language Instructionfollowing Dataset, named MGVLID. MGVLID consists of two main parts, i.e., image-text instruction-following data and region-text instruction-following data. The former data consists of the image and the caption of the entire image, while the latter consists of the image, the bounding boxes of the target in the image, and the target captions. As shown in Fig. 3, the whole MGVLID covers 11 source datasets and we hold out 4 datasets for model evaluation purposes.

Image-level Instruction Data. To gather image-level instruction-following data, we collect a wide range of publicly available multimodal datasets that have been humanannotated. We then transform these datasets into a unified instruction-following format. Specifically, we assemble a plethora of commonly used Question-Answering (QA), captioning, and object detection datasets, including CC595K (filtered based on CC3M [Sharma et al., 2018]), OCRVQA [Mishra et al., 2019], ST-VQA [Biten et al., 2022], DocVQA [Mathew et al., 2021], TextVQA [Singh et al., 2019] and Object365 [Shao et al., 2019]. For each dataset, we design a series of unique instruction templates. These templates are subsequently carefully filtered and refined manually to ensure optimal rationales and diversity of the conversation. Due to the considerable differences in label lengths among various task datasets (such as caption or category words), we incorporate additional instruction tags to specify the desired response style. For instance, we include tags like "answer in shot" for short-answer data and "answer in detail" for long-answer data.

Region-level Instruction Data. While image-level instruction data empowers the model's global visual perception and human instruction-following ability, region-level instruction data offers region-level observation and more fine-grained instructions, enabling the model to further acquire spatial perception and reasoning abilities. In order to construct region-level instruction datasets, we first collect region-text pairs based on existing region-level task (object detection and OCR) datasets, i.e., Object365 [Shao et al., 2019], COCO text [Veit et al., 2016], HierText [Long et al., 2022] and Art [Chng et al., 2019]. We collected region-text pairs that consist of instance-level bounding boxes and their corresponding content. Subsequently, we utilize unique instruction templates to further refine these region-text pairs, resulting in a series of questions and answers. Furthermore, we also collect the PointQA datasets from LookTwice-QA [Mani et al., 2020] to support point-wise referring instruction tuning, where the models are asked to answer questions based on the input points or boxes. By incorporating the high-quality dense region chatting data generated based on GPT-4, the final region-level instruction data is constructed.

4 Experiments

4.1 Implementation Details

To build ChatSpot, we implement the CLIP ViT-L/14 [Radford et al., 2021] as the visual encoder to encode images. For the large language model, we choose open-sourced Vicuna7B [Chiang et al., 2023] as the language decoder, a LLaMA model fine-tuned with instructions. For alignment projection, we just adopt a simple linear layer to connect vision and language embedding space.

Inspired by LLaVA, the model is trained in a two-stage fashion. Firstly, we initialize the model using pre-trained weights from LLaMA and CLIP ViT. During this first stage, we only train the projection layer. Meanwhile, we freeze the majority of the LLM parameters. In this stage, we mainly use the image-text instruction-following data of MGVLID to train the model for vision-language instruction-following alignment learning. In the second stage, we only freeze the visual encoder and unfreeze the LLM parameters. In this stage, we mainly use the region-text instruction-following data including RegionChat to train the model for region-level instruction-following and multi-turn chatting ability. Specifically, the model is fine-tuned over 3 epochs, with a batch size of 128. AdamW [Loshchilov and Hutter, 2019] optimizer is employed, and the learning rate is set to 2e - 3 in the first training stage and 2e - 5 in the second training stage. For LLM, the maximum length of tokens is set to 2, 048.

4.2 Task Evaluation

In this part, we select several downstream tasks and general multimodal benchmarks to showcase ChatSpot's region recognization and zero-shot ability. Notably, all the experiments are conducted by the shared-parameter generalist model, and we just change the language instructions for different tasks.

Regional Classification. Object detection is a fundamental vision task that consists of object location and recognition subtasks. In this part, we mainly evaluate the regional classification ability of ChatSpot in COCO [Lin et al., 2014], which is a common dataset in object detection tasks. Specifically, we first use the GT boxes or the bounding boxes generated by SOTA detectors, such as DINO [Zhang et al., 2022a], as region prompts to ask ChatSpot to answer what category it is. For an example, we use "What can you see in this region? <region>", where <region> denotes the coordinates of the region boxes. Then we compute the metrics of Average Precision (AP) and Accuracy about the bounding boxes with the predicted classes. Notably, due to ChatSpot's output typically being a single sentence, it cannot be directly used as a category for evaluation. Here, we employ the CLIP text encoder to calculate the text feature similarity between the output and all COCO categories for category determination.

As shown in Table 1, we randomly select 1, 000 images from the COCO validation set, namely COCO-1000, to evaluate ChatSpot for efficiency. Our ChatSpot achieves 64.5% accuracy on COCO-1000 with the provided GT boxes. When given DINO-generated bounding boxes as region prompts, our ChatSpot achieves 39.6% AP, which is also a competitive performance. Notably, We do not use any annotations from COCO. The results show that ChatSpot achieves impressive zero-shot classification ability in region-level recognization. Regional Optical Character Recognition. Optical Character Recognition (OCR) is a visual entity recognition task that requires the recognition of the graphemes in a written text. In this part, we select COCO text [Veit et al., 2016] to evaluate the regional text recognization ability of ChatSpot. COCO Text is a large-scale dataset for text detection and recognition. We first use the provided region boxes of the datasets as the region referring and ask the ChatSpot "What text can you see in this region?"<region>. Then ChatSpot will respond to the specific answer. When the answered sentence includes the correct GT answer, we consider ChatSpot's response to be correct. As shown in Tab. 2, our ChatSpot achieves 31.8% accuracy on the COCO Text validation set.

Referring Expression Comprehension (REC). REC is also an important task for evaluating the regional referring ability of the model. We have additionally included experiments on the RefCOCO dataset and conducted a comprehensive comparison with existing SOTA solutions including traditional CV methods and LLM-based methods. As shown in Tab. 3, ChatSpot achieves promising performance compared to other generalist models, which shows the powerful spatial perception and referring instruction ability.

Visual Question Answering (VQA). VQA is the task of answering open-ended questions based on the whole image or the region of interest (RoI), which is well-suited for evaluating the perceptual and reasoning abilities of MLLMs in understanding image content. In this part, we choose two datasets to evaluate ChatSpot. One is VizWiz-VQAGrounding dataset [Gurari et al., 2018], a dataset that visually grounds answers to visual questions asked by people with visual impairments. Another is PointQA [Mani et al., 2020], a set of datasets that require a pointer to an object in the image to be answered correctly. These two datasets both provide specific regions (boxes or points) and ask the question about the corresponding area. Therefore, it requires the model to possess the ability of region-level perception and reasoning ability. Specifically, given a region of interest and question, we first input the coordinate of region boxes to ChatSpot with the question. If the output answer sentence of ChatSpot includes the GT answer, we consider the response to be correct. As shown in Tab. 2, benefiting from ChatSpot's strong regionlevel instruction following abilities, the model achieves competitive performance that obtains 63.0% accuracy on VizWiz and 68.2% accuracy on PointQA (boxes given) and 62.0% on PointQA (points given).

General Multimodal Understanding. In order to showcase the general multi-modal ability of ChatSpot, we further evaluate ChatSpot on recent benchmarks proposed for evaluating the comphrehensive capabilities of MLLMs. As shown in Tab. 4, we select several mainstream MLLM benchmarks including MMBench, MMVet and POPE to evaluate ChatSpot. On MMBench, we report results on the both development and test sets. On POPE, we mainly report the results of "random" setting. For fair comparison, we utilizd the SFT data of LLaVA-1.5 (LLaVA665K). It is encouraging that ChatSpot outperforms the strong baseline LLaVA-1.5 across all three benchamrks. The experimental outcomes demonstrate the superior comprehensive capabilities of ChatSpot, which means powerful referring instruction ability is also beneficial to the general multi-modal understanding and reasoning.

4.3 Qualitative Analysis

In order to provide a comprehensive showcase of ChatSpot, we selected several classic cases to demonstrate its specific abilities. We specifically demonstrate four core capabilities of ChatSpot through these examples as follows:

Region Perception Ability. As shown in Fig. 4 (left top), we first show the case that depicts ChatSpot's ability to perceive the region of interest and recognize the corresponding context. In this case, ChatSpot can identify the selected area of different levels of granularity, e.g., the head of the brown bear and the nose of the brown bear. It can also perceive some specific features of the target like the wet hair of the bear. By collaborating with human referring prompts (point or boxes), ChatSpot showcases its powerful capability in perceiving details, which provides sufficient detailed information for robots to perform more refined operations.

Content Generation Capability. ChatSpot also possesses a powerful content generation capability related to regions of interest, as illustrated in Fig. 4 (left-right). In this case, ChatSpot first recognizes the subject object (lemon) in the region of interest. Then it can also generate responses for other information that cannot be captured in the image, i.e., the precautions when cutting a lemon with a knife. Thanks to the huge knowledge of LLMs, ChatSpot can provide rich content explanations or suggestions based on visual information.

Optical Character Recognition (OCR) Ability. ChatSpot also achieves impressive performance in recognizing the optical character, e.g., text, number, and signal. As shown in Fig. 4 (bottom left), ChatSpot can accurately identify the number written on the signboard and analyze the purpose of these numbers based on the global context information.

Special Reasoning Ability. In addition to perception, another important capability of ChatSpot is spatial reasoning which can further analyze the region of interest based on its knowledge after recognition, which is important for robotics and automation. As shown in Fig. 4 (bottom right), ChatSpot first identifies the fridge and then determines that the refrigerator is powered on according to the power wine being plugged into the power strip and connected to the refrigerator. Furthermore, ChatSpot provides a series of detailed action instructions when the user wants to take a Coke from the fridge. This enables the robots to be able to make further decisions regarding fine-grained operations after inferring the specific status (fridge is plugged in.) of the RoI.

evaluate existing multimodal large language models? These limitations require further research in the future.

6 Conclusion

In this work, we first propose precise referring instruction tuning for MLLMs that utilizes diverse reference representations for referring special regions. Based on precise referring instruction, we build ChatSpot, a fully end-toend MLLM that supports diverse region referring prompts, i.e., points, and boxes. Then we construct a large-scale multi-grained vision-language instruction-following dataset, MGVLID. Trained on MGVLID, ChatSpot demonstrates outstanding performance both in interactive chatting and downstream tasks. Results suggest that combining precise referring instructions with MLLMs stimulates the model's ability for special region understanding and reasoning.

Acknowledgements

This work was supported by National Science and Technology Major Project of China (2023ZD0121300).

5 Discussion

In this section, we are dedicated to delving deeper into the capabilities and features of ChatSpot, as well as identifying the limitations that currently hinder further enhancement of ChatSpot's abilities.

5.1 Robustness on Region Referring

The process of selecting regions, whether through drawing boxes or clicking, plays a crucial role in ChatSpot. However, users often face difficulties in accurately annotating their areas of interest. In such instances, it is essential for ChatSpot to exhibit a high level of robustness in region selection. Hence, an analysis of the robustness of ChatSpot in the region referring is conducted. Specifically, we randomly introduce box noises of different scales (scale = 0.1, 0.2, and 0.3) to the region boxes of the COCO and VizWiz as illustrated in Fig. 5 (left). Fig. 5 (right) demonstrates that the performance of ChatSpot on COCO and VizWiz does not show a significant decrease after introducing box noises into the region bounding box, which means ChatSpot possesses strong robustness in region referring.

5.2 Limitations

Although ChatSpot has achieved remarkable performance in precise region referring and special region reasoning, it still has some noticeable limitations. For example, ChatSpot currently lacks the ability to support the recognition of certain special symbols, such as license plate numbers. These shortcomings can be attributed to insufficient training data. Furthermore, it is hard to evaluate the region recognition ability of multi-modal large language models since its evaluation is essentially different from traditional visual-language models. Though we take the first step to quantitatively evaluate it by giving pre-defined boxes, it is still an open problem: how can we establish a comprehensive and automatic benchmark to

Contribution Statement

In this work, Liang Zhao and En Yu are co-first authors, and Zheng Ge is the corresponding author.