An Entanglement-driven Fusion Neural Network for Video Sentiment Analysis

1 Introduction

Video sentiment analysis is an emerging interdisciplinary area in multimedia information processing, bringing together Artificial Intelligence (AI) and cognitive science. In particular, it studies a speaker's sentiment expressed by verbal (i.e., linguistic) and non-verbal (i.e., visual, acoustic) streams. At its core, this research area focuses on modelling interactions among distinct modalities, a.k.a., crossmodal dynamics. In recent years, research has made significant strides towards the inference of human-like sentiment judgments. In particular, neural approaches have been investigated to model crossmodal interactions by operating primarily feature-level fusion [Tsai et al., 2019; Liu et al., 2018; Zadeh et al., 2017]. However, current models neglect other important aspects, such as model transparency, post-hoc interpretability, and how people would understand and reason about sentiment states.

∗Contact Author

†Contact Author

The modelling of distinct modalities for sentiment analysis is a challenging problem. This is due to the spectrum of sentiment polarities that an utterance can emerge (e.g., positive, neutral, or negative), depending on the context of individual modalities. For instance, let us consider that we want to identify the binary sentiment of the utterance "Well, what a surprise!". The utterance's sentiment state is ambiguous, i.e., it could be either positive or negative, since there is no specific context to bias the sentiment state of the utterance. However, once the context (i.e., other modalities) is informative, unambiguous, and simultaneously present, the sentiment state to a particular sentiment judgment becomes apparent. This implies that we shall not consider distinct modalities in isolation. Rather, we must model modalities as non-separable, called entangled. Quantum Theory (QT) is the only theory which models non-separability. Thus, there is a reason to suppose that QT provides an adequate theory to capture the crossmodal correlations and how such correlations influence the final decision about the utterance's sentiment.

Quantum probability theory has been extensively studied in the domain of human cognition and decision making [Busemeyer and Bruza, 2012]. In particular, it has been shown that in some cases human language understanding exhibits certain non-classical properties [Bruza et al., 2008; Bruza et al., 2009], enabling quantum probabilities a suitable framework. Recent advances in quantum probabilistic neural models [Li et al., 2019; Li et al., 2021] have demonstrated improved performance and high-level explainability due to their theoretical root on the well-established quantum physics meanings. Nevertheless, they treated the interactions among quantum states as either a classical mixture of states [Li et al., 2019] or a separable product of states [Li et al., 2021], which cannot fully exploit the potentials of quantum probabilities in modelling the non-separability of multiple modalities. The expressiveness of quantum probabilities goes beyond classical correlations.

In line with the above observation, we propose a quantum probabilistic neural network, which captures non-classical correlations of sememes across distinct modalities. In particular, we transform the real-valued input features of different modalities into pure quantum states of complex values. The particular way in which the modality states interact with each other allows modelling of both classical correlation and non-separability (entanglement) between modalities in a unified framework. Our work is fundamentally different from the previous probabilistic neural network approaches in that we address the issues of contextuality, i.e., a modality activates ambiguous sentiment polarities in the context of other modalities, and non-separability, i.e., a modality cannot be separated from the rest of modalities occurred concurrently. The proposed model is empirically evaluated on two benchmark datasets for video sentiment analysis. Experimental results show that our model significantly improves performance over a wide variety of state-of-the-art (SOTA) approaches. We also show that the degree of non-separability of entangled states can be used to improve the post-hoc interpretability.

2 RelatedWork

The application of quantum theory in representation learning began after van Rijsbergen's pioneering work [Van Rijsbergen, 2004] by integrating geometric spaces, probabilities, and logic into a unified theoretical framework. Then, the probabilities of quantum theory were exploited for various representation learning tasks [Uprety et al., 2020]. Among them, quantum formalism was successfully utilised for modelling word dependencies through density matrices [Sordoni et al., 2013] and formulating the semantic composition of words [Sordoni et al., 2013] in IR tasks. Quantum-inspired models were also introduced to address NLP tasks. Recently, the deployment of quantum measurement into a joint complex-valued neural network led to improved performance and better interpretability [Li et al., 2019].

Quantum-inspired strategies were also investigated for multimodal representation learning. Early work exploited a tensor-based representation for an image-text IR task [Wang et al., 2010]. Preliminary work investigated non-classical correlations, i.e., entanglement, based on the combination of uni-modal decisions [Gkoumas et al., 2018]. The notion of quantum incompatibility has been exploited to fuse decisions from different modalities for video sentiment analysis [Gkoumas et al., 2021a]. Recently, a quantum-inspired network for videos sentiment analysis exploits an early fusion of tri-modals via the tensor product of modalities [Li et al., 2021]. However, the interactions across different modalities are largely neglected, assuming the interactions are decomposable. Instead, inter-modal interactions are implemented in the sentiment decision process. Our model is fundamentally different from [Li et al., 2021] in that we take a quantumcognitively motivated view on the non-decomposability of cross-modality interactions, which is modelled as quantum entanglement.

3 Preliminaries on QuantumTheory

In this section, we present fundamental concepts of quantum theory (QT) [Melucci, 2015; Busemeyer and Bruza, 2012] that we exploit to construct the quantum probabilistic neural model. In consistency with the convention of QT, we adopt the widely-used Dirac Notations, known as "bra-ket" notation. A complex-valued unit vector ⃗u and its conjugate transpose ⃗u∗T are denoted as a ket |u〉 and a bra 〈u|, respectively. The inner product of two vectors |u〉 and |v〉 is defined by 〈u|v〉, while |u〉 〈u| and |v〉 〈v| define operators.

The starting point to modelling quantum states is a set of basis states. A basis is a set of n mutually orthogonal vectors {|ej〉}nj=1 of unit length. The vector space employed in QT is a vector space over complex numbers, called Hilbert space H, offering the structure of an inner product to enable the measurement of angles and lengths [Halmos, 1987].

Any pure state |s〉 of a quantum system is regarded as a linear superposition, i.e., an appropriate weighted sum, of one set of n basis states, represented by a unit vector in Hn . That is, the pure state |s〉 can be written as a probability distribution of complex probability amplitudes, as follows:

where √rjeiφj correspond to complex probability amplitudes, rj are non-negative scalars ∈ R satisfying ∑nj=1 |rj | = 1, i is imaginary number, and φj are phases ∈ [0, 2π].

Quantum measurement with respect to a basis (i.e., a set of eigenvectors), yields one out of all the observable eigenvalues and causes the collapse of the state to the corresponding eigenvector. The probability of a given outcome is obtained via the projection postulate. That is, according to the Born's rule [Halmos, 1987], the probability of the pure state |s〉 to collapse onto the basis state |ej〉 is calculated by the inner product of the two vectors as follows:

A composite quantum system describes a compound system composed of multiple individual quantum systems. For two nA-dimensional and nB-dimensional spaces, the state vector sc ∈ A ⊗ B is expressed as a linear combination of an arbitrary basis of the product space |i〉 ⊗ |j〉, as follows:

where √ci,j are probability amplitudes so that ∑i,j ci,j = 1. When a composite quantum system evolves under a Hamiltonian that includes interactions between individual subsystems, the resulting state of the composite system is no longer separable, a.k.a. entanglement [Melucci, 2015]. That is, Eq. 3 cannot be expressed as a tensor product of individual systems. The bipartite Von Neumann entanglement entropy [Vivo et al., 2016] is a measurement of the degree of quantum entanglement for a composite pure state. For instance, for an arbitrary bipartite quantum state consisting of two sub-states, i.e., A ∈ HnA and B ∈ HnB , the Von Neumann entropy is calculated as follows:

where ai are singular values of the Schmidt decomposition of the bipartite quantum state over the sub-system A or B. Eq. 4 makes clear that the entanglement entropy is the same regardless of whether the decomposition is over A or B sub-system. Crucially, if the entropy S is zero, there is no entanglement.

4 Task Formulation

The goal is to infer the sentiment of video utterances. The multimodal dataset consists of N labeled video utterances U = (U1, ...,Ui, ...,UN). Each utterance Ui is associated with linguistic, visual, and acoustic features, denoted as Ui = (U l i , U v i ,U a i ). The corresponding labels for the N utterances are denoted as y = (y1, ..., yi, ..., yN), yi ∈ R. Essentially, the objective is to establish a function, mapping each video utterance Ui to its corresponding sentiment label yi.

5 Entanglement-driven Fusion Neural Network

We propose a quantum probabilistic neural model, namely, Entanglement-driven Fusion Neural Network (EFNN).

5.1 General Architecture

The architecture of EFNN is illustrated in Fig. 1. In particular, we deploy an intermediate fusion strategy into a neural network modelling paradigm that incorporates QT-inspired complex representation of information, composite quantum system and entanglement to capture the non-separability of modalities, and quantummeasurement for abstract sentiment concept extraction.

Specifically, EFNNfirst takesmultimodal information, i.e., linguistic, visual, and acoustic, and feed it into three separated neural branches, one for each modality (see Fig. 1). At the outset, multimodal information is projected into a common-dimensional space, and then a preparation step converts the information to its quantum analogues, i.e., quantum states. Afterwards, we operate a pairwise fusion of modalities, i.e., linguistic-visual, linguistic-acoustic, and visualacoustic, via the tensor product of bi-modals (any of two modalities). A weight vector captures the correlations within the bi-modal tensor-based representations (see Fig. 1, Entangled Bi-Modals step). A set of parameterized measurements map the complex-valued representation to a real-valued highlevel representation via the quantum measurement postulate. Then, a row-max pooling operator is applied, followed by a fully connected layer passed to a softmax function for classification. In the remaining part of the section, we elaborate on the methodology according to the above procedural steps.

5.2 Preparation of States

In this work, each utterance is modelled as a uni-modal pure quantum state into modality-specific Hilbert spaces Hm, where m ∈ {l, v, a}, for linguistic, visual and audio modalities. In line with previous works [Li et al., 2019], we consider the exponential form of complex numbers to express quantum states: z = reiθ , where amplitude r is a real non-negative coefficient, phase θ ∈ [0, 2π), and i is imaginary number satisfying i2 = -1.

Then, according to Eq. 1, the modality-specific pure state of an utterance |um〉 could generally be expressed by the following modulus-augment form:

where d is the dimension of modality features and ⊙ refers to element-wise vector product. In the modulus-argument form, any operation on the complex numbers will lead to a nonlinear combination of the constituent moduli and arguments. This implies that a non-linear feature combination is inherently produced when we assign Eq. 5 with linguistic, visual, and acoustic features.

In Eq. 5, the first vector, i.e., rm = [r1,m, r2,m, ..., rd,m]T , corresponds to amplitudes, where the moduli r is a realvalued vector of unit length. To construct amplitudes, we transform the input real-valued features to their quantum analogues as follows. Suppose the input word-level features are Xl ∈ RL×dl , Xv ∈ RL×dv , Xa ∈ RL×da , where dl, dv, da represents feature dimensions for linguistic, visual, and acoustic modalities respectively, and L is the sequence length, i.e., total number of words in an utterance. At the outset, we project the input features into the same dimension d via convolutional neural networks [Lai et al., 2015] from the respective input features with Rectified Linear Unit (ReLU) as the activation function in the last hidden layer, to ensure all elements {ri,m}di=1 are non-negative: ˆm = ReLU(CNNm(Xm)) ∈ Rd , where m ∈ {l, v, a}. Despite the projection of modalities into a common-dimensional space, the convolutional neural networks CNNm capture local structure of words in an utterance. Then, we normalize the outputs to create vectors of unit length: rm = ˆm || ˆm||2 .

The second vector, i.e., θ = [θ1,m, θ2,m, ..., θd,m]T , is also real-valued, with all its elements in [0, 2π]. The assignment of the phases θ is an open research question. In this work, to enable each utterance to carry temporal information, i.e., we assign the position of words in a sentence to the phase part. With this way, we capture the global structure of words in an utterance. The phase θ is hence calculated by

where fpe(k) defines a map fpe : N → Rd from a discrete position index to a d-dimensional real-valued vector.

5.3 Entanglement-drivenModality Fusion

After the transformation of feature inputs to quantum states into uni-modal Hilbert spaces, we feed them into the modality fusion component (see Fig. 1). In particular, we deploy a fusion module, which takes the utterance states of pairwise modalities, i.e., linguistic-visual, linguistic-acoustic, visual-acoustic. For each pairwise of states, a composite but separable state is created by computing the tensor product of them. The composite separable state is defined on a d2-dimensional joint space Hm1,m2 := ⊗2(Hm)2 and formulated as

where m1,m2 are any of two modalities, and ⊗ defines the outer product of two states.

Then, a complex-valued neural layer W is injected to induce interactions of pairwise modalities (see Fig. 1), as follows:

where W in H ∈ Rd2 is a shared weight vector, and ⊙ stands for element-wise vector product. The output is an unnormalized vector |̂ um1,m2 〉, which is then normalized to get a unit vector in Hm1,m2 ∈ Rd2 , i.e., a valid quantum state: | ̂ um1,m2 〉 = | ̂ um1,m2 〉 || ̂ um1,m2 ||2 , in short |̂ um1,m2 〉 = |um1,m2 〉.

From the representation point of view, Eq. 7 can be considered as a weighted linear transformation layer. From the quantum point of view, W can be realized as a unitary operator U [Banchi et al., 2018]. Throughout the bipartite modality interaction process, W acts as a quantum Hamiltonian control on different Hilbert spaces, i.e., Hm1 , Hm2 , and entanglement is hence generated after the transformation. This means that the output after the transformation cannot be written in the decomposable form, thus giving the potential to capture nonclassical correlations across pairwise modalities.

5.4 Measurement

The measurement component acts upon the set of three nonseparable pairwise modalities to identify the discriminating information for sentiment classification. In particular, a set of parameterized measurements {Ok}Kk=1 are performed on the set of non-separable pairwise modalities (see Fig. 1), generating a sequence of positive scalars for each pair of modalities,

where m1,m2 are any pair of modalities and each Ok represents an abstract sentiment concept. The output is a K×3 matrix of positive real values produced by measurement. Each value corresponds to the likelihood of a non-separable pairwise modality state collapsing to a basis state Ok, which is in effect a basis context representing abstract sentiment concepts. Note that the measurement component can be thought of as a dictionary learning approach.

Then a row-wise maximum pooling operator is conducted to cascade the three sequences of abstract concepts into one high-level utterance representation (see Fig. 1). Finally, the high-level representation is passed to a fully connected layer followed by a softmax classifier.

6 Experiments

6.1 Experimental Settings

We performed experiments on two widely used benchmarking video sentiment analysis datasets: CMU Multimodal Opinion-level Sentiment Intensity (CMU-MOSI) [Zadeh et al., 2016], and the largest available dataset for multimodal sentiment analysis, CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) [Zadeh et al., 2018c]. We refer the curious readers who are interested in the feature extraction process to [Gkoumas et al., 2021b].

In line with [Gkoumas et al., 2021b], we chose EarlyFusion LSTM (EF-LSTM) and Late-Fusion LSTM (LFLSTM) as baselines, Multi-Attention Recurrent Network (MARN) [Zadeh et al., 2018b], Memory Fusion Network

(MFN) [Zadeh et al., 2018a] and Contextual GRU with Attention (c-GRU) [Ghosal et al., 2018] as advanced LSTM approaches, Tensor Fusion Network (TFN) [Zadeh et al., 2017] and Low-rank Multimodal Fusion (LMF) [Liu et al., 2018] as tensor-based fusion approaches, Multimodal Transformer (MulT) [Tsai et al., 2019] as a seq-to-seq approach, and QMF [Li et al., 2021] as a quantum-inspired fusion approach.

To evaluate the effectiveness of our model on CMU-MOSI and CMU-MOSEI tasks, we adopted a series of evaluation performance metrics used in prior work [Gkoumas et al., 2021b; Zadeh et al., 2018c; Zadeh et al., 2018a], including: binary accuracy (i.e., Acc2 : positive sentiment if values ≥ 0, and negative sentiment if values < 0), 7-class accuracy (i.e., Acc7 : sentiment score classification in Z ∩ [-3, 3]), F1 score, Mean Absolute Error (MAE) of the score, and the Pearson's correlation (Corr) between the model predictions and regression ground truth. For all the metrics, the higher values denote a better performance, except MAE where the lower values denote better performance.

A grid search for the best hyper-parameters was conducted for all models. At each search, the models were trained for 100 epochs. Out of 50 searches, the model with the lowest validation loss was used to produce the test performance. The parameters in the proposed EFNN model were determined by the set of hyper-parameters Θ = {D, K}, where D is the embedding dimension of input features into same dimensional spaces and K is the number of measurement vectors. For both datasets, we searched over a parameter pool.

We trained EFNN by feeding the real and imaginary parts of the complex-valued layers as different input parts and simulated complex operations using real values. EFNN was hence trained via the backpropagation algorithm. Measurements were initialized from standard normal distributions. All the parameters were trainable with respect to L1-loss defined on the extracted features. We chose Adam as the optimization algorithm.

6.2 Performance Analysis

Table 1 shows the comparison results between EFNN and the SOTA baseline approaches for the CMU-MOSI task. The approaches that apply attention mechanism to align pairwise modalities, i.e., c-GRU and MulT, exhibit the highest binary accuracy as compared to the rest of the baselines. TFN achieves the highest accuracy for Acc7 among the baselines.

Finally, the results show that the proposed EFNN is the most effective approach for the CMU-MOSI task. In particular, it achieves an increased binary accuracy 80.9% as compared to 78.7% of MulT, which is the next best model in terms of binary accuracy. That is a significant improvement of 2.7% (t-test<.05). Overall, EFNN shows performance improvements for all evaluation metrics.

Table 2 presents the results for the CMU-MOSEI task. All approaches attain an improved performance compared to that of the CMU-MOSI dataset. We suspect this is because CMUMOSEI is a much larger dataset. c-GRU is the most effective model among the baselines for the CMU-MOSEI task. MulT achieves similar performance to c-GRU, without a significant difference. EFNN gains an increased binary accuracy of 82.8% as compared to 80.7% of c-GRU, which is a significant improvement of 2.6% (t-test<.05). Finally, EFNN achieves an improvement for all evaluation metrics on CMUMOSEI.

In summary, EFNN significantly outperforms the baselines for both CMU-MOSI and CMU-MOSEI tasks. The analysis of results has shown that EFNN is capable of coping with both balanced and skewed datasets.

6.3 Ablation Test

We also carried out an ablation test on CMU-MOSEI to investigate the effect of introduced quantum components. In particular, to examine the effectiveness of convolution neural networks CNNm projecting modalities to commondimensional spaces, we replace the component with GRU layers (a.k.a. EFNNgru). Furthermore, we would like to investigate the impact of non-separable modalities, by introducing two other variants of EFNN, after removing the weight vector W (see Figure 1): a) EFNNtensor fuses all modalities into a unified tensor-based representation, i.e., trimodal fusion; and b) EFNNcon concatenates all modalities into a vector representation, and then the outputs interact with the measurement component. Moreover, we also consider the impact words' position in an utterance by initializing phases from standard normal distributions (a.k.a. EFNNrand). We finally replace the measurements with a convolutional neural network (CNN), whereby the K filters of CNN serve as K measurements, in order to investigate the impact of the measurement component (a.k.a. EFNNcnn).

The results of the ablation test, illustrated in Table 3, show that each component plays a crucial role in the EFNN. In particular, the comparison with EFNNrand shows the effectiveness of modelling words' position into the phase part of complex-valued representations. At the same time, the decreased performance of EFNNtensor and EFNNcon reveals the superiority of encoding the non-classical correlations (i.e., entanglements) between modalities. Moreover, the comparison with EFNNcnn shows the usefulness of trainable measurements. Finally, EFNNgru shows that convolutional neural networks could be a more appropriate way to project modalities into common-dimensional spaces. Overall, the ablation test reveals that the entanglement-driven fusion component plays the most crucial role in the architecture of EFNN.

6.4 Post-hoc Interpretability

Further, we evaluated the post-hoc interpretability by investigating the bi-modal correlations within composite utterance states after the modality context interaction. In particular, according to Eq.4, we calculated the degree of quantum entanglement for bipartite composite utterance states of linguistic and visual modalities.

Table 4 illustrates some examples of the most and least entangled linguistic-visual modalities according to entanglement entropy. The most entangled pairs are those that one of two modalities is ambiguous or uninformative. For example, in Table 4, the linguistic content of the first utterance is ambiguous, while the visual content of the second utterance is uninformative. By contrast, when the context of both modalities is informative, unambiguous, and simultaneously present (e.g., the last two utterances in Table 4), the entanglement entropy is close to zero. In those cases, the composite representation is separable, and there is no need to exploit the quantum probabilistic interpretation. However, through the concept of non-separability, EFNN is able to capture both separable and non-separable bi-modal interactions, as a generalization of existing probabilistic modality fusion approaches. This attribute is the core reason that EFNN has achieved performance improvement.

7 Conclusion

In this work, we have introduced a transparent and joint quantum probabilistic neural model for video sentiment analysis. The concept and formalism of non-separability as quantum entanglement to fuse bi-modalities enables the model to capture both classical and non-classical correlations between modalities. The information encoded in the non-classical correlations yielded significant improvements in performance over a range of strong baselines. Besides, non-classical correlations were quantified by an appropriate measure, which optimized post-hoc interpretability. In the future, we would like to extend the framework for video emotion detection in conversations that requires leveraging the conversational context and evolution of systems.

Acknowledgements

This work is funded in part by the National Key Research and Development Program of China (grant No. 2018YFC0831704), Royal Society project on Big Code (grant No. IES/R1/191138), EPSRC STRIDE project (grant No. EP/T017465/1), and the Trusted Programming Lab at Huawei Ireland Research Center.