Quantum-Inspired Interactive Networks for Conversational Sentiment Analysis

1 Introduction

Sentiment analysis (SA) targets at judging sentiment polarities for various types of texts at document, sentence or aspect levels [Tripathy et al., 2017; Yang and Cardie, 2014; Pontiki et al., 2016]. The recent boom of social network services produces a huge volume of textual records of communications between humans. Such data carry a rich source of information including sentiments or opinions, which often evolve during the conversation. It brings forth a new challenge of judging the evolving sentiment polarities of different people in a conversational discourse. Therefore, the research on conversational sentiment analysis has attracted an increasing attention from both academia and industry.

∗The corresponding author.

Conversational sentiment analysis aims to detect the affective states of multiple speakers during and after an conversation, and study the sentimental evolution of each speaker in the course of the interaction. The interaction dynamics in a conversation mainly consists of intra- and inter-utterance interactions. Intra-utterance interaction refers to the correlations between terms within an utterance, while inter-utterance interaction involves repeated interactions between the speakers' utterances. Fig. 1 provides an example from the MELD dataset [Poria et al., 2018], from which, we can notice that the evolution of Jen and Ross's affective states is influenced by both intra- and inter-utterance interactions.

Existing research in conversational sentiment analysis mainly focused on leveraging intra-utterance interactions, e.g., learning relations between words, extracting effective features, etc., to judge sentiment, while the inter-utterance interactions are largely neglected. For instance, Ojamaa et al. [2015] used a lexicon-based method to extract speakers' attitudes from conversational texts. However, they neglected the interactions and used only 23 dialogue files. Bhaskar et al. [2015] proposed to combine acoustic and textual features in audio conversations to enhance the efficiency of emotion classification. However, they did not consider interactions among speakers. Huijzer et al. [2017] performed affective analysis of emails. They did notice, but did not model the interaction between customer support and customers.

In recent years, quantum theory (QT), as a mathematical formalism to model the complex interactions and dynamics in quantum physics, has been adopted for constructing text representation in various information retrieval (IR) and NLP tasks [Sordoni et al., 2013; Wang et al., 2018; Li et al., 2018; Zhang et al., 2018c]. For instance, the Quantum Language Model (QLM) [Sordoni et al., 2013] represents a query or document as a density matrix on a quantum probabilistic space, and computes density matrix-based metrics as ranking function. the Neural Network based QLM (NNQLM) [Zhang et al., 2018a] builds an end-to-end network for question answering (QA) to jointly model a question-answer pair based on their density matrix representations. Zhang et al. [2018b] leverages an improved version of QLM for twitter sentiment analysis. Such QT-based models can be considered as a generalization of classical approaches in that they are capable of capturing inherent intricacies in interactions. This motivates us to explore the use of quantum theory as a theoretical basis for capturing the intra- and inter-utterance interaction dynamics, both of which are complex in nature.

In this paper, we propose quantum-inspired interactive networks (QIN) that jointly captures intra- and inter-utterance interactions for conversational sentiment analysis. The model extracts textual features with a density matrix based CNN (DM-CNN) to capture the intra-utterance correlations between words. The inter-utterance interaction is extracted by a strong-weak influence model, which is inspired by quantum measurement theory, to measure the influence between speakers across utterances. The influence is integrated into the output gates of an LSTM with textual features as inputs. The obtained hidden states of the LSTM are fed to a softmax function to determine the affective state for each utterance.

We have designed and carried extensive experiments on the MELD and IEMOCAP datasets to evaluate the QIN model, in comparison with a wide range of baselines, including six deep neural network approaches: a deep convolutional neural network (CNN) and five variants of long short term memory (LSTM) networks. The experimental results demonstrate the effectiveness of the QIN model.

2 QuantumTheory Preliminaries

2.1 Basic Notations and Concepts in Quantum Theory

In quantum theory, the probabilistic space for quantum theory is an infinite Hilbert Space [Bourbaki, 1966] 1 , noted as H. For simplicity and in line with previous quantum-inspired models [Zhang et al., 2018a; Wang et al., 2018], we restrict our model to finite vector spaces over real numbers in R.

With Dirac's notation, a state vector ϕ and its transpose are expressed as a Ket |ϕ〉 and a Bra 〈ϕ| respectively. The inner product between two state vectors |ϕ1〉 and |ϕ2〉, is represented as 〈ϕ1|ϕ2〉. Similarly, the representation of the wave function (which is also a mathematical description of the quantum state) in Hilbert Space is given by the inner product ϕ (x) = 〈x|ϕ〉.

Quantum Probability (QP) is a generalization of the classical probability theory. In QP, an event is a subspace of Hilbert Space represented by an orthogonal projector Π. Assume |u〉 is a unit vector, i.e., ‖⃗u‖2 = 1, the projector Π on the direction u is written as |u〉 〈u|. ρ = ∑i pi|u〉〈u| can represent a density matrix. Density matrix ρ is symmetric, ρ = ρT , positive semi-definite (ρ ≥ 0), and of trace 1. The quantum probability measure µ is associated with the density matrix. It satisfies two conditions: (1) for each projector |u〉〈u|, µ(|u〉〈u|) ∈ [0, 1], and (2) for any orthonormal basis {|ei〉}, ∑ni=1 µ(|ei〉〈ei|) = 1. The Gleason's Theorem [Gudder, 2014] has proven the existence of a mapping function µ (|u〉〈u|) = tr (ρ|u〉〈u|) for any |u〉.

1 complex vector space possessing the structure of inner product

The density matrix representation in QP provides a comprehensive mathematical formalism to capture the intrautterance interactions, which will be detailed in Section 3.

2.2 Preliminaries of QuantumMeasurements

QuantumMeasurement (QM) theory includes ordinary quantum measurements (i.e., strong measurements) and weak measurements. Quantummeasurement consists of two steps: (i) the measurement device is weakly coupled to the quantum system; (ii) the measurement device is strongly measured, and its collapsed state is referred to as the outcome of the measurement process.

Let |φd〉 denote the wave function of measurement device and represent the position basis. It could be written as:

where x is the pointer position, and σ is the standard deviation of a normal distribution around 0.

Suppose the quantum system being measured S is in a state |ψ〉 = α|0〉 + β|1〉, in which α and β are the complex probability amplitudes satisfying |α|2 + |β|2 = 1. |0〉 and |1〉 are the eigenstates corresponding to the eigenvalues 0, 1 respectively. The system and the measurement device are then entangled as such:

The details of entanglement process can be referred to [Von Neumann, 2018]. Next, a strong measurement is carried out on the pointer of the measuring device, resulting in a collapse of the pointer to the state |x0〉. The entangled system is also collapsed to:

where the eigenvalue x0 could be anywhere around 0 or 1, or even further away. Whether the quantum measurement is strong or weak is determined by the ∆ = σ2 . The collapse of the pointer biases the system's vector. However, if σ is very big, the bias will be very small and the system's outcome will be very similar to the original vector. A detailed analysis is shown in Table 1.

QM provides a principled and effective mechanism to capture the inter-utterance interactions.

3 Learning Interaction Dynamics with the Quantum-Inspired Interactive Networks

3.1 ProblemFormulation and Network Procedure

In this work, we target determining the attitude of each speaker at the utterance (sentence) level, in terms of positive, negative and neutral. The problem we investigate thus takes each utterance u as input and produces its sentiment label y as output. Hence, we formulate the problem as follows:

In a multi-turn conversation, how can we capture the interactions between speakers to determine their emotional changes during the conversation?

The architecture of the proposed quantum-inspired interactive network (QIN) is shown in Fig. 3. We first extract textual features of conversational discourses ⃗x = [⃗r1, ⃗r2, ..., ⃗rn] through a density matrix based convolutional neural network (DM-CNN), which takes the semantic dependencies into consideration. Second, inspired by quantum measurement theory, a strong-weak influence model is developed to compute the inter-utterance influences between speakers within the whole conversation, denoted as R. Last, an LSTM variant is built on top of the extracted textual features ⃗x to model the evolution of sentiments in the conversation, with the output gate ot combined with the inter-utterance influences R.

3.2 DensityMatrix-Based CNN

Nowadays, a series of pioneering studies provide the evidence that density matrix, which is defined on the quantum probabilistic space, could be applied in natural language processing as an excellent representation method [Sordoni et al., 2013; Zhang et al., 2018a; Li et al., 2018]. Compared with embedding vector, density matrix could encode more semantic dependencies. Motivated by Zhang's work [Zhang et al.,

2018a], we develop a density matrix based convolutional neural network (DM-CNN) to represent utterances. The representation procedure is described below.

Suppose |wi〉 = (wi1, wi2, ..., wid)T is a normalized word vector. The projector Πi for a single word wi is formulated in Eq. (5). One-hot representation of words over other words is known to suffer from the curse of dimensionality and difficulty in representing ambiguous words. In this work, we use word embeddings to construct projectors in semantic space.

Based on word projectors Πi, we represent an utterance with a density matrix ρu, which is formulated as:

where pi is the probability of event (word) Πi satisfying ∑i pi = 1. In this work, we set pi with equal probabilities pi = 1D , where D is the number of words in a document.

The utterance density matrix ρu is then fed into a deep CNN architecture to obtain abstract textual features. The CNN consists of two convolutional layers with max pooling, a fully connected layers and a softmax layer. The first convolutional layer has eight 5×5 filters. The second convolutional layer has sixteen 3 × 3 filters. The obtained textual features ⃗x = [⃗r1, ⃗r2, ..., ⃗rn] are used as inputs of our QIN model.

3.3 AQuantumMeasurement-Inspired Strong-Weak Influence Model

Influence is an indirect and invisible way of altering the state of an entity, and is thus difficult to model. When one talks to another person, he is often influenced by their style of interaction. We posit that the intensity of interaction determines whether a speaker's affective state might or might not change. In particular, a strong interaction is in effect if his affective state changes, indicating he is greatly affected by others. On the other hand, weak interaction is used to describe such influence as too weak to offer changes to a speaker's affective state.

In QT, quantum measurement describes the interaction (coupling) between a quantum system and the measurement device. Strong measurement leads to the collapse of the quantum system while weak measurement disturbs the quantum system very little. The variance of pointer readings of the measurement device could distinguish whether the interaction is strong or weak. In this work, we treat each speaker as a learning system. Accordingly, the interaction could be characterized as a coupling between two systems. The interaction between quantum system and the measurement device is exactly similar to the interaction between speakers. Inspired by this, we associate strong and weak interaction with quantum measurement, and develop a strong-weak influence model.

Specifically, our model is an extension of the dynamical "influence model", which generalizes HMM by constructing the influence matrices for describing the influence each Markov chain has on the others [Pan et al., 2012]. Suppose there are C entities in the system, and each entity e is associated with a finite set of possible states {1, 2, ..., S}. At different time t, each entity e is in one of the states, denoted by qe t ∈ {1, 2, ..., S}. Influence is treated as the conditional dependence between each entity's current state qe t at time t and the previous states of all entities q1 t-1 , q2 t-1 , ..., qC t-1 at time t - 1. qe t is only influenced by all entities at time t - 1. Therefore, the conditional probability can be formulated as:

where R(rt) is a C ×C matrix (R(rt)e,c represents the element at the eth row and the cth column), rt ∈ {1, 2, 3, ..., J}, t = 1, ..., T, and J is a hyperparameter set by users to define the number of influence matrices R(rt). P (qe t |qc t-1 ) is the transition probability from state qc t-1 to qe t , controlled by an S × S matrix M c,e specific to a pair of entities (c, e): P (qe t |qc t-1 ) = M c,e qc t-1 ,qe t , where M c,e qc t-1 ,qe t represents the element at the qc t-1 th row and qe t th column of matrix M c,e .

However, in a turn-taking conversation, the speakers' states in each turn are influenced by both the current states of speakers who speak in front of e at turn (time) t, i.e., q1 t , q2 t , ..., qe-1 t and the previous states of other speakers who have not yet spoken (including the current speaker under concern) in the current round, i.e., qe t-1 , qe+1 t-1 , ..., qC t-1 . In particular, the state of the first speaker is influenced solely by previous states of all entities. The conditional probability then becomes

Referring to the example shown in Fig. 1, i.e., C = {Jen(J),Ross (R)}. Each speaker is in one of three affective states, which are positive, negative and neutral, e.g., S = 3, and qR t , qJ t ∈ {-1, 0, 1}. The conditional probability is measured as:

where R(rt)JJ , R(rt)JR, R(rt)RJ , R(rt)RR are four elements of the influence matrix R(rt), denoting how Jen influences Jen, how Ross influences Jen, how Jen influences Ross, and how Ross influences Ross.

Inspired by quantum measurement, we use two influence matrices (i.e., J = 2, rt ∈ {1, 2}) to represent strong and weak influences. The switching of rt is determined by the average standard deviation of speakers' sentimental scores σavg. We set the eigenvalues of speaker's affective state to -1, 0 and 1, i.e., x ∈ {-1, 0, 1}. Hence, we introduce the following prior for rt:

where p (x) = (2σ2π)- 12 e-(x-µavg)2 /2σ2 , denoting the probability density to get x, and µavg is set to the average of all expectations in this work.

We illustrate the difference between the dynamical influence model and the strong-weak influence model in Fig. 2. Finally, we obtain two influence matrices, which capture the strong and weak influences, i.e., R(2) and R(1), from one speaker over another speaker under different interactive environments 2 .

3.4 Quantum-Inspired Interactive Networks

Since we have learned the interaction information (including interactions between terms and interactions between speakers), then we incorporate them into the quantum-inspired interactive networks (QIN), which is a variant of LSTM.

The QIN model is proposed for conversational sentiment analysis. The main idea is: (1) for each LSTM unit, combining the output gate ot with the learned influence matrices R to constitute new output gate, describing what information we're going to output. The new output gate has considered the previous speakers' influences. (2) Taking textual vectors that are built by DM-CNN as inputs, obtaining their hidden states ht, and thus making decisions. Fig. 3 represents the overall architecture of the QIN model.

Let ⃗xei t = [⃗r1,⃗r2, ...,⃗rn] represents the input of speaker ei, which has been learned by DM-CNN. hei t represents the outputs of speaker ei, where t = {1, 2, ..., T}. We put hei t into a softmax layer to obtain the sentiment label. That is,

where Ws and bs are the parameters.

In the conversation, the influence that one speaker has on the other speaker would control the affected speaker's response. In Fig. 3, for two adjacent speakers (denoted as e1 and e2) at turn t = 1 (i.e., Spe1 t=1 , Spe2 t=1 ), Spe1 t=1 actually determines how Spe2 t=1 is constructed. Furthermore, at the next turn t = 2, the construction of Spe1 t=2 would be influenced by both Spe1 t=1 and Spe2 t=1 , and the construction of Spe2 t=2 would be influenced by both Spe2 t=1 and Spe1 t=2 . Influence controls what information one speaker is going to flow out, which is similar to the role of the output gate. This influence has already been described by the influence matrix R (in Section 3.3). Hence, we consider the influences of the previous speakers on the current speaker by incorporating the influence scores into the computation of the output gates in the LSTM of QIN, which can be formulated as:

2 The detailed inference process is given on https://github.com/anonymityanonymity/influence-model.git

where We1 and We2 are the normalized weights. Re1,e1, Re1,e2, Re2,e1, Re2,e2 are elements in R(rt).

Model training. In QINmodel, cross entropy with L2 regularization is used as the loss function, which is defined as:

where yi denotes the ground truth, ˆ yi is the predicted sentiment distribution. i is the index of utterance, j is the index of class. λr is the coefficient for L2 regularization. We use the back propagation method to compute the gradients and update all the parameters.

4 Experiments

4.1 Experimental Settings

Datasets. We conduct experiments on the MELD 3 and IEMOCAP 4 datasets to validate the effectiveness of QIN model. MELD contains 13,708 utterances from 1433 dialogues. The utterances in each dialogue are annotated with three sentiments (which are positive, negative and neutral) and seven emotions (which are anger, disgust, fear, joy, neutral, sadness and surprise). IEMOCAP is a multimodal database of ten speakers involved in two-way dyadic conversations. Each utterance is annotated using the following emotion categories: anger, happiness, sadness, neutral, excitement, frustration, fear, surprise, and others.

3 This dataset is available on https://affective-meld.github.io/.

4 http://sail.usc.edu/iemocap/.

Evaluation metrics. Considering the imbalanced sample problem, we adopt weighted F1 score, Accuracy as the evaluation metrics to evaluate the classification performance. We employ t-test to perform the significance test.

Hyperparameters setting. In this work, we use the GloVe word vector [Pennington et al., 2014] to find word embeddings. The dimensionality is set to 300. All weight matrices are given their initial values by sampling from a uniform distribution U(-0.1, 0.1), and all biases are set to zeros. We set the initial learning rate to 0.001. The batch size is 60. The coefficient of L2 normalization in the objective function is set to 10-5 , and the dropout rate is set to 0.5.

4.2 Comparative Models

In order for a comprehensive evaluation of the QIN model, we include a range of baselines for comparison. They are listed as follows.

CNN. We employ a CNN [Kim, 2014] including three convolutional layers and a fully connected layer. It is trained on top of word embeddings for utterance-level classification.

LSTM & biLSTM. We implement a standard LSTM and bi-directional LSTM. They take word embeddings as input so as to get the hidden representation of each word.

ATAE-LSTM. We implement an attention based LSTM with aspect embedding [Wang et al., 2016]. We obtain aspect embeddings by averaging the vectors of words, and append it with each word embedding vector.

Contextual biLSTM & Hierarchical biLSTM. We implement a contextual biLSTM [Poria et al., 2017] and hierarchical contextual biLSTM to model semantic dependency among the utterances.

4.3 Results and Analysis

Table 2 shows the performance comparison of QIN with other baselines. In the case of sentiment classification on MELD dataset, CNN, LSTM, biLSTM and ATAE-LSTM achieve worse performance against all neural network baselines, because they ignore the contextual dependencies among utterances. The complete meaning of an utterance might be determined by preceding utterances. Hence, the introduction of attention mechanism does not help improve the performance. This suggests the importance of contextual modeling. Moreover, on fine-grained (e.g., 7-class and 9-class) emotion classification tasks, all the above four baselines have their victory and defeat. This implies that distinguishing fine-grained emotions is a more difficult and intricate task. Through taking utterances as inputs, contextual biLSTM has extracted contextual features. Contextual biLSTM performs consistently better over other baselines. Our QIN takes a further step towards emphasizing the importance of modeling interactions. Through learning both the intra- and inter-utterance interaction dynamics, QIN achieves the best performance among all baselines.

In the case of emotion classification on IEMOCAP dataset, all models get very poor performance because of the large number of classes. However, QIN still achieves the best performance. Compared with contextual biLSTM, QIN improves the performance by 7.1% by accuracy. The main reason is that QIN has modelled more semantic dependencies and previous speakers' influence. The results demonstrate the effectiveness and necessity of modelling the interactions in conversational sentiment analysis.

5 Ablation Study

In this subsection, we design a series of sub-models to study the impact of different components of the QIN model: (1) DM-LSTM, which does not model influences, but only uses density matrix-based CNN; (2) Influence-LSTM, which only uses quantum measurement inspired by strong-weak influence model and incorporates influences into the output gate.

From Table 3, we observe that QIN achieves the best performance among all models. The results verify that modeling both intra- and inter-utterance interactions makes a positive contribution to judging the sentiment polarity of an utterance. Influence-LSTM is worse than DM-LSTM on MELD, but performs better on IEMOCAP. Because IEMOCAP only contains two-way dyadic conversations, and capturing the interactions between two speakers is easier. DMLSTM achieves better performance than CNN, LSTM and ATAE-LSTM, showing that density matrix representation can more effectively encode the semantic dependencies and their probabilistic distribution information. Influence-LSTM outperforms the baselines, proving that modeling inter-utterance interactions benefits the sentiment classification performance.

6 Conclusions

In this paper, we propose the QIN model, which could capture the correlations between terms and measure the influence of the previous speakers. The main idea is to use a density matrix based CNN and a strong-weak influence model inspired by quantum measurement theory to model such interaction dynamics. The experimental results on MELD and IEMOCAP demonstrate that our proposed QIN largely outperforms a number of state-of-art sentiment analysis algorithms, and also prove the importance of modeling interactions.

Acknowledgments

This work is supported by The National Key Research and Development Program of China (grant No. 2018YFC0831704), Natural Science Foundation of China (grant No. U1636203, 61772363), Major Project of Zhejiang Lab (grant No. 2019DH0ZX01), and the European Union's Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement No 721321.