BlueMemo: Depression Analysis through Twitter Posts

1 Introduction

Social media services (such as Twitter, Facebook, Weibo) have become popular platforms for content sharing and information dissemination. The widespread use of social media also provides a new convenient medium for users to express their emotions. Studies have shown that social media has become a powerful platform for people with depression to self-disclose psychological problems and seek social support [Manikonda and De Choudhury, 2017]. Users' posts are an expression of their inner emotions, and discovering the clues offers a promising new opportunity to spot depression. More than 350 million people worldwide are affected by depression [Organization, 2010], and without proper treatment, people with depression will increasingly develop severe symptoms. Early detection and timely treatment may benefit the prognosis of most adult patients with mental illness [Picardi et al., 2016]. Therefore, it is necessary to carry out early detection and intervention for patients with depression to minimize its impact on public health [Reay et al., 2011]. The traditional approaches for depression detection require the psychiatrist to measure the severity of depression in a faceto-face interview. With limited resources, the effectiveness of this clinical diagnosis is not satisfactory. Encouragingly, advances in artificial intelligence have made it possible to detect depression-related indicators from between the lines automatically. Many clinical studies have shown that text and image features extracted from video signals in interviews can be used to detect depression objectively [Stolar et al., 2015; Gillespie et al., 2017]. Other studies of social networks have found that users can be divided into groups based on their profiles or information posted [Ding et al., 2019; Hu et al., 2017; He et al., 2018]. Inspired by such work, the new study has used visual social media, such as images and text on twitter, to encode predictors of depression with widespread success. Shen et al., for the first time [Shen et al., 2017], attempted to detect users' depression by combining users' various demographic information and visual features of pictures. Gui et.al [Gui et al., 2019] used features such as text features posted by users and color saturation of images to check the depression of Twitter users.

∗Both authors contributed equally to this research.

However, as users vary greatly in their use of text and become more inclined to publish image content, a well-trained model will lose its power by failing to capture the subtle expressions of emotion in text and the target information implicit in photo content. Fortunately, the latest research on multi-modality [Liu et al., 2018] shows that by incorporating more feature expressions of users into the learning process, the model can fully understand users' mental state and significantly improve performance. As such, there is an urgent need for developing a system to learn from users' emotions and desires in a multi-modal way to help us find patients with depression on social media. In this paper, on the basis of the multi-modal user posts, we demonstrated a new analysis system of depression: BlueMemo 1 . Our system feeds textual feature, image multi-label textual feature, and visual feature of social media users into the learning process, then uses tensor transformation to fuse multi-modal features to optimize the detection strategy. BlueMemo is presented as a browser-based service interface, where users can freely customize the detection target and training population on the interactive page, and obtain a patient analysis experience that resembles clinical diagnosis. Figure 1 illustrates the architecture of our system.

2 Blue Memo

2.1 Proposed Approach

Feature Extraction. Three types of features are considered and extracted in our system, textual feature, visual to textual feature and visual feature. We extracted textual feature representation from users posted tweets by using a pre-trained Bert[Devlin et al., 2019], which has been proved effectively in contextualized sentence understanding and representation. In order to take full advantage of user posted information in social media, users posted images were also acquired for visual and visual to textual feature extraction. A deep CNNbased multi-label classifier was trained using COCO datasets [Lin et al., 2014], which contains 80 common objects and can be effectively used to describe the images. For each posted image, its multi-label text information can be predicted by using this classifier. Finally, the predicted multi-label text information was converted into textual feature representation by using the Bert model. In such way, visual to textual feature can be acquired. In addition, the visual color feature representation was extracted by using posted images. We adapted conventional image processing method to generate color-related attributes, including the image histograms in RGB and HSV color space. This color-related attribute was then normalized and fused as the visual feature representation.

Feature Fusion and Classification. To generate more robust feature for depression detection and analysis, similar to the feature fusion method in [Lin and Kumar, 2017; Lin and Kumar, 2018] ,we combined the extracted multimodal features. Let ft be the textual feature extracted from the tweets, fv be the visual features calculated from the posted images and fvt be the visual to textual feature acquired from the image multi-label. The multi-feature fm was combined and trained by using a tensor fusion network [Liu et al., 2018], which can efficiently learn the features from different modalities. During the network training process, the multi-feature fm was combined by using a tensor multiplication operation and soft-max function was used for classification. Finally the input multi-features are classified into two classes, i.e. depression user and non-depression user.

1 A demo video can be viewed at: https://youtu.be/SkLR3kIB8dg

2.2 SystemMain Features

BlueMemo is a system designed to detect people with depression from social media and further assist psychologists in their analysis. It goes beyond the clinical diagnosis of depression based on doctor-patient interviews and gives psychologists the ability to retrieve and visualize users' past social media posts. BlueMemo can assess a user's risk of depression based on a customized scope, and psychologists can receive systematic diagnostic support as well as more detailed understanding of a user's psychological expression through interactive functions. More specifically, BlueMemo provides the following facilities for psychologist:

User/Posts. provides a quick preview for system users. To some extent, users' posts have replaced the traditional Q&A record of interview diagnosis. Therefore, we set up the preview function of the user's homepage information and the user's posts, respectively.

Diagnostics. presents four visual analysis results for system users. Once we have identified the user to test, we can analyze by clicking diagnostics-button. The postedwordcloud is an intuitive display of the modal data of the text. HistogramcomparisoninHSV colorspace is a visual display of the image visual color feature. The depressiononanannualbasis can be used to look at users' depression risk scores yearly for different modality. Diagnosedwithdepressionforthefirsttime provides a time when the first depression risk score exceeds the threshold, facilitating the psychologist to pinpoint the onset time.

Depression Risk. mainly presents the final assessment score. Trendofuserposts are used to show the overall trend in the number of posts from subjects. Considering that visual feature has reference basis in clinic, we show the visualfeature count table on this page. Depression score provides a user's final risk assessment score and compares it with the mean of all users.

2.3 Evaluation

We followed the rules in [Shen et al., 2017; Shen et al., 2018] to collect the twitter dataset, and used it to train and evaluate the model. The data collection rule is to label users' selfreports as depression. Specifically, if a user ever posted a message on twitter claiming to be diagnosed or experiencing depression, we marked it as depression user and took the posts before the time of this message as a training sample. The dataset consists of two subsets, D1(depressed users) and D2(non-depressed users), which mainly includes tweets and images posted by users. During the offline training, all 1,402 depressed users and 292,564 tweets in D1 were enlisted. Non-depressed users were randomly selected from D2with more than 300 million non-depressed users to combine with D1into training, testing, and evaluation set. When evaluating the model, we conducted 5-fold cross-validation experiments on the proposed method. We used F1 scores for evaluation, and the F1 scores of MDL [Shen et al., 2017], COMMA [Gui et al., 2019] and proposed models were 84.9, 90.0 and 96.1, respectively. The detection model of BlueMemo had better performance than previous work.

3 Conclusion

We propose a new depression analysis system for people at risk for depression on Twitter. People are increasingly inclined to express their feelings, to show their life state, so this system is designed to fully explore the multi-modal data in social networks and detect users' abnormal mental state. The proposed model extracted text features, image multilabel textual feature, and visual feature and used tensor transformation based classifier to detect those depression groups. The system realizes the practical detection with high precision and carries on the visual analysis of the detection result. The system is promising to screen online users for depression and has the potential to help psychologists review patient histories and support diagnosis.