RoTuEl: A Semi-Automated Method for Labeling Political Tweets

2 Method

The semi-automated model proposed by the authors of this paper includes the following steps: pre-processing, processing and postprocessing.

Preprocessing

In this step public tweets posted during elections will be collected from the Twitter API and stored in a database. Besides tweets, other user information are stored, such as profile description, user id, location, URL of profile and page background image and number of times the tweet was retweeted. These information are used in the next step.

Processing

A set of users stored in a database who expressed support and explicit rejection of each candidate during the elections are identified and clustered from an automated method. The criteria used for this method are: (i) analysis of the user's public profile description, (ii) the user profile picture analysis, (iii) analysis of user's account background image and (iv) analysis of messages created by user and published in his timeline. Then, one user of each group is selected and also a random set of messages posted at the beginning, middle and end of his timeline is chosen. Only one message of each block (beginning, middle and end) is chosen randomly and the polarity of then (positive/negative) are valued by people through a crowdsourcing interface. During the evaluation, published adjacent messages (before and after) the selected message is displayed on interface to users. Messages with 100% agreement between evaluators are named seeds and used to locate other positive (support) and negative (rejection) tweets. Users that retweeted seeds are identified. Messages blocks (beginnig, middle and end) of these new users are selected to be evaluated by crowdsourcing interface in order to check if they have same polarity seed retweeted. Finally, the result of the analysis of that set of tweets will be used by the algorithm to identify if other messages not analyzed by crowd have the same polarity.

Postprocessing

At this stage a list of tweets which were labeled positively and negatively is arranged to be used for classifier training algorithm used in the prediction calculation step.

3 Evaluation and Results

A preliminary assessment of this proposal was carried out and the results were quite encouraging. An online application was built to collect tweets published at the time of the presidential elections in Brazil in 2014. We collected approximately 8 million tweets of almost 460.000 users. The method was used to identify positive tweets of one candidate. By using it approximately 2.200 positive messages could be labeled, from a data of only 60 messages of 20 users stored in a database that retweeted seed. Five evaluators were invited to evaluate messages. In order to check the accuracy of the method one person manually analyzed 2200 messages, all users' description, profile and page background image. The result of the analysis showed that all users explicitly supported the same user candidate whose seed had been retweeted. During the study analysis, it was noticed that many messages that would have been classified as positive, they may be classified as negative if they were evaluated separately. To prove this hypothesis, a questionnaire containing 12 tweets randomly chosen from those users' timeline were selected. A questionnaire was created and an invitation sent to 41.000 users chosen from the database. In 24 days 424 responses were collected. The average agreement of positive opinion among the messages was 60.8% (the lowest rate was 3.1% and the highest case was 90.2%). Socioeconomic information from users were also collected: men (69%) have undergraduate degree or higher (75.9%), up to 15 years old (2.1%), between 16 and 24 years old (26.7%), 25 and 34 years old (24.7 %), 35 and 44 years old (20.9%), 45 and 59 years old (21.4%), 60 and over (4.2%) and live in urban areas (96.2%).

4 Future Work

The authors have completed the tool development to collect political tweets and approximately 8 million political tweets were stored in a database. The criterion (ii), (iii) and (iv) the algorithm for selecting users and crowdsourcing interface for evaluation of messages will be implemented. Buil the algorithm that will be used to find other tweets of the same polarity of selected seeds and finally analysis of the accuracy of the proposed method over other methods for tweets labeling.