--- title: "For Women, Life, Freedom: A Participatory AI-Based Social Web Analysis of a Watershed Moment in Iran's Gender Struggles" authors: - "Adel Khorramrouz" - "Sujan Dutta" - "Ashiqur R. KhudaBukhsh" doi: "10.24963/ijcai.2023/667" url: "https://ijcai.org/proceedings/2023/667" pdf: "https://ijcai.org/proceedings/2023/0667.pdf" venue: "Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence" year: 2023 pages: "6013-6021" track: "AI for Good" keywords: - "AI for Good: Multidisciplinary Topics and Applications" - "AI for Good: Natural Language Processing" - "AI for Good: Machine Learning" --- # For Women, Life, Freedom: A Participatory AI-Based Social Web Analysis of a Watershed Moment in Iran's Gender Struggles Adel Khorramrouz, Sujan Dutta, Ashiqur R. KhudaBukhsh *Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence*, pages 6013-6021. DOI: [10.24963/ijcai.2023/667](https://doi.org/10.24963/ijcai.2023/667) ## Abstract In this paper, we present a computational analysis of the Persian language Twitter discourse with the aim to estimate the shift in stance toward gender equality following the death of Mahsa Amini in police custody. We present an ensemble active learning pipeline to train a stance classifier. Our novelty lies in the involvement of Iranian women in an active role as annotators in building this AI system. Our annotators not only provide labels, but they also suggest valuable keywords for more meaningful corpus creation as well as provide short example documents for a guided sampling step. Our analyses indicate that Mahsa Amini's death triggered polarized Persian language discourse where both fractions of negative and positive tweets toward gender equality increased. The increase in positive tweets was slightly greater than the increase in negative tweets. We also observe that with respect to account creation time, between the state-aligned Twitter accounts and pro-protest Twitter accounts, pro-protest accounts are more similar to baseline Persian Twitter activity. ## Full text ## 1 Introduction Words are the only victors. - Salman Rushdie; Victory City; 2023. On 16 September 2022, Mahsa Amini, a 22-year-old woman died under police custody in Iran. Reportedly, she was arrested because of not wearing her hijab (headscarf) properly. As media and police presented conflicting accounts of her death [Alkhaldi and Mostaghim, 2022], Mahsa Amini's death enraged Persian (Farsi) Twitter users in an unprece#مهسا امینی dented manner [Kermani, 2023]. The hashtag (#MahsaAmini) became one of the most repeated hashtags on Persian Twitter and initiated a Twitter protest where Iranians expressed their grievances against the government like never before. Support and solidarity for gender equality poured over in from prominent world leaders [France-Presse, 2022], artists [Pina, 2022], and sports personalities [Alkhaldi, 2022] across the globe. ∗Ashiqur R. KhudaBukhsh is the corresponding author. #MahsaAmini was undoubtedly the overwhelming toptrending hashtag on Persian Twitter for months during the relentless protest. However, for a brief period of time, hashtags with an opposite stance toward the protest (e.g., #ExecuteThem or #ISupportKhamenei 1 ) trended. Prior literature conjectured state-aligned trolling in Iran on Instagram [Kargar and Rauchfleisch, 2019]. Also, social bot accounts' capability to spread extreme ideology is well-documented [Stella et al., 2018; Berger and Morgan, 2015]. Via a substantial corpus of 30.5 million tweets relevant to the protest, this paper makes three key observations: 1. The grievances of protesters against the current government mention a broad range of incidents spanning decades. 2. With respect to account creation time, between the statealigned Twitter accounts and pro-protest Twitter accounts, pro-protest accounts are more similar to baseline Persian Twitter activity. 3. There was a noticeable shift in positive stance toward gender equality after the protests on Persian Twitter discourse. To our knowledge, no computational analysis relying on sophisticated natural language processing methods exists that has examined gender equality in Persian social media discourse let alone at this unprecedented scale. That said, we believe our key contribution lies elsewhere. Our paper marks an important effort to include the stakeholders - the Iranian women - in this AI-building process. All examples in our supervised solution's training set are annotated by Iranian women. Our examples are thus grounded in cultural contexts and first-person experience about the gender struggles in Iran. Datasets addressing issues faced by vulnerable communities often end up being annotated by annotators with little or no documentation [Guest et al., 2021; Ramesh et al., 2022]. Since annotated examples often form the core of a supervised AI system, it is important to involve stakeholders in the annotation process. For example, Ramesh et al. [2022] present a lexicon of queer-related inappropriate words where one of the annotators identifies as queer. Similarly, Guest et al. [2021] present a misogyny dataset where the majority of the annotators identify as women. 1 Ali Khamenei is the second and current supreme leader of Iran who is in office since 1989. Our annotators' role is not limited to mere annotation. Rather, they take an active role in guiding how to curate more meaningful data by suggesting suitable keywords to curate our dataset and providing a valuable seed set of examples to initiate an active learning pipeline. Our results indicate that the annotators' contributions yielded a richer seed set than a random baseline. At a philosophical level, we see this work as a part of the growing conversation of participatory AI [Harrington et al., 2019; Delgado et al., 2022; Bondi et al., 2021; Birhane et al., 2022] where the goal is to develop systems for the people and by the people. ## 2 Datasets As we already mention, #MahsaAmini initiated a protest with global participation. Understandably, tweets in global languages such as English or French are likelier to reflect the global perspective on this issue. Given that Twitter is banned in Iran and users reportedly use VPNs to access Twitter [Kermani, 2023], considering geo-tagged tweets is not a reliable option either to understand and analyze the Iranian perspective. Therefore, we restrict our analyses to only tweets authored in the Persian language. We assume that our choice of language can act as an effective filter to ensure our dataset is less likely to be diluted by the global discourse. We use Twitter's official language label as ground truth. We collect three corpora: Dprotest ; Dgender ; and Dbaseline . Our dataset spans the time duration of Jan 15, 2022, to Jan 15, 2023 2 . We define the time period from January 15, 2022, to September 15, 2022, as Tbefore. We define the time period from September 16, 2022, to January 15, 2023, as Tafter. A short description follows next. Throughout the paper, if we use a Persian word or phrase, we present an English translation in parentheses following the word. ## 2.1 Dgender Dgender consists of 6,036,012 Tweets which has been posted by 700,189 unique users. - "دختر" or (women) "زنان" 1. All tweets that have either (girl). - im- the means which "ناموس" 2. All tweets that have either mediate female family members (daughter, mother, sister, wife) whom the male member of the family (father, brother, husband) should protect and sometimes control jealousy of form positive the means which "غیرت" or that men have upon their female family members against other men. These two search keywords were suggested by our annotators. 3. All tweets that have gender insult words against women "a or prostitute" "a indicating both "کصده" and "جنده" promiscuous woman" in a pejorative way (the second insult word mostly accompanies with sister). 2 On January 7, 2023, two executions relevant to this protest happened [Radford and Fowler, 2023]. We thus set our end date one week after the executions. 4. all tweets that have at least one word from the two fol"خواهر" (woman), "زن" (girl), "دختر{" lowing subsets: sister}; and 2. (rights), "حقوق" ,)noitulover("انقلاب" (life), "زندگی{" (freedom)}. "آزادی" ## 2.2 Dprotest Dprotest consists of: - yield- them in (#MahsaAmini) #مهسا امینی 1. tweets with ing 21,308,449 Tweets posted by 655,303 unique users. 2. tweets that support government through the hashused been has hashtag This .#لبیک یا خامنه ای tag 1,051,792 times by 71,484 unique users across the entire Twitter timeline accessible through the APIs. - means which #اعدام کنید 3. tweets that have the hashtag execute them. This hashtag has been used 11,292 times by 5,130 unique users across the entire Twitter timeline accessible through the APIs. ## 2.3 Dbaseline In order to estimate baseline Persian Twitter behavior, we col- and )که ,از ,در ,با ,به( consider five Persian stop words lect 6,000 tweets per day (evenly distributed across the hours) that contain at least one of these stop words. Our dataset, Dbaseline , consists of 2,190,000 tweets. We compute the unigram distributions of subsets of Dbaseline that was authored during Tbefore and Tafter . Table 1 lists the top 20 high-frequency non-stop words present when (1) we subtract the unigram distribution of Tafter from the unigram distribution of Tbefore (left); and (2) we subtract the unigram distribution of Tbefore from the unigram distribution of Tafter (left). In plain English, these are the words that appeared more frequently during one period and much less frequently during the other. From the right column of Table 1, we note that several of these words are not indicative of civic unrest while the left column does not indicate similar unrest. We conduct a similar experiment to track shift in high-frequency hashtag usage between the two time periods. We again observe that even in the baseline Persian Twitter discourse, Tafter showed several hashtags relevant to the protest. Table 1: Biggest shift in token usage in Dbaseline between Tbefore and Tafter . | More presence during Tbefore | More presence during Tafter | |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | "خوش" (happy), "انسان" (human), "زبان" (language), "آقا" (M.R), "نمیدونم" (I do not know), "توییتر" (Twitter), "پسر" (boy), "قبول" (ok), "گوش" (ear), "حد" (limit), "خواب" (sleep), "جدید" (new), weareoneEXO, ,)nosaer("دلیل" ,)evol("عشق" ,)aissuR("روسیه" ,))hallA(dog("الله" ym("نظرم" opinion), ,)gnileef("حس" ,)ssendoog("خوبی" | "مادر" (mother) "کشته" (killed) "صدای" (voice) "مرگ" (death) "آزادی" (freedom) "خون" (blood) "جمهوری" (republic) "خبر" (news) "اعدام" (execution) "بخاطر" (for sake of ) "هشتگ" (hashtag) "ادامه" (continue) "خانواده" (family) "نظام" (can be translated to regime but not exact) "شهر" (city) "لطفا" (please) "انقلاب" (revolution) "اسم" (name) "ج.ا" (I.R (stands for Islamic re- public which represents regime)) "خیابون" (street) | Table 2: Shift in top hashtags present in Dbaseline between Tbefore and Tafter . | Top ten hashtags during Tbefore | Top ten hashtags during Tafter | |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | EXO, امید" "مکتب (the hope attitude), حسینیم" امام ملت "ما (we are nation of Imam Hossein), امید" "ماه (month of hope), قوی" "ایران (strong Iran), یجمعنا" الحسین "حب (love of Hossein gathers us), "اوکراین" (Ukraine), EXO (in Korean), الفرج" لولیک عجل "اللهم (Oh God, please fasten merg- ing relief (Imam Zaman)for us), امید" "عید (Hope Eyd), | امینی" "مهسا (mahsa amini), سراسری" "اعتصابات (Na- tionwide strikes), OpIran, MahsaAmini, StopHaz- araGenocide, IRGCterrorists, شاکرمی" "نیکا (Nika Shakarami), آزادی" زندگی "زن (women life freedom), Mahsa Amini, سراسری" "اعتراضات (Nationwide protests), شکاری" "محسن (Mohsen Shekari), | Table 3: Percentage of because of tweets that matched with individual lines in Grammy-winning song Baraye by Shervin Hajipour. | Line from Baraye | Translation | Percentage of match | |---------------------------------|------------------------------------------|-----------------------| | آرزو که دختری برای بود پسر داشت | for a girl who wished she were a boy | 24.86 | | آزادی برای | for freedom | 15.72 | | خواهرت خواهرم برای خواهرامون | for my sister, your sister, our sisters | 9.76 | | شعار همه این برای خالی تو های | For all these mean- ingless slogans | 7.91 | | اجاری بهشت این برای | For this forced "heaven" | 7.16 | | که ها مغز تغییر برای پوسیدن | for changing rusted minds | 7.09 | | که ای چهره برای میخنده | For smiling faces | 4.33 | | گرد زباله کودک برای هاش آرزو و | for child labor and their crushed dreams | 3.82 | ## 3 Baraye - Because Of A large fraction of tweets of Dprotest contains a phrase (because of ). These tweets were an outlet for Persian Twitter users to vent their frustrations about the situation. Specifically, the tweets aimed at answering a reason for the protests. The tweets expressing a complex collection of emotions on why the current government has failed the nation captured the imagination of Shervin Hajipour, a talented Iranian singer who won the 2023 Grammy award for his song Baraye (because of ). Each line of this song starts with because of and paints a picture of Iranian hope and despair. We collect 1.92 million tweets from Dprotest with the phrase because of. We train a FastText [Bojanowski et al., 2017] word embedding on Dprotest and for each tweet, we compute the line in the Baraye song that is the nearest neighbor in the embedding space. Table 3 lists the top 10 lines from the Baraye song that matched with the tweets. While the poignant song by Hajipour brilliantly captures Iranian aspirations and struggles, Table 4 indicates there is much more to Iranian angst than what the song could hold. The second most common trigram indicates the access barrier to Twitter. Most prominent social media platforms are blocked in Iran [Kargar and McManamen, 2018; Sohrabi, 2021] and reportedly, Iranians primarily take recourse to VPNs to participate in the social web [Kermani, Table 4: Top ten trigrams from because of tweets. We omit two spam trigrams (e.g., please follow me) aimed at gaining followers. Hadis Najafi was killed by a gunshot during the Mahsa Amini protest. | Trigram | Translation | |----------------------|---------------------------------------------------| | مهسا نجفی حدیث | Hadis Najafi Mahsa [Amini] | | آزادی زندگی زن | women life freedom | | ندارد توییتر دسترسی | have not Twitter access | | آزادی میشود رمزی | [your name] will become [the] symbol [of] freedom | | میشود رمزی نامت | your name will become symbol | | افکاری نوید بختیاری | [Pouya] Bakhtiari Navid Afkari | | شده ریخته های | spilled [blood] | | کولبران سانچی پلاسکو | Plasco Sanchi kolbar | | پویا آبان متروپل | Metropoll Aban (November) Pouya [Bakhtiari] | | کوی رکس سینما | Cinema Rex kouye [university] | 2023]. We observe that Navid Afkari 3 , an executed Iranian wrestler, is mentioned among the top trigrams. Multiple structural failures got also mentioned in the common trigrams [Bozorgmehr, 2017]. From the Cinema Rex fire incident that happened in 1978 [Ali and Ali, 2018], to the attacks on student-dormitory in 1999 [Bozorgmehr, 2017], to the current reality of web censorship, the most striking takeaway from Table 4 is perhaps the time duration between the events people mentioned. ## 4 Account Creation Time Prior literature has examined state-aligned trolling in Iran on Instagram platforms [Kargar and Rauchfleisch, 2019]. In this section, we present an analysis based on account creation time. We define four sets of users: Upro-protest; Ustate-aligned; Upro-execution; and Ubaseline. Upro-protest represents all users who used the hashtag #MahsaAmini (indicating support for Mahsa Amini) at least once in our dataset. Ustate-aligned represents all users who used the hashtag #ISupportKhamenei (indicating support for Ali Khamenei) at least once in our dataset. Upro-execution represents the set of users who used the hashtag #ExecuteThem at least once in our dataset. Finally, Ubaseline is the set of unique users who contributed to Dbaseline. We compute the account creation time for each user at the granularity of months and obtain normalized histograms for each of these sets. Figure 1 illustrates the account creation temporal distributions of the four user sets. All subsets exhibit a sharp spike around September 2022, however Upro-execution exhibits two different spikes. In fact, in September 2022, Google Trends indicates one of the most popular search queries from Iran Twitter). (download "دانلود توییتر" was Table 5 computes the KL divergence of the account creation time distributions with respect to Ubaseline. We observe that the distribution of Upro-protest is closest to Ubaseline while Ubaseline is the farthest. The order remains unchanged with other distributional distance measures (e.g., Bhattacharyya distance). Our qualitative findings remain unchanged even if we limit Ustate-aligned and Upro-execution to only those user accounts that used these hashtags during Tbefore and Tafter. [3 https://www.hrw.org/news/2020/09/12/iran-suddenlyexecutes-wrestler-navid-afkari](https://www.hrw.org/news/2020/09/12/iran-suddenly-executes-wrestler-navid-afkari) Figure 1: Distributions of account creation dates of different user sets. ![Image](https://ijcai.org/proceedings/2023/fulltext/667/image_000000_9445716a8ae84d2736ec9088bcbb72cb50893ef5b91e043ce6582807495273dd.png) Table 5: KL-divergence of the distribution of account creation time for different user subsets with respect to the distribution of account creation time of Ubaseline . | User Set | KL-Divergence | |----------------|-----------------| | Upro-protest | 0.15 | | Ustate-aligned | 0.25 | | Upro-execution | 0.46 | ## 5 Annotation All our annotation was conducted by four different annotators. All annotators identify as Iranian women and are fluent speakers of Persian. All of them have undergraduate degrees. Annotation task. Our text prediction task is to predict the stance toward gender equality. For each tweet, we ask the annotator: does this short document indicate a positive, neutral, or negative stance toward gender equality? Inter-rater Agreement. Between any two annotators, we have at least 500 overlapping samples. Across all rounds of annotation, the Cohen's κ ranged from 0.41 to 0.52. On a misogyny annotation task, Guest et al. [2021] reported Fleiss' κ of 0.48 and the Krippendorf's alpha as 0.49. Sanguinetti et al. [2018] report category-wise κ = 0.37 for offence and κ = 0.54 for hate. We further note that our observed interrater agreement is higher than Gomez et al. [2020] (κ = 0.15) and Fortuna and Nunes [2018] (κ = 0.17). Disagreement resolution. Since our task is likely to be subjective, resolving disagreements has to be grounded in the literature. Prior literature has considered diverse approaches to resolving inter-annotator disagreements (e.g., majority voting [Davidson et al., 2017; Wiegand et al., 2019] or third objective instance [Gao and Huang, 2017]). We resolve any disagreement in the following manner. For positives and neutrals, we only consider consensus labels. Following Golbeck et al. [2017], if any annotator marks an example as negative and the other annotator marks it as negative or neutral, we consider the aggregate label as negative. In order to ensure the anonymity of the annotators, we do not conduct any postannotation adjudication step to resolve disagreements. ## 5.1 Toward Participatory AI A notable feature in our work is the active involvement of Iranian annotators in both corpus creation and annotation. Our annotators helped us in the following two ways. Search keywords. At a deeper level, which data could contain relevant information may require a clear understanding of the social realities. While constructing Dgender , choosing woman or girl and gendered insults as search keywords required little cultural context. However, our annotators sugbe to "غیرت" and "ناموس" gested nuanced keywords such as "ناموس" included in our list of search keywords. Recall that, means the immediate female family members (e.g., daughter, mother, sister, or wife) whom the male members (e.g., father, brother, or husband) should protect and sometimes control; have men that jealousy of form positive a means "غیرت" and upon their female family members against other men. Seed set. A notable feature of our work is the active involvement of Iranian women in the annotation process, where they not only provide labels but also present important representative short documents to construct meaningful seed sets during the guided sampling step described in Section 6. ## 6 Active Learning Pipeline Research question: Is there a noticeable change in support for gender equality in Persian Twitter discourse before and after the demise of Mahsa Amini while in police custody? To estimate the support for gender equality in Persian Twitter discourse, we build a robust classifier detecting content supportive of gender equality. Since hashtag hijacking [Hadgu et al., 2013] is a common phenomenon where users with opposite views may use the most-popular hashtag to express an opposite stance, our goal is to predict the stance toward gender equality from tweet texts only. We first estimate to which extent tweets supporting gender equality are present in Dbaseline. We randomly sample 500 tweets weighing both Tbefore and Tafter equally (i.e., 250 from each time slice). In addition, we randomly sample 1,000 tweets from Dgender weighing equally Tbefore and Tafter . Table 6 summarizes the label distribution. We note that a large fraction of Dbaseline consists of neutral tweets. Table 6: Label distribution of the first stage of annotation (random sampling) during the seed set construction. | Dataset | Positive | Tbefore Neutral | Negative | Positive | Tafter Neutral | Negative | |-----------|------------|-------------------|------------|------------|------------------|------------| | Dgender | 11.9% | 51.9% | 36.1% | 30.4% | 35.3% | 34.2% | | Dbaseline | 0.4% | 98.3% | 1.2% | 2.8% | 93.4% | 3.6% | In order to construct a dataset that is diverse and representative of the unlabeled pool, we present an active learning pipeline that consists of well-known sampling steps. A short description of active learning follows next. ## 6.1 Background Active Learning is a powerful and well-established form of supervised machine learning technique [Settles, 2009]. It is characterized by the interaction between the learner, aka the classifier, and the teacher (oracle or labeler or annotator) during the learning process. At each iteration, the learner employs a sampling strategy to select an unlabeled sample (unlabeled samples) and requests the supervisor to label it (them) in agreement with the target concept. The data set is augmented with the newly acquired label, and the classifier is retrained on the augmented data set. The sequential labelrequesting and re-training process continues until some halting condition is reached (e.g., annotation budget is expended or the classifier has reached some target performance). At this point, the algorithm outputs a classifier, and the objective for this classifier is to closely approximate the (unknown) target concept in the future. The key goal of active learning is to reach a strong performance at the cost of fewer labels. Since retraining the model and running inference on a large, unlabeled pool is computationally costly, prior literature has examined the trade-offs present in a batch active learning setting [Yang and Carbonell, 2013]. In this work, we follow the batch active learning setting. ## 6.2 Seed Set Construction Random Sampling. In order to capture a diverse set of examples, we randomly select 1,000 samples fromDgender and 500 samples from Dbaseline. Table 6 indicates that solely relying on Dbaseline to construct the seed set will result in extreme class imbalance with very few positives and negatives and predominantly neutrals. Sampling from Dgender might yield slightly more positives (and negatives), however, a keyword-based starting point runs the risk of biasing the whole active learning pipeline. In what follows, we present a guided sampling approach similar to Palakodety et al. [2020]. Guided Sampling. When faced with the challenge to find high-quality positive examples championing the Rohingya community, Palakodety et al. [2020] proposed a documentembedding-based, guided sampling method where annotators provide example short documents conforming to a given label. We employ a similar technique where we asked three annotators to provide five examples each indicating positive and negative stances toward gender equality. For each example, we select 25 unique nearest neighbors in the document embedding space from the unlabeled pool giving equal weightage to tweets from Tbefore and Tafter . This yields 750 samples. Upon annotation and resolving disagreements, we obtain 166 positives, 145 negatives, and 231 neutrals. We note that our sampling method yielded substantially more positives (and negatives) than the random sampling baseline. Table 7 presents a few randomly selected positive and negative seed examples provided by our annotators. We observe that the examples are grounded in women's cultural struggle in Iran [Kazemzadeh, 2002]. Beyond discussions around hijab, inequality in marital and inheritance law [Doherty et al., 2021], restrictions on activities such as visiting stadiums to watch football [Lewis, 2019; Abtahi et al., 2022] echoed in these examples. Table 8 lists a random sample of retrieved tweet texts when we used the guided sampling method. This table shows that not only we found more positives (and negatives) than the random baseline, but the tweet texts also exhibit richness, diversity, and nuance. Overall, we obtain 343 positives, 440 negatives, and 1,051 neutrals from the random sampling and guided sampling step. In what follows, we describe two well-known sampling strategies that we employ to further expand our dataset. ## 6.3 Certainty and Uncertainty Sampling Certainty sampling. Since our goal is to use the trained model for a social inference task, it is important to rectify high-confidence misclassifications. Minority class certainty sampling has found its use in rectifying high-confidence misclassifications involving short documents such as movie reviews and messages [Sindhwani et al., 2009; Attenberg et al., 2010]; search queries [KhudaBukhsh et al., 2015]; and comments on YouTube videos [Palakodety et al., 2020; Yoo and KhudaBukhsh, 2023]. We conduct certainty sampling for the positive class and select 750 instances that the model predicts as positive with the highest confidence. We also conduct certainty sampling for the negative class and select 750 instances that the model predicts as negative with the highest confidence. In this step, we obtain 338 positives, 345 negative, and 487 neutrals. Uncertainty sampling. Uncertainty sampling is one of the most well-known sampling strategies used in active learning [Settles, 2009]. Since we have multiple label categories in our prediction task, we use margin sampling, an active learning variant designed for multiple labels [Scheffer et al., 2001]. In this step, we sample 1,500 examples. Upon annotation and resolving the disagreements, we obtain 115 positives, 247 negatives, and 819 neutrals. To summarize, our active learning pipeline consists of the following steps: 1. Construct an initial seed set by randomly sampling from Drandom, and Dgender, and using guided sampling (Dseed : 343 positives, 440 negatives, and 1,051 neutral instances) using random sampling. 2. Conduct certainty sampling on the positive class and certainty sampling on the negative class (Dcertainty : 338 positives, 345 negatives, and 487 neutral instances). 3. Finally, conduct uncertainty sampling (margin sampling) (Duncertainty : 115 positive, 247 negative, and 819 neutral instances). Table 7: Random sample of seed examples presented by annotators. Blue indicates a positive stance toward gender equality and red indicates a negative stance toward gender equality. | Seed examples produced by annotators | Translation | |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | توانایی و داره عقل خودش و است بالغی آدم که زن یک چرا دیگه مرد نفر یک از باید دیگه، کشور یه بره که داره اینو نسبتی هر با حالا بگیره؟ خروج اجازه | Why does a woman, who is an adult, and has the abil- ity to go to another country, need to seek another man's permission to leave regardless of whatever the relation is? | | و بیاندازیم کنار را مان استانداردهای در تفاوت همه این باید روا هم زنها بر دانیم می مناسب مردها برای را رفتاری اگر که زنی و مرد بین نیست فرقی هیچ که کنیم درک و بداریم صدای با یا رقصند می یا کشند می سیگار مهمانی یک توی خندند می بلند | We have to put aside all the differences in our standards and if we consider a behavior appropriate for men, we should allow women as well, and understand that there is no difference There is no difference between a man and a woman who smoke or dance or laugh loud at a party. | | باشند داشته آقایون با برابر حقوق طلاق هنگام باید خانمها باشند. داشته را و... طلاق حق و بچه حضانت حق یعنی | When it comes to divorce, women should have the same rights as men, that means, they should have the right to custody of the child and the right to divorce, etc. | | توی که رو حجاب بی دخترای این پوشش وضعیت این آدم توی رو آزادی اینا که میشه مطمئن بیشتر میبینه خیابون نیستن. هیچی دنبال این از غیر و میبینن شدن لخت همون | When you see these hijabless girls' attire in the street, you can be sure that they see freedom in being naked and they are not looking for anything else. | | خونه تو بشینن خب دارن دوست فوتبال خیلی اگه دخترا استادیوم برن توجه جلب برای نمیخواد ببینن، | If girls like football so much, they can sit at home and watch it. They don't have to go to the stadium and attract attention. . | | خونواده آور نون مردا باشه؛ مرد نصف زن دیه باید که معلومه هست، نفعشونم به میرسه بچهه و زن به دیه تازه و هستن ندید قانون به الکی گیر | It is obvious that a woman's blood money should be half a man's. Men are the breadwinners of the family, and the blood money goes to the women and children. Do not tinker around with the law for no good reason. | Figure 2: Temporal trend of tweets expressing positive and negative stance toward gender inequality on Dbaseline . ![Image](https://ijcai.org/proceedings/2023/fulltext/667/image_000001_896dd6309f4c1348f0cd7a38aa168b6127d6c9d0427447c84c327de2c9257fbf.png) Overall, we obtain 796 positive, 1,032 negative, and 2,357 neutral examples. ## 6.4 Model Performance and Analysis Table 9 summarizes the performance of our trained models. The performance improves at each active learning step and we finally achieve a Macro F1 performance of 73.27%. We note that if we train a binary classifier with just the positive class and the negatives and neutrals clubbed together as the notPositive class, it is possible to achieve slightly better performance (Macro F1: 77.76 ± 2.29). To track shifts in stance toward gender equality, we run inference using Mcertainty on Dbaseline. Figure 2 indicates that the discourse became more polarized during Tafter with both percentages of tweets expressing positive and negative stances increasing. However, we also observe that the increase in positive discourse (by a factor of 2.89) is greater than the increase in negative discourse (by a factor of 1.50). ## 7 Discussions In this paper, we present the first-ever computational analysis (to the best of our knowledge) of the stance toward gender equality in Persian Twitter discourse following a watershed moment in Iran's history. Our analyses reveal that the grievances of Persian Twitter users against the government span decades and the protest following Mahsa Amini's death perhaps presented an outlet for the angst harbored for a long time. Second, we observe that the distribution of account creation time can present important signals. We find that with respect to account creation time, pro-execution and statealigned user sets are distributionally different from baseline Persian Twitter users. We follow an ensemble active learning pipeline to construct a robust classifier that detects stance toward gender equality. As a step towards participatory AI, our annotators take an active role in building our machine learning model. There is a growing concern that our ML conversations barely include marginalized community which can further widen the gap of AI-haves and AI-have-nots. All our annotators are Iranian women, with first-person experience of gender struggles. Their role in our system was far more profound than typical annotators. In a guided sampling step, they provided seed examples to expand our dataset lending cultural grounding. They also suggest important keywords to curate our dataset. Table 8: Random sample of tweet texts retrieved through guided sampling. Blue indicates a positive stance toward gender equality and red indicates a negative stance toward gender equality. | Examples obtained through guided sampling | Translation | |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | برای خانوادگی نام من برای زایمان درد تو! برای حضانت حق خانه بیگاری تو! نام به خانه سند من برای خانواده زحمت تو! صیغه هزار من برای عشق حسرت تو! برای عقد چهار من برای من برای عفاف تو! برای هوس من برای سنگسار حکم تو! برای مرد؟ و زن حقوق برابری بود این | Custody right for you! Labor pain for me Surname for you! Family trouble for me The document of the house in your name! Hardwork in home for me Four marriages for you! Missing love for me A thousand concubines for you! A sentence of stoning for me Lust for you! chastity for me Are these equal rights for men and women? | | کسی به تنها نه من ثبتی نسبی، ، خونی های خانم حجاب انسان یک زن نیست مربوط هم بنده به بلکه نیست مربوط مردی هیچ برده زنان بردارید مالکیت از دست هست مستقل بلکه خود به تنها نه میکنند مرد تسلیم را خود که آنان نیستند میکنند خیانت هم دیگر زنان به | Neither me nor anyone can tell my relatives if they should wear a hijab or not. Women are not slaves of men. Those women who submit themselves to men are not only betray- ing themselves, but also betraying other women. | | که بیاره دلایلی مگر شه جدا نمیتونه زن ایران دیگه میدونی . داره و حق اون دیفالت بای مرد یک که درحالی !. . . . هم و اوردی دستش به تازه شما و اول. از داشته همسرشما . . شدی.. سطح | You know, Iranian women can't ask for divorce unless they give reasons.! While a man by default has that right. Your husband had it from the beginning. And you just earned it and leveled up.. . . | | کنند می کار زنان از بهتر مردا | Men work better than women. | | ؟!! مقصرند مردها ، بگیرن رو حقشون ندارن عرضه زنها خود در حضور حق گرفتن برای زنها که شنیدی حالا تا بار چند مردها از ولی ؟ کنن تجمع فدراسیون جلوی برن استادیوم متاسفانه . بشه حمایت زنها از تا نرن استادیوم به که دارن انتظار . بلدن رو زدن غر فقط ایران زنان جامعه اکثریت | Is this men's fault that women themselves do not have abil- ity to take their rights?!! How many times have you heard that women gather in front of the federation to get the right to attend the stadium? But they expect men not to go to the stadium to support women. Unfortunately, the majority of Iranian women only know how to nag . | | رو زن شخصیت میخوان ها بعضی که بیشتری آزادی اون خودش برای هم این و میکنه ظاهری های زیبایی به منحصر داره ضرر جامعه برای هم و | The more freedom people want for women, the more women's character gets overshadowed by physical beauty, which is harmful to both herself and the society. | Table 9: Performance comparison of models trained on various stages of our active learning pipeline. Mseed denotes a popular Persian language model [Farahani et al., 2021] trained on Dseed. Subsequent models are fine-tuned on top of this. For all models trained in this paper, performance is reported over five different training runs on a fixed evaluation set of randomly sampled 400 instances from our annotated dataset ensuring no overlap between train and tests. | Data | Model | Macro F1 | |---------------------------------|--------------|--------------| | Dseed | Mseed | 67.09 ± 1.46 | | Dseed ∪ Dcertainty | Mcertainty | 69.28 ± 0.61 | | Dseed ∪Dcertainty ∪Duncertainty | Muncertainty | 73.27 ± 1.87 | Table 4 suggests that a major Iranian grievance is limited access to the internet. Free and fair access to Twitter was many users' wish. While we were working on this paper, Twitter as a platform underwent several significant changes. With the deprecation of academic Twitter and developer accounts being monetized, at this point, it is unclear howmuch of our collected data would be accessible in the future and at what cost. This is a curious juxtaposition of a community longing for access to a platform to voice their concerns while the very same platform is limiting academic researchers' access to study global politics. On top of the current uncertainties surrounding Twitter, the inherently transient nature of the social web, censorship, and fear of persecution can contribute to missing content for posthoc analyses. In that sense, our paper is a humble attempt to preserve a vulnerable chunk of the social web that chronicled a watershed moment in the gender struggles of Iranian history. ## Ethical Statement We use publicly available tweets collected using academic Twitter API. Since our data is highly sensitive, we only conduct aggregate analyses without revealing personally identifiable information. We also do not conduct any post-annotation adjudication steps that are typical to many annotation tasks to ensure the privacy of the annotators. We trained our model on top of a large language model. Several lines of recent research have indicated that large language models have a wide range of biases that reflect the texts on which they were originally trained, and which may percolate to downstream tasks [Bender et al., 2021]. ## Acknowledgements We thank Lipika Mazumdar and Shawnee Fereydouni for their input. ## References 1. [Abtahi et al., 2022] Zahra Abtahi, Leila Zahedi, Zarrin Eizadyar, and Nicole M Fava. # bluegirl: A study of collective trauma on twitter. Journal of Traumatic Stress, 35(6):1631-1641, 2022. 2. [Ali and Ali, 2018] Luman Ali and Luman Ali. Reacting to iran's descent into chaos. British Diplomacy and the Ira- nian Revolution, 1978-1981, pages 67-99, 2018. 3. [Alkhaldi and Mostaghim, 2022] Celine Alkhaldi and Ramin Mostaghim. Iranian police say death of Mahsa Amini 'unfortunate' as protestors take to the streets, 2022. 4. CNN. 5. [Alkhaldi, 2022] Celine Alkhaldi. Iranian woman takes part in international chess tournament without mandatory hijab. https://www.cnn.com/2022/12/28/sport/iran-sara- khadem-chess-tournament-hijab-spt-intl/index.html, 2022. CNN. 6. [Attenberg et al., 2010] Josh Attenberg, Prem Melville, and Foster Provost. A unified approach to active dual supervi- sion for labeling features and examples. In ECML/PKDD, pages 40-55, 2010. 7. [Bender et al., 2021] Emily M. Bender, Timnit Gebru, An- gelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In ACM FaccT, pages 610-623, 2021. 8. [Berger and Morgan, 2015] Jonathon M. Berger and Jonathon Morgan. The ISIS Twitter Census: Defining and describing the population of ISIS supporters on Twitter. Brookings Institution, 2015. 9. [Birhane et al., 2022] Abeba Birhane, William Isaac, Vinod- kumar Prabhakaran, Mark Díaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. Power to the people? opportunities and challenges for participatory ai. Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1-8, 2022. 10. [Bojanowski et al., 2017] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vec- tors with subword information. Transactions of the Asso- ciation for Computational Linguistics, 5:135-146, 2017. 11. [Bondi et al., 2021] Elizabeth Bondi, Lily Xu, Diana Acosta-Navas, and Jackson A. Killian. Envisioning communities: a participatory approach towards ai for social good. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 425-436, 2021. 12. [Bozorgmehr, 2017] Shirzad Bozorgmehr. Six days that shook Iran. http://news.bbc.co.uk/2/hi/middle east/ 828696.stm, 2017. CNN. 13. [Davidson et al., 2017] Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Pro- ceedings of the international AAAI conference on web and social media, volume 11, pages 512-515, 2017. 14. [Delgado et al., 2022] Fernando Delgado, Solon Barocas, and Karen Levy. An uncommon task: Participatory design in legal ai. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW1):1-23, 2022. 15. [Doherty et al., 2021] William J. Doherty, Seyed Moham- mad Kalantar, and Mahdieh Tarsafi. Divorce ambiva- lence and reasons for divorce in Iran. Family process, 60(1):159-168, 2021. 16. [Farahani et al., 2021] Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, and Mohammad Man- thouri. Parsbert: Transformer-based model for Persian language understanding. Neural Processing Letters, 53:3831-3847, 2021. 17. [Fortuna and Nunes, 2018] Paula Fortuna and Sérgio Nunes. A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), 51(4):1-30, 2018. [France-Presse, 2022] Agence France-Presse. Iran protests: Joe Biden says US stands with 'brave women' after Mahsa Amini death. https://www.theguardian.com/us- news/2022/oct/15/iran-protests-joe-biden-says-us- stands-with-brave-women-after-mahsa-amini-death, 2022. The Guardian. 18. [Gao and Huang, 2017] Lei Gao and Ruihong Huang. De- tecting online hate speech using context aware models. In Ruslan Mitkov and Galia Angelova, editors, RANLP 2017, pages 260-266. INCOMA Ltd., 2017. 19. [Golbeck et al., 2017] Jennifer Golbeck, Zahra Ashktorab, Rashad O Banjo, Alexandra Berlinger, Siddharth Bhag- wan, Cody Buntain, Paul Cheakalos, Alicia A Geller, Quint Gergory, Rajesh Kumar Gnanasekaran, et al. A large labeled corpus for online harassment research. In Proceed- ings of the 2017 ACM on web science conference, pages 229-233, 2017. 20. [Gomez et al., 2020] Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas. Exploring hate speech detection in multimodal publications. In Proceedings of the IEEE/CVF winter conference on applications of com- puter vision, pages 1470-1478, 2020. 21. [Guest et al., 2021] Ella Guest, Bertie Vidgen, Alexandros Mittos, Nishanth Sastry, Gareth Tyson, and Helen Mar- getts. An expert annotated dataset for the detection of online misogyny. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pages 1336-1350, On- line, April 2021. Association for Computational Linguis- tics. 22. [Hadgu et al., 2013] Asmelash Teka Hadgu, Kiran Garimella, and Ingmar Weber. Political hashtag hi- jacking in the us. In Proceedings of the 22nd international conference on world wide web, pages 55-56, 2013. 23. [Harrington et al., 2019] Christina Harrington, Sheena Erete, and Anne Marie Piper. Deconstructing community- based collaborative design: Towards more equitable participatory design engagements. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1-25, 2019. 24. [Kargar and McManamen, 2018] Simin Kargar and Keith McManamen. Censorship and collateral damage: Ana- lyzing the Telegram ban in Iran. Berkman Klein Center Research Publication, (2018-4), 2018. 25. [Kargar and Rauchfleisch, 2019] Simin Kargar and Adrian Rauchfleisch. State-aligned trolling in iran and the double- edged affordances of instagram. New media & society, 21(7):1506-1527, 2019. 26. [Kazemzadeh, 2002] Masoud Kazemzadeh. Islamic funda- mentalism, feminism, and gender inequality in Iran under Khomeini. University Press of America, 2002. 27. [Kermani, 2023] Hossein Kermani. # mahsaamini: Iranian twitter activism in times of computational propaganda. So- cial Movement Studies, pages 1-11, 2023. 28. [KhudaBukhsh et al., 2015] Ashiqur R. KhudaBukhsh, Paul N. Bennett, and Ryen W. White. Building effective query classifiers: a case study in self-harm intent detec- tion. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 1735-1738, 2015. 29. [Lewis, 2019] Samantha Lewis. Death of Blue Girl shines light on women's rights in Iran. https://www.theguardian.com/football/2019/sep/21/death- of-blue-girl-shines-light-on-womens-rights-in-iran, 2019. The Guardian. 30. [Palakodety et al., 2020] Shriphani Palakodety, Ashiqur R. KhudaBukhsh, and Jaime G. Carbonell. Voice for the voiceless: Active sampling to detect comments supporting the rohingyas. In AAAI 2020, volume 34, pages 454-462, 2020. 31. [Pina, 2022] Christy Pina. Asghar Farhadi Invites Artists to Declare Solidarity With the People of Iran. https://www.hollywoodreporter.com/news/general- news/asghar-farhadi-solidarity-people-of-iran-mahsa- amini-1235227129/, 2022. The Hollywood Reporter. 32. [Radford and Fowler, 2023] Antoinette Radford and Sarah Fowler. Iran protests: Two men hanged over killing of militiaman. https://www.bbc.com/news/world-middle- east-64196635, 2023. BBC. 33. [Ramesh et al., 2022] Krithika Ramesh, Sumeet Kumar, and Ashiqur R. Khudabukhsh. Revisiting queer minorities in lexicons. In Proceedings of the Sixth Workshop on On- line Abuse and Harms (WOAH), pages 245-251, Seattle, Washington (Hybrid), July 2022. Association for Compu- tational Linguistics. 34. [Sanguinetti et al., 2018] Manuela Sanguinetti, Fabio Po- letto, Cristina Bosco, Viviana Patti, and Marco Stranisci. An italian twitter corpus of hate speech against immi- grants. In Proceedings of the eleventh international con- ference on language resources and evaluation (LREC 2018), 2018. 35. [Scheffer et al., 2001] Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden Markov models for in- formation extraction. In International Symposium on In- telligent Data Analysis, pages 309-318. Springer, 2001. 36. [Settles, 2009] Burr Settles. Active learning literature sur- vey. University of Wisconsin-Madison Department of Computer Sciences, 2009. 37. [Sindhwani et al., 2009] Vikas Sindhwani, Prem Melville, and Richard D. Lawrence. Uncertainty sampling and transductive experimental design for active dual supervi- sion. In ICML, pages 953-960, 2009. 38. [Sohrabi, 2021] Hadi Sohrabi. New media, contentious pol- itics, and political public sphere in Iran. Critical Arts, 35(1):35-48, 2021. 39. [Stella et al., 2018] Massimo Stella, Emilio Ferrara, and Manlio De Domenico. Bots increase exposure to neg- ative and inflammatory content in online social sys- tems. Proceedings of the National Academy of Sciences, 115(49):12435-12440, 2018. 40. [Wiegand et al., 2019] Michael Wiegand, Josef Ruppen- hofer, and Thomas Kleinbauer. Detection of abusive lan- guage: the problem of biased datasets. In Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies, volume 1 (long and short papers), pages 602-608, 2019. 41. [Yang and Carbonell, 2013] Liu Yang and Jaime Carbonell. Buy-in-bulk active learning. Advances in neural informa- tion processing systems, 26, 2013. 42. [Yoo and KhudaBukhsh, 2023] Clay H. Yoo and Ashiqur R. KhudaBukhsh. Auditing and Robustifying COVID-19 Misinformation Datasets via Anticontent Sampling. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 37(12):15260-15268, 2023.