5.2.1.1. Collection of the English corpus

Although other corpora had already been developed (Reyes et al. 2013), we decided to construct our own corpus of ironic and non-ironic tweets in English. The existing corpora were essentially composed of personal tweets (e.g. Don’t worry about what people think. They don’t do it very often), meaning that they were not comparable to the corpus used for research in French. To create our corpus, we followed the same collection procedure used for the French corpus FrIC. We selected a set of themes belonging to the same categories used previously, adapted to language and culture-specific news. For example, for the politics category, we selected Obama, Trump, Clinton, etc.; for artists, we selected keywords including Justin Bieber, Kardashian, Beyoncé, etc. Next, we selected ironic tweets containing these keywords along with the hashtag #ironic or #sarcasm. Non-ironic tweets (e.g. tweets without the #ironic or #sarcasm hashtags) were collected in the same way.

Following collection, we removed all duplicates, retweets and tweets containing images. At the end of the filtering process, we had a corpus of 11,289 tweets, of which 5,173 were ironic and 6,116 were non-ironic. The distribution of these tweets by category is shown in Table 5.1.

5.2.1.2. Collection of the Italian corpus

For Italian, we made use of two existing resources, annotated as part of the Senti-TUT project²:

• – The Sentipolc corpus used in the Evalita 2014 evaluation campaign³ for sentiment analysis and irony detection in tweets (Basile et al. 2014). The Sentipolc corpus is a collection of tweets in Italian derived from two existing corpora: Senti-TUT (Bosco et al. 2013) and TWITA (Basile and Nissim 2013). It contains tweets using keywords and hashtags on the theme of politics (names of politicians, etc.). Each tweet in Sentipolc is annotated using five mutually exclusive categories: positive opinion, negative opinion, both positive and negative opinion, ironic and objective.
• – The TW-SPINO corpus, containing tweets from Spinoza⁴, a satirical political blog in Italian. These tweets were selected and reviewed by a team of editors who identified them as being ironic or satirical.

The Italian corpus is made up of 3,079 ironic tweets (806 from Sentipolc and 2,273 from TW-SPINO) and 5,642 non-ironic tweets (from Sentipolc).

5.2.2.1. Quantitative results for ironic/non-ironic annotation

Based on the reference hashtags #ironic and #sarcasm in the English corpus, 440 (80%) of the tweets were ironic and 110 (20%) were non-ironic. The human annotators assessed 427 (77.63%) of the tweets as ironic and 99 (18%) as non-ironic, with the remaining 24 tweets (4.37%) being placed into the “no decision” class (Figure 5.2). These results show that, whatever the language used, a tweet including an irony hashtag is not necessarily ironic, nor is the hashtag a prerequisite for irony.

Unlike the French and English corpora, the Italian corpus was annotated as ironic/non-ironic by humans instead of using hashtags (as part of the Senti-TUT project). This corpus contains 400 ironic tweets and 100 non-ironic tweets, with no tweets in the “no decision” class (Figure 5.2).

5.2.2.2. Quantitative results for irony type annotation

Table 5.2 shows the total number of annotated tweets and the type of irony trigger in each corpus.

In the English corpus, out of 427 tweets annotated as ironic, 283 featured implicit contradiction and 144 contained explicit contradiction. This indicates that irony is generally expressed implicitly in English (66.28% of cases), as it is in French (73.01%).

In the Italian corpus, however, the majority of ironic tweets featured explicit contradictions (65%). This may stem from the fact that Italian users do not use specific hashtags to indicate irony, perhaps causing them to be more explicit in their expressions.

5.2.2.3. Quantitative results for irony category annotation

Table 5.3 shows the percentage of tweets belonging to each category of irony, split into explicit/implicit trigger groups⁶. Significant differences can be seen in terms of the categories of irony found in the French, English and Italian corpora. The results show that:

• – for irony including explicit contradiction, oxymoron/paradox is the most common category for all three languages (French, English and Italian);
• – for irony with implicit contradiction, false assertion and other are the most common categories in French and English. In Italian, the most frequent categories are false assertion, analogy and other;
• – considering the tweets in the other category, we see that the majority are ironic with implicit contradiction. This indicates that the decision task is harder for humans in cases where irony is expressed through an implicit contradiction, whichever language is used.

As the classes are not mutually exclusive:

• – for the English corpus, 35 tweets with explicit contradiction belong to more than one category, and 62 tweets with implicit contradiction belong to more than one category. For explicit contradictions, the most frequent combination is oxymoron/paradox + rhetorical question, while for ironic tweets with implicit contradiction, the most frequent contradiction is metaphor/comparison + other;
• – for the French corpus FrIC, the most frequent combination for explicit contradictions is oxymoron/paradox + hyperbole/exag-geration; for implicit contradictions, the most frequent combination is false assertion + hyperbole/exaggeration;
• – for the Italian corpus, the annotators chose to assign a single category to each tweet, selecting the option which seemed to express irony most strongly.

Thus, for French and English, the oxymoron/paradox category is one of the most frequent combinations for irony with explicit contradiction. For cases featuring implicit contradiction, the combinations are different for the two languages.

5.2.2.4. Quantitative results of the annotation procedure for irony markers

Three statistical studies were carried out for these levels. The first is a quantitative study looking at the first and fourth levels of the annotation scheme, concerning the presence of different markers in ironic and non-ironic tweets (Table 5.4). The second study concerns the second and fourth levels, focusing on the presence of different markers in tweets with explicit versus implicit contradiction (Table 5.4). Finally, our third study relates to the third and fourth levels, specifically the presence of markers in each irony category (Table 5.5).

Table 5.4 indicates the percentage of tweets containing markers in the ironic category (making a distinction between explicit/implicit) and the non-ironic category (NIR, in gray).

In French, the intensifier, punctuation and interjection markers were most common in ironic tweets, while quotations were most frequent in non-ironic tweets.

In English, the discursive connectors, quotation, comparison words and reporting verbs were twice as common in ironic tweets than in non-ironic tweets; the reverse is true for personal pronouns. Note that the English corpus does not contain any ironic tweets including URLs, as all tweets of this type were annotated as “no decision”: the annotators were unable to understand the tweet and the content of the web page linked to the URL.

In Italian, most markers were more common in ironic tweets, although some, such as quotations and URLs, occurred more frequently in non-ironic tweets.

Table 5.5 indicates the percentage of tweets containing markers in each irony category.

Negation is most common in the euphemism category for French, and in the register shift, euphemism and rhetorical question categories in English.

Intensifiers were most common in the euphemism and hyperbole categories in both French and English.

Punctuation occurred most frequently in the register shift, euphemism and rhetorical question categories in French, and in the register shift and rhetorical question categories in English.

False propositions were most common in the hyperbole and false assertion categories in both French and English. Opposition words occurred most often in the oxymoron/paradox category in French and English.

Personal pronouns were most frequent in the register shift category for French, and in the register shift, hyperbole and false assertion categories for English.

Comparison words occurred most frequently in the analogy category for French and English.

Opinion words were most common in the hyperbole category for both French and English.

For Italian, the percentage of tweets containing markers in each irony category was very low, and tweets were distributed almost equally across different categories (for example, negation is found in all categories at between 1% and 3%, and the frequency of opposition words does not exceed 2%).

This set of quantitative studies show that, whatever the language, the authors of ironic tweets tend to use markers such as opinion words, named entities and negation words in categories including analogy, rhetorical question, oxymoron/paradox, false assertion and other.

5.2.2.5. Quantitative results of the annotation procedure for contradiction relationships

Figure 5.3 shows that the distribution of relationships is almost identical in the French and English corpora. For example, the opposition relationship is most common in ironic tweets in French (69%) and in English (80%). Furthermore, the number of comparison relationships is smaller in both English (20%) and French (14%), while cause/consequence relationships are the least frequent in the two languages (6% for English, and 11% for French). Marker annotation was carried out automatically for the Italian corpus, meaning that contradiction relationships were not noted, given that this must be done manually.

5.2.2.6. Correlation between different levels of the annotation scheme

We also carried out a comparative study of correlations between different levels of the scheme in all three languages: French, English and Italian. The approach previously used in French (see Chapter 3) was applied to the English and Italian results.

We looked at the correlation between irony markers and trigger types (explicit or implicit contradiction) and between irony markers and category. The aim of this study was to analyze the extent to which markers may be used to predict irony in the languages in question.

Applying Cramer’s V (Cohen 1988) to the number of occurrences of each marker, we obtained the following results (all of the correlations are statistically significant):

• – between markers and the ironic/non-ironic class:
  • - a strong correlation for French (V = 0.156, df = 14) and Italian (V = 0.31, df = 6);
  • - a medium to strong correlation for English (V = 0.132, df = 9);

• – between markers and irony trigger type (explicit or implicit contradiction):
  • - a strong correlation for French (V = 0.196, df = 16);
  • - a medium to strong correlation for Italian (V = 0.138, df = 5);
  • - a medium correlation for English (V = 0.083, df = 12).

We also analyzed correlations by marker (df = 1). The markers with the strongest class correlation (ironic/non-ironic) were:

• – negations, interjections, named entities and URLs for French (0.14 < V < 0.41);
• – negations, discursive connectors and personal pronouns for English (0.12 < V < 0.17);
• – quotations, named entities and URLs for Italian (0.310 < V < 0.416).

The markers most strongly correlated with triggers (explicit/implicit) are:

• – opposition markers, comparison words and false assertions for French (0.140 < V < 0.190);
• – opposition markers and discursive connectors for English (0.110 < V < 0.120);
• – discursive connectors, punctuation and named entities for Italian (0.136 < V < 0.213).

We noted that, in spite of the high frequency of opinion words in ironic tweets in both French and English, the opinion marker is not correlated with the ironic/non-ironic classification or with explicit/implicit triggers (V < 0.06), as many non-ironic tweets also include opinion words.

Finally, we analyzed the correlation between markers and irony categories. According to the results of Cramer’s V test, the most decisive markers were:

• – intensifiers, punctuation, false assertion and opinion word for French, with strong correlation;
• – negation, discursive connectors and personal pronouns for English, with medium correlation;
• – punctuation, interjections and named entities for Italian, with medium correlation.

5.3.2.1. Collection of the first Arabic corpus for irony

Given the absence of an existing corpus of ironic tweets in Arabic, we followed the same procedure as for French and English. Initially, we planned to use the same categories as for the French corpus (politics, economics, health, etc.) with themes specific to the Arab world. However, it rapidly became apparent that the vast majority of ironic tweets concerned political subjects. For this reason, we only collected tweets in the politics category, using a set of five themes: (Hillary), (Trump), (Al-Sissi: the Egyptian president), (Mubarak: former Egyptian president) and (Morsi: former Egyptian president).

To construct a corpus of ironic/non-ironic tweets, we harvested examples containing the hashtags and (translations of #irony and #sarcasm). Tweet (5.1) is an example of an ironic message, while (5.2) is non-ironic.

Tweet (5.2) is written in standard Arabic. Tweet (5.1) combines standard Arabic with a single Egyptian/Tunisian dialect word, (not).

(5.1)
	(Actually, it is obvious that the military coup against President-Elect Morsi was in the best interests of Egypt, and not of ambitions, and now conflicts, regarding the presidency #irony)
(5.2)
	(Life changes, but women never change. That’s what I learned from Hillary Clinton’s tears when she was defeated by Trump)

After the collection stage, we removed duplicates, retweets and tweets containing images. This filtering process left us with a corpus of 3,479 tweets, of which 1,733 were ironic and 1,746 were non-ironic. The corpus included tweets in both standard and colloquial Arabic, and in the majority of cases, a combination of the two forms. Given that Twitter’s API does not distinguish between standard Arabic and colloquial Arabic or between different dialects, several dialects are present in the corpus, including Egyptian, Syrian and Saudi. Other dialects, such as Tunisian and Algerian, occur less frequently. For the purposes of our study, the hashtags and (#irony and #sarcasm) were removed.

5.3.2.2. Linguistic resources

Our automatic detection approach, described in the previous chapter, makes use of dedicated lexicons to identify opinion words, intensifiers, negations, emotions, etc. To study the portability of our system for Arabic, we looked for existing lexicons (for both standard and dialect forms of the language). Some of the lexicons we found performed well in the context of irony detection; others, less so. We constructed our own lexicons to replace those in the latter category.

The following linguistic resources were used:

• – a lexicon of Arabic discursive connectors based on work by Iskandar (Keskes et al. 2014). This lexicon includes 416 connectors, such as (furthermore) and (thus);
• – a lexicon of 4,501 named entities Keskes et al. (2014), to which we added the entities used for tweet collection, e.g. (Clinton);
• – a lexicon of 119 reporting verbs used by Keskes et al. (2014), such as (to say) and (to announce);
• – a lexicon of opinion words, made up of 22,239 negative opinion words and 26,777 positive words. This was obtained by combining two resources: the Arabic Emoticon Lexicon and the Arabic Hashtag Lexicon (dialectal)¹⁰ (Saif et al. 2016). These lexicons were used in Task 7 of the SemEval’2016 campaign^11,12, and include entries such as (failure) and (indignation);
• – a lexicon of 681 emoticons, used previously for French;
• – a lexicon of personal pronouns and a lexicon of negation words, which we constructed manually, including terms such as (I), (we), or (not/no/isn’t);
• – a lexicon of 25 intensifiers, translated from a set of intensifiers used for French, including (lots) and (very).

5.3.3.1. Features used for irony detection

For the French corpus, our system used 30 features, eight of which were obtained using morphosyntactic pre-processing tools such as MElt. In the absence of a satisfactory analyzer for texts in standard and colloquial Arabic, we manually selected a subset of 22 features, which can be extracted without these tools. These were divided into four groups, as shown in Table 5.8.

5.3.3.2. Experiments and results

To classify tweets as ironic/non-ironic, we used all of the features defined above, testing several classifiers using the Weka platform: SMO, Naive Bayes, multinomial logistic regression, linear regression, random tree and random forest. We trained the classifiers using a balanced corpus of 1,733 ironic tweets and 1,733 non-ironic tweets. For the first experiment, 80% of the corpus was used for training and 20% for testing. For the second experiment, we used 10-fold cross-validation. We chose to focus on these two experiments due to the limited size of the corpus (3,466 tweets). The best results were obtained using the random forest classifier with default parameters. These are shown in Table 5.9.

Three feature selection algorithms were applied using Weka with the aim of improving our results: Chi2 and GainRatio (see Table 5.10) to obtain a list of features in decreasing performance order (best to worst), and CfsSubsetEval option to obtain the best feature combination, considering the individual predictive capacity of each feature and the degree of redundancy between features. The last algorithm indicated that the combination of number of emoticons, exclamations, negations, number of interjections, named entities and number of named entities would produce the best results.

Unfortunately, this subset of features identified by the selection algorithms failed to produce better results than those obtained using all features. We thus tested a different approach, adding features one by one in the training process for the random forest classifier (following the order given by the selection algorithm) in order to assess the influence of each feature. In this, we aimed to identify a subset of features, which would maximize performance.

This approach showed that the use of all features with the exception of reporting verb produced the best results, with an accuracy value of 72.76% compared to 72.36% for all features, an F-measure of 73% instead of 72.70% for the ironic class and an F-measure of 72.50% instead of 72.10% for the non-ironic class (see Table 5.11).

5.3.3.3. Discussion

Although most of the features used here are surface features, the results obtained are very encouraging. Comparing the results obtained for the classification of Arabic tweets into ironic/non-ironic sets with those obtained using the same features in French, we see that the classification algorithms behave differently in the two languages. In French, the SMO classification algorithm was most effective, with an F-measure of 85.70% for the ironic class; in Arabic, the F-measure did not exceed 62.50% using this approach. The random forest classification algorithm performed better for classifying ironic tweets in Arabic, with an F-measure of 73%, but its performance in French was lower than that of the SMO algorithm, with an F-measure of 75.40%.