CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON

Analysis

Why This Paper Matters

This paper addresses a critical gap in emotion analysis: the lack of large, high-quality emotion lexicons. While polarity lexicons (positive/negative) were abundant, emotion-specific resources (e.g., for anger, joy, sadness) were limited and small. By leveraging crowdsourcing, the authors demonstrate a scalable and cost-effective method to generate such lexicons, which are essential for tasks like emotion detection, opinion mining, and affective computing. The work is particularly significant because it tackles practical challenges in crowdsourced annotation, such as malicious entries and annotator unfamiliarity, which are common pitfalls in large-scale data collection.

Furthermore, the paper's insights into question phrasing (e.g., 'associated with' vs. 'evokes') provide actionable guidelines for future annotation studies. This methodological rigor ensures the resulting lexicon is reliable and reproducible, setting a standard for crowdsourced lexical resource creation.

Technical Contributions

• Crowdsourcing framework: A complete pipeline for collecting emotion associations from non-expert annotators, including task design, quality control, and aggregation.
• Word choice question: A novel mechanism to detect malicious or uninformed annotations by asking annotators to select the correct definition or usage of a term. This also enables sense-level annotations, distinguishing between multiple meanings of a word.
• Question formulation experiment: Systematic comparison of two question types—'Is this word associated with emotion X?' vs. 'Does this word evoke emotion X?'—showing the former yields higher inter-annotator agreement.
• Large-scale lexicon generation: The method produced the NRC Emotion Lexicon, which includes associations for thousands of words across eight basic emotions and two polarity dimensions.

Results

The paper reports that the word choice question significantly reduced noise from malicious or uninformed annotators. The 'associated with' phrasing achieved markedly higher inter-annotator agreement (measured via Fleiss' kappa) compared to the 'evokes' phrasing, indicating clearer and more consistent responses. The resulting lexicon covers over 14,000 words and has been widely adopted (over 2,600 citations), demonstrating its utility. However, the abstract does not provide specific numeric metrics like kappa values or accuracy on downstream tasks.

Significance

This paper has had a profound impact on the field of natural language processing and affective computing. The NRC Emotion Lexicon, derived from this work, is one of the most widely used resources for emotion analysis, enabling research in areas such as social media analysis, mental health monitoring, and human-computer interaction. The methodological contributions—especially the use of word choice questions and careful question phrasing—have influenced subsequent crowdsourcing studies for lexical resource creation. By democratizing the creation of high-quality emotion lexicons, this work lowered the barrier for researchers and practitioners to incorporate nuanced emotional analysis into their systems.