Estimating Contribution Quality in Online Deliberations Using a Large Language Model

Analysis

Why This Paper Matters

Online deliberation platforms are increasingly used to address polarization and foster democratic discourse at scale. However, assessing the quality of contributions—a key metric for evaluating platform features and interventions—has traditionally required expensive and time-consuming human annotation. This paper tackles that bottleneck by demonstrating that a large language model can serve as a viable proxy for human quality ratings in structured online discussions. The significance is twofold: it offers a practical, scalable tool for platform operators and researchers, and it provides empirical evidence that nudges (prompts after inactivity) can increase participation without degrading the quality of contributions, a finding with direct implications for designing more effective deliberation systems.

The work is grounded in real-world data from large-scale events, including a collaboration with Meta spanning 32 countries and another involving 38 US post-secondary institutions. This ecological validity strengthens the case for deploying LLM-based quality estimation in production settings. By showing that the model is competitive with individual human annotators and only slightly behind small groups, the paper opens the door to automated quality monitoring that can run continuously and at low cost.

Technical Contributions

• Automated quality rating framework: The authors define four specific metrics for contribution quality—justification, novelty, expansion of the conversation, and potential for further expansion—and operationalize them with a 1–5 rating scale, enabling both human and LLM annotation.
• Comparative evaluation methodology: Using the average of eight human annotators as ground truth, the paper systematically compares the LLM's performance against individual humans and human pairs/triples, providing a clear benchmark for automated quality estimation.
• Application to nudge analysis: The paper demonstrates a concrete use case by measuring the effect of nudges on both participation rates and contribution quality, showing that nudges increase the likelihood of speaking by 65% while maintaining similar quality levels.
• Scalable annotation pipeline: The approach reduces the need for human annotation, making it feasible to analyze large-scale deliberation datasets that would otherwise be prohibitively expensive to annotate manually.

Results

• The LLM outperforms individual human annotators when compared against the average human rating as ground truth.
• Pairs of human annotators outperform the LLM on the justification metric; groups of three outperform it on all four metrics.
• Despite this, the LLM remains competitive, suggesting it can serve as a cost-effective alternative to human annotation in many scenarios.
• Nudges after prolonged inactivity increase the likelihood of an individual requesting to speak in the next 30 seconds by 65%.
• Quality ratings for statements prompted by nudging are similar to those made without nudging, indicating that nudging generates more ideas without sacrificing quality.

Significance

This research advances the field of AI-assisted social science by providing a validated method for automated quality assessment in online deliberation. It bridges the gap between human annotation and machine learning, offering a practical tool for researchers and platform designers. The finding that nudges can boost participation without harming quality has immediate implications for designing more inclusive and productive deliberation environments. More broadly, the work demonstrates how LLMs can be used to evaluate complex social constructs like contribution quality, paving the way for similar applications in other domains such as online education, collaborative writing, and community moderation. By reducing reliance on costly human annotation, this approach enables larger-scale studies and faster iteration on platform features, ultimately supporting more effective democratic engagement.