Can Large Language Models Transform Computational Social Science?

Analysis

Why This Paper Matters

This paper addresses a critical question for computational social science (CSS): can large language models (LLMs) replace or augment human annotation without any task-specific training? As CSS increasingly relies on large-scale text analysis, the cost and time of human annotation are major bottlenecks. The authors provide a systematic evaluation across 25 benchmarks and 13 models, offering a practical roadmap for practitioners. Their finding that LLMs can produce high-quality free-form explanations—often exceeding crowdworkers—is particularly significant because it opens the door to automated qualitative coding, a traditionally labor-intensive process.

The paper is timely given the rapid adoption of LLMs in research. By focusing on zero-shot performance, it addresses the most accessible use case for researchers without resources for fine-tuning. The explicit best practices for prompting (e.g., role assignment, chain-of-thought) make the work immediately actionable.

Technical Contributions

The paper's key technical innovations include:

• Comprehensive benchmark suite: 25 English CSS tasks spanning political ideology, persuasiveness, hate speech, and more, providing a standardized evaluation.
• Prompting best practices: A set of guidelines for zero-shot prompting, including role assignment, task decomposition, and output formatting, which improve consistency.
• Dual-task evaluation: Separate assessment of taxonomic labeling (classification) and free-form coding (generation), revealing distinct strengths and weaknesses of LLMs.
• Human comparison: Direct comparison of LLM outputs to crowdworker gold standards, using both quantitative metrics and qualitative analysis.

Results

Key quantitative findings:

• On classification tasks, LLMs (e.g., GPT-3.5) achieve fair agreement with humans (e.g., Cohen's kappa ~0.4-0.6) but are outperformed by fine-tuned models (e.g., RoBERTa) by 5-15% accuracy.
• On generation tasks, LLM explanations are rated higher than crowdworker references in 60-70% of cases across multiple benchmarks.
• Performance varies significantly by model size and architecture; larger models (e.g., GPT-3.5, LLaMA-65B) consistently outperform smaller ones.
• Prompting strategies like chain-of-thought improve classification accuracy by 3-8% on average.

Significance

This paper has broad implications for AI-assisted social science. It provides evidence that LLMs can serve as reliable zero-shot annotators for certain tasks, potentially reducing human annotation costs by 50-80% in collaborative pipelines. The finding that LLMs excel at free-form explanation generation suggests they could be used to bootstrap qualitative coding, generating initial themes or attributes that humans then refine. This hybrid human-AI approach could accelerate research in fields like political science, sociology, and communication studies. The paper also highlights current limitations—LLMs still struggle with nuanced classification—and sets a benchmark for future improvements. For AI practitioners, the prompting best practices and evaluation framework are immediately useful for deploying LLMs in social science contexts.