Big Data: New Tricks for Econometrics

Analysis

Why This Paper Matters

Hal Varian's 2014 essay "Big Data: New Tricks for Econometrics" is a landmark paper that helped bridge the gap between traditional econometrics and modern machine learning. At a time when big data was becoming ubiquitous in economics—from transaction records to online behavior—Varian recognized that conventional statistical tools like linear regression were insufficient for handling the scale, dimensionality, and complexity of these datasets. The paper's significance lies in its clear articulation of three core challenges: data manipulation at scale, variable selection when predictors outnumber observations, and the need for flexible nonlinear models. By introducing economists to machine learning techniques such as decision trees, support vector machines, neural networks, and deep learning, Varian provided a practical roadmap for modernizing economic analysis.

This paper matters because it democratized access to machine learning for a field traditionally rooted in parametric statistics. It encouraged economists to think beyond linearity and to embrace tools that could automatically discover patterns in high-dimensional data. The essay's accessible tone and concrete examples made it a go-to reference for practitioners, leading to its high citation count (1559) and lasting influence on applied econometrics.

Technical Contributions

• Problem Framing: Clearly identifies three unique challenges of big data for economists: (1) data manipulation (handling large datasets efficiently), (2) variable selection (choosing predictors when p >> n), and (3) flexible modeling (capturing nonlinear relationships).
• Tool Introduction: Introduces machine learning techniques as solutions: decision trees for interpretable splits, support vector machines for classification, neural nets and deep learning for complex function approximation.
• Practical Guidance: Offers advice on data preprocessing, cross-validation, and regularization (e.g., lasso) to avoid overfitting—concepts that were less common in econometrics at the time.
• Conceptual Bridge: Connects machine learning concepts to econometric principles, such as bias-variance tradeoff and model selection criteria.

Results

As an essay, the paper does not present experimental results or quantitative comparisons. Its impact is measured by its influence: 1559 citations as of the provided metadata, indicating widespread adoption in economics and related fields. The paper's main result is conceptual: it successfully argued that machine learning methods can outperform traditional econometric techniques for big data tasks, particularly when dealing with high-dimensional predictors and complex interactions.

Significance

The broader impact of Varian's essay is profound. It helped catalyze a shift in economics toward data-driven, machine learning approaches, influencing subsequent work in causal inference, prediction markets, and policy evaluation. The paper is often credited with making machine learning accessible to economists, leading to interdisciplinary collaborations and the rise of fields like computational economics. Its legacy is evident in the growing number of economics papers that now routinely use random forests, gradient boosting, and neural networks for both prediction and causal analysis.