On August 11, 2026, freeCodeCamp News published a detailed tutorial by Rudrendu Paul, an Applied AI/ML and Marketing Measurement Science leader, on implementing doubly robust estimation for causal inference in LLM product experiments. The article, titled "Product Experimentation with Doubly Robust Estimation: When Both Your Models Are Wrong in LLM Applications," walks data scientists through the augmented inverse propensity weighting (AIPW) estimator, demonstrating that it remains valid if either the propensity or outcome model is correctly specified.
Paul, who has 15+ years of experience at Fortune 50 companies and specializes in Causal Inference, Experimentation, and Agentic AI systems, has published with Springer Nature, Elsevier, ICML, and IEEE. The tutorial is aimed at data scientists running noisy AI product experiments, particularly those where users self-select into treatment groups, creating selection bias.
The core promise of AIPW is simple: it combines a propensity model and an outcome model, and the estimator remains consistent if either one is correctly specified. Both models must fail simultaneously for AIPW to break. As Paul puts it, "AIPW gives you one free mistake; the limit is exactly one." This redundancy engineering for causal estimates is compared to keeping distributed systems online when a single node fails.
The tutorial implements AIPW from scratch using scikit-learn, with no causal-inference-specific library needed. The companion notebook is available at github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm/tree/main/12_doubly_robust/, with the notebook file being aipw_demo.ipynb. The repository can be cloned with git clone https://github.com/RudrenduPaul/product-experimentation-causal-inference-genai-llm.git, and requires Python 3.11 or newer with numpy, pandas, scikit-learn, and scipy installed.
The Problem: Self-Selection in LLM Experiments
The tutorial begins with a scenario familiar to many product teams. Six months ago, an AI product shipped an agent-mode opt-in feature. Users could choose to enable the agent mode, and the team wanted to know if it improved task completion rates. The problem is that users self-select into treatment, creating selection bias that a naïve comparison cannot handle.
The synthetic dataset simulates a SaaS product with agent mode opt-in, containing 50,000 users. The data generator command is python data/generate_data.py , seed 42 , n-users 50000 , out data/synthetic_llm_logs.csv, using seed 42. Opt-in rates vary dramatically by engagement tier: heavy users opt in at 65%, medium users at 35%, and light users at only 12%. This creates a clear confounding structure where more engaged users are both more likely to opt in and more likely to complete tasks.
The ground-truth causal effect of agent-mode opt-in is +8 percentage points, baked into the synthetic data generator. Yet the naïve comparison, which simply subtracts the mean outcome of the control group from the treated group, produces a raw difference of +0.2106. This overstates the true effect by nearly a factor of three. In the tutorial, the treated group has 13,451 users, while the control group has 36,549 users.
The article mentions a quarterly business review where a +8 percentage-point lift was reported, and the number made it into the quarterly business review and everyone was pleased. The problem is that this number was likely inflated by selection bias, and a rigorous data scientist will ask about confidence in the propensity model.
The AIPW Estimator: Redundancy for Causal Estimates
The AIPW estimator targets the average treatment effect (ATE) and combines two components: regression adjustment and IPW correction terms. The formula is:
ATE_AIPW = mean( m1(X) - m0(X) + T*(Y - m1(X)) / e(X) - (1-T)*(Y - m0(X)) / (1 - e(X)) )
In this formula, e(X) is the propensity score, m1(X) is the predicted outcome under treatment, m0(X) is the predicted outcome under control, T is the treatment indicator, and Y is the observed outcome. All decimal values represent proportions, so 0.08 equals 8 percentage points.
The estimator has two parts. The first part, regression adjustment (m1(X) - m0(X)), uses the outcome models to predict what would have happened under both treatment and control. The second part consists of IPW correction terms that adjust for any residual error in those predictions. If the outcome models are perfect, the residuals are zero and the correction vanishes. If the propensity is correct, the IPW correction terms produce an unbiased estimate by themselves.
Paul breaks down the mechanics in the tutorial. The IPW correction adds noise that averages out across the sample, but it provides a safety net when the outcome models are wrong. When both models are correctly specified, AIPW reaches the semiparametric efficiency bound asymptotically, meaning it achieves the lowest possible variance among all consistent estimators.
The theoretical background dates to Robins et al., 1994, when Robins, Rotnitzky, and Zhao published the foundational work on this class of estimators. The tutorial cites this work as the basis for the double-robust guarantee.
From Theory to Code: The Tutorial Walkthrough
The tutorial implements AIPW from scratch using scikit-learn for logistic regression and linear regression models. The companion notebook allows reproducing every code block, and the article suggests running misspecification tests on your own data.
The first step is estimating propensity scores. After clipping propensities to [0.01, 0.99] for numerical stability, the estimated propensity range in the synthetic data is 0.114 to 0.675. The mean propensity for the treated group is 0.401, while the mean propensity for the control group is 0.220. This separation reflects the strong selection effect.
The tutorial then fits outcome models for both treatment and control groups. The regression adjustment estimate, using only outcome models, produces an ATE of +0.0847. The full AIPW estimate also comes in at +0.0847, matching the regression adjustment exactly in this case. Both are much closer to the ground truth of +0.0800 than the naïve estimate of +0.2106.
The remaining gap between 0.0847 and 0.0800 reflects the outcome model's own limitations. The tutorial notes that the regression adjustment estimate is much closer to ground truth than the naive estimate, but the gap persists because the outcome models are not perfect.
To quantify uncertainty, the tutorial includes a bootstrap confidence interval function that refits all models on each resample. Using 500 resamples with seed 7, the 95% bootstrap confidence interval for the AIPW ATE is [+0.0744, +0.0952], with a bootstrap standard deviation of 0.0053. This interval comfortably contains the ground truth and excludes the naive estimate, providing strong evidence that AIPW is doing its job.
The Misspecification Tests: Proving Double Robustness
The heart of the tutorial is a pair of deliberate misspecification scenarios designed to empirically prove the double-robust property. These tests transform double robustness from a theoretical property into a concrete number.
Scenario 1 uses a wrong propensity model with a constant value of 0.3 for all users, while keeping the outcome models correct. The tutorial prints the results with specific output labels. The first line is:
"=== Scenario 1: constant propensity (e = 0.3) ==="
Then it reports:
"IPW with wrong propensity: {ate_ipw_wrong:+.4f} (should be wrong)"
The IPW estimate with the wrong propensity comes in at +0.2106, matching the naive estimate exactly. This makes sense because a constant propensity of 0.3 applied uniformly does nothing to correct for selection bias. The next line is:
"AIPW with wrong propensity: {ate_aipw_wrong_ps:+.4f} (should stay ~0.085)"
The AIPW estimate with the wrong propensity is +0.0847, essentially unchanged from the correct estimate. The outcome models rescue the estimator even though the propensity model is completely wrong.
Scenario 2 flips the script. The outcome models are now wrong, set to a constant 0.5 for all users, while the propensity model is correct. The tutorial prints:
"=== Scenario 2: constant outcome models (m1 = m0 = 0.5) ==="
Then:
"Regression with wrong outcome models: {ate_regression_wrong:+.4f} (should be 0.0)"
The regression adjustment with wrong outcome models produces +0.0000, exactly as expected. Constant outcome models cannot distinguish between treatment and control, so the regression adjustment collapses to zero. The next line is:
"IPW with correct propensity: {ate_ipw_correct:+.4f} (should be ~0.085)"
The IPW estimate with the correct propensity is +0.0851, very close to the true effect. The propensity model alone, when correct, produces an unbiased estimate. Finally:
"AIPW with wrong outcome models: {ate_aipw_wrong_out:+.4f} (should stay ~0.085)"
The AIPW estimate with wrong outcome models is +0.0849, again staying close to the true effect. The correct propensity model rescues the estimator even though the outcome models are garbage.
These results demonstrate the key property: AIPW remains consistent if either the propensity model or the outcome model is correctly specified. Both models have to fail simultaneously for AIPW to break. The tutorial notes that AIPW fails when both models are misspecified simultaneously, which is the only scenario where the estimator loses its protection.
Why Models Fail and the Limits of AIPW
The tutorial goes beyond the mechanics to analyze why propensity and outcome models fail in real LLM opt-in analyses. Propensity models fail in three specific ways, and understanding these failure modes is essential for applying AIPW correctly.
First, features downstream of the opt-in decision can leak into the propensity model. If a feature is affected by the treatment itself, including it in the propensity model introduces bias. Second, logistic regression cannot capture nonlinear interactions between features. The true propensity function may be highly nonlinear, and a linear model will miss it. Third, unmeasured confounders are invisible to any model. No matter how many features you include, there may be factors driving both opt-in and task completion that you cannot observe.
Outcome models fail for different reasons. Task completion depends on query complexity, model version, and enterprise workspace custom prompts. These factors interact in complex ways that simple linear models cannot capture. The tutorial notes that you can improve both models and still not know if you've fixed the fundamental problem.
The article also discusses the limits of diagnostic tools. Balance diagnostics can't detect unmeasured confounding, and residual plots can't reveal what covariates left out. These tools can tell you if your models are internally consistent, but they cannot tell you if you've captured all the relevant confounders.
The tutorial emphasizes three identification assumptions that must hold regardless of model quality: Unconfoundedness (strong ignorability), Overlap (positivity), and SUTVA. AIPW relaxes the requirement that models correctly capture these assumptions, but the assumptions themselves must still hold in the data. If unmeasured confounding exists, AIPW carries it forward unchanged.
The tutorial is careful to document the limits of AIPW, even as it demonstrates the estimator's power. Four specific limitations are discussed in detail.
First, simultaneous misspecification breaks the estimator. If both the propensity and outcome models are wrong, AIPW provides no protection. The misspecification tests demonstrate this, and the tutorial warns that in practice you rarely know which model is correct.
Second, extreme propensity scores inflate variance. The tutorial uses an example where a propensity of 0.02 leads to Y / 0.02 = 50 * Y, showing how small propensities amplify noise. Clipping propensities to [0.01, 0.99] provides minimal protection against this variance inflation. The tutorial notes that propensity trimming changes the estimand to ATE over the overlap region, narrowing the effective population.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
Third, finite-sample variance exceeds what asymptotic theory predicts. AIPW's efficiency advantage is a large-sample property. With 500 or 1,000 observations, variance inflation can be substantial. In very small experiments, naïve regression adjustment may give tighter intervals. The bootstrap confidence interval in the tutorial, with its standard deviation of 0.0053, reflects this finite-sample reality.
Fourth, model selection for both components requires judgment. Using flexible models like gradient boosting or random forests for nuisance components requires cross-fitting to avoid overfitting bias. The from-scratch version in the tutorial won't get you through a serious observational study without cross-fitting.
Production-Ready Approaches and the Bottom Line
For production setups, the tutorial recommends cross-fitting and data-adaptive nuisance models. Cross-fitting is the setup behind targeted maximum likelihood estimation (TMLE), a framework developed by Mark van der Laan at UC Berkeley. The targeted learning framework, published in van der Laan & Rose, 2011, corrects for regularization bias and produces valid confidence intervals.
TMLE is described as a production-ready approach that extends the AIPW idea. While AIPW gives you one free mistake, TMLE is designed to handle the case where both models are misspecified by using a targeting step that optimizes the estimate for the specific parameter of interest.
The tutorial also references a real-world implementation. The Lyft engineering team published a detailed account of a doubly robust pipeline for ride-share causal inference (Nassiri & Chu, Lyft Engineering, 2026). This provides a concrete example of how these methods scale to production systems.
The article notes that the author's views are their own, a standard disclaimer given the depth of technical opinion in the tutorial.
The tutorial concludes that AIPW works because it's designed for situations where neither model is verified. The estimator's double-robust property means that as long as one of your two models captures the true relationship, you get a consistent estimate of the treatment effect.
The practical implications for data scientists are clear. The naive estimate of +0.2106 is heavily inflated by selection bias, and would have led to an overoptimistic quarterly business review. The AIPW estimate of +0.0847 is close to the true effect of +0.0800, with a confidence interval that excludes the naive estimate.
The tutorial's misspecification tests are particularly valuable because they show the estimator's behavior under controlled failure conditions. The IPW with wrong propensity matches the naive estimate at +0.2106, demonstrating that propensity weighting alone fails if the propensity model is wrong. The regression with wrong outcome models collapses to +0.0000, showing that regression adjustment alone fails if the outcome model is wrong. But AIPW stays near +0.085 in both scenarios, proving the redundancy works.
The article is published by freeCodeCamp, a non-profit organization whose open source curriculum has helped more than 40,000 people get jobs as developers. This context matters because the tutorial is designed to be accessible to working data scientists, not just academic researchers.
For teams running LLM product experiments without randomization, the takeaway is practical. Users will self-select into treatment, creating selection bias that naïve comparisons cannot handle. Propensity weighting alone fails if the propensity model is wrong, and regression adjustment alone fails if the outcome model is wrong. AIPW provides a safety net that keeps the estimate consistent as long as one model is correct.
The tutorial also offers strategic advice for production. Use cross-fitting with flexible models, consider TMLE for its targeting step, and always run misspecification tests on your own data to understand how your specific models behave under failure conditions.
The article's timing is notable. With LLM-based features rolling out across the industry, the need for rigorous causal inference in product experiments has never been greater. The freeCodeCamp publication makes these techniques accessible to a wide audience of practitioners who may not have formal training in causal inference.
The tutorial's structure is methodical. It starts with the problem of selection bias, introduces the AIPW estimator, walks through the implementation, demonstrates the double-robust property with misspecification tests, and concludes with production recommendations. Every code block is reproducible through the companion notebook, and the article suggests running the tests on your own data.
The bootstrap confidence interval implementation is particularly useful for practitioners. By refitting all models on each resample, the bootstrap captures the full uncertainty in the estimation pipeline, including model fitting error. The 95% confidence interval of [+0.0744, +0.0952] provides a realistic range for the treatment effect, and the fact that it excludes the naive estimate of +0.2106 is a powerful argument for using AIPW.
The tutorial also distinguishes between ATE and ATT, noting that the choice of estimand matters for interpretation. Propensity trimming narrows the effective population, changing the estimand to ATE over the overlap region. This is an important consideration for practitioners who need to communicate results to stakeholders.
The article's references provide a solid foundation for further reading. Robins et al. (1994) established the theoretical background for AIPW, van der Laan & Rose (2011) developed the targeted learning framework, and the Lyft engineering team (2026) demonstrated a production implementation. These references give practitioners a path from the tutorial to deeper understanding.
For data scientists working with LLM product experiments, the tutorial offers a concrete solution to a pervasive problem. Users self-select into features, creating selection bias that undermines simple comparisons. AIPW provides a robust alternative that works even when models are imperfect, as long as one of them captures the true relationship.
The tutorial's emphasis on misspecification testing is a valuable contribution. Rather than assuming models are correct, the article encourages practitioners to deliberately break their models and observe the consequences. This empirical approach to validation is more convincing than theoretical arguments alone.
The article also addresses the psychological challenge of causal inference. A rigorous data scientist will ask about confidence in the propensity model, but the honest answer is often that you don't know. AIPW is designed for exactly this situation, providing protection against the uncertainty that plagues observational studies.
The quarterly business review scenario is a reminder of the stakes. A +8 percentage-point lift that is actually +8.47 percentage points, with a confidence interval from +7.44 to +9.52, is a meaningful difference from the naive +21.06 percentage points. Getting this wrong can lead to misallocation of resources and overoptimistic product decisions.
The tutorial's use of synthetic data is a strength. By knowing the ground truth, the article can demonstrate exactly how well AIPW performs. The ground truth of +0.0800 is recovered within the confidence interval, while the naive estimate is far outside it. This validation gives practitioners confidence in the method.
The article's technical depth is appropriate for its audience. Data scientists running noisy AI product experiments need more than a high-level overview; they need working code and empirical demonstrations. The tutorial delivers both, with a from-scratch implementation that avoids black-box libraries.
The recommendation to use cross-fitting for production is important. The from-scratch version in the tutorial is educational, but serious observational studies require the additional protection that cross-fitting provides. TMLE extends this idea further, correcting for regularization bias and producing valid confidence intervals.
The article's publication on freeCodeCamp News is significant. The platform's reach, combined with its reputation for high-quality technical content, ensures that the tutorial will reach a wide audience of practitioners. The fact that freeCodeCamp's open source curriculum has helped more than 40,000 people get jobs as developers speaks to the platform's impact.
The tutorial's conclusion is practical and grounded. AIPW works because it's designed for situations where neither model is verified. The double-robust property provides a safety net that keeps estimates consistent even when models fail, as long as they don't fail simultaneously.
For teams implementing LLM product experiments, the message is clear. Run your experiments without randomization if you must, but use AIPW to analyze the results. The estimator's redundancy protects against the inevitable imperfections in your models, and the tutorial provides everything you need to implement it.
The article's references to external sources add credibility. Robins et al. (1994) established the theoretical foundation, van der Laan & Rose (2011) developed the targeted learning framework, and the Lyft engineering team (2026) demonstrated a production implementation. These references give practitioners a path from the tutorial to deeper understanding.
The tutorial's use of scikit-learn, scipy, numpy, and pandas is appropriate. These are standard tools in the Python data science ecosystem, and the fact that no causal-inference-specific library is needed makes the tutorial accessible to a wide audience. The required Python version of 3.11 or newer is reasonable for modern data science environments.
The article's treatment of propensity trimming is nuanced. Clipping to [0.01, 0.99] provides minimal protection against variance inflation, but it changes the estimand to ATE over the overlap region. This trade-off is important for practitioners to understand.
The tutorial's bootstrap implementation is thorough. By refitting all models on each resample, it captures the full uncertainty in the estimation pipeline. The use of 500 resamples with seed 7 ensures reproducibility.
The article's discussion of identification assumptions is clear. Unconfoundedness, overlap, and SUTVA must hold in the data, regardless of model quality. AIPW relaxes the requirement that models correctly capture these assumptions, but it cannot fix violations of the assumptions themselves.
The tutorial's analysis of why propensity models fail is particularly valuable. Features downstream of the opt-in decision, nonlinear interactions, and unmeasured confounders are all realistic failure modes in LLM product experiments. Understanding these failure modes helps practitioners build better models.
The article's discussion of outcome model failures is equally insightful. Query complexity, model version, and enterprise workspace custom prompts are all realistic factors that affect task completion. The tutorial's synthetic data captures these complexities, making the demonstration more realistic.
The article's conclusion that AIPW carries unmeasured confounding forward unchanged is an important caveat. The estimator protects against model misspecification, but it cannot fix fundamental data problems. Practitioners must still think carefully about confounding.
The tutorial's recommendation to run misspecification tests on your own data is practical. By deliberately breaking your models, you can understand how your specific implementation behaves under failure conditions. This empirical approach is more convincing than theoretical arguments alone.
The article's publication date of August 11, 2026, places it in a period of rapid LLM adoption. The need for rigorous causal inference in product experiments has never been greater, and the tutorial addresses this need directly.
The tutorial's focus on LLM applications is timely. As more products ship agent-mode features and other AI capabilities, the need for robust causal inference grows. The article provides a practical solution that data scientists can implement today.
The article's structure is effective. It starts with the problem, introduces the solution, demonstrates the solution, and discusses limitations and production considerations. This logical flow makes the tutorial easy to follow.
The tutorial's code examples are clear and reproducible. The companion notebook allows practitioners to run every code block themselves, and the article suggests running the misspecification tests on your own data.
The article's tone is practical and grounded. It avoids hype and focuses on what works. The tutorial's emphasis on empirical validation, through misspecification tests and bootstrap confidence intervals, reflects this practical approach.
The tutorial's treatment of the IPW correction terms is clear. The correction adds noise that averages out across the sample, providing a safety net when outcome models are wrong. This mechanical understanding helps practitioners trust the estimator.
The article's discussion of the semiparametric efficiency bound is appropriately brief. This is a theoretical property that matters asymptotically, but the tutorial's focus is on practical implementation.
The tutorial's recommendation of TMLE for production is sensible. TMLE extends AIPW with a targeting step that corrects for regularization bias, making it more robust in practice. The reference to van der Laan & Rose (2011) provides a path to deeper understanding.
The article's reference to the Lyft engineering team's publication is valuable. It shows that these methods work in production at scale, not just in synthetic examples. The 2026 publication date makes it current.
The tutorial's conclusion is a fitting end to a thorough article. AIPW works because it's designed for situations where neither model is verified. The double-robust property provides a safety net that keeps estimates consistent even when models fail, as long as they don't fail simultaneously.
For data scientists, the takeaway is clear. When running LLM product experiments without randomization, use AIPW. The estimator's redundancy protects against the inevitable imperfections in your models, and the tutorial provides everything you need to implement it.