Part III: Useful, credible estimates, and next steps

Published

December 27, 2025

Modified

November 3, 2026

Keywords

A/B Testing, Causal Inference, Causal Time Series Analysis, Data Science, Difference-in-Differences, Directed Acyclic Graphs, Econometrics, Impact Evaluation, Instrumental Variables, Heterogeneous Treatment Effects, Potential Outcomes, Power Analysis, Sample Size Calculation, Python and R Programming, Randomized Experiments, Regression Discontinuity, Treatment Effects

Get news about the book. Leave your email here.
Found an error or have a suggestion? Submit feedback here.
Follow the book’s page for new content and follow the author.

By now, you can estimate a causal effect, but that’s only part of the job because a correct number can still be useless to a decision-maker or fall apart the moment someone pushes on it. These three chapters cover what comes after the estimate: stress-testing it before a skeptic does, turning it into decision metrics a CFO will act on, and knowing which methods to reach for when the standard toolkit runs out.

Chapter 13: Falsification and robustness – stress test your results before someone else does

Key message: An estimate is trustworthy only if it withstands our deliberate attempts to refute it. Placebo tests examine settings in which the assumptions imply a zero effect. Therefore, finding an effect where there should be none raises doubts about those assumptions. - Run the placebo family: placebo-in-time, negative-control outcomes, placebo treatments, and placebo units; also test the conditional independencies your DAG implies. - Check balance and overlap (standardized mean differences, love plots, propensity overlap), then re-estimate the effect using defensible model specifications. - Probe what you can’t see with sensitivity analysis for unobserved confounding.

Why it matters: A free-shipping banner that looks like R$7.86 per user falls to an estimate of R$1.87 from the interaction model, standardized to targeted users, yet remains above the targeted-user ATT truth of R$0.99. This remains a regression estimate. Standardizing the adjusted difference in mean profit defines the target population; it does not identify the ATT. Nor does a targeted-user result answer whether the banner should roll out to the whole base. Finding an effect in a placebo exercise where no effect is expected provides evidence against the assumption being tested, while finding no effect offers only limited reassurance that the assumption holds.

Chapter 14: Translating causal estimates into metrics for decision making

Key message: Statistical significance doesn’t make an investment profitable. Before a go/no-go call, you have to discount a raw estimate down to a realistic financial forecast.

  • Pick the estimand that matches the decision (ITT vs. LATE), then scale it from your sample to the full population using unit economics and cost structure.
  • Walk the “projection waterfall”: effect decay over time, adoption ramps, cannibalization, and capacity limits that shrink a naive number into a more realistic estimate.
  • Report ranges instead of point estimates: best- and worst-case bounds from your confidence interval, plus sensitivity to cost and decay assumptions.

Why it matters: This is where a R$2.34-per-user-per-month lift turns into a real answer to “does the R$4M rollout pay off?” Once you account for decay, adoption, representativeness, and cannibalization, a naive R$140M projection can land closer to R$70M. Finance and product leadership make decisions based on those numbers, not on p-values.

Chapter 15: What’s next for your causal journey

Key message: The hardest problems still come down to three questions: What is the effect? What assumptions am I making? And can I defend them? The advanced topics that follow build on the toolkit you already have rather than replace it.

  • Advanced methods: structural causal models and do-calculus (front-door identification), causal discovery, causal ML (DML, meta-learners), and Bayesian causal inference.
  • Problems that strain the standard setup: sequential and time-varying treatments (g-methods, marginal structural models); interference and spillovers on networked platforms; policy learning, which turns “what’s the effect?” into “whom should we treat?”; and mechanisms and mediation, which ask how an effect comes about.
  • The throughline: Bayesian methods don’t fix confounding and ML doesn’t fix omitted-variable bias. Identification still carries the load.

Why it matters: In real business settings, treatments can affect people beyond those receiving them (violating SUTVA); treatment decisions can adapt over time; and targeting rules must account for budget and fairness constraints. When you encounter these challenges, you’ll know what they’re called and where to learn more.

Why this pays off

Estimating an effect is not the final step. The people trusted to make the next decision are usually those who also rigorously challenge their own findings before presenting.