Part III: Useful, credible estimates, and next steps
A/B Testing, Causal Inference, Causal Time Series Analysis, Data Science, Difference-in-Differences, Directed Acyclic Graphs, Econometrics, Impact Evaluation, Instrumental Variables, Heterogeneous Treatment Effects, Potential Outcomes, Power Analysis, Sample Size Calculation, Python and R Programming, Randomized Experiments, Regression Discontinuity, Treatment Effects
By now you can estimate a causal effect. Estimating it is only part of the job, though. A correct number can still be useless to a decision-maker, or fall apart the moment someone pushes on it. These three chapters cover what comes after the estimate: stress-testing it before a skeptic does, turning it into money a CFO will act on, and knowing which methods to reach for when the standard toolkit runs out.
Chapter 13: Falsification and robustness – stress test your results before someone else does
Key message: An estimate earns trust only by surviving deliberate attempts to break it. Falsification tests build settings where the maintained assumptions imply a zero result, so a fired test is evidence against that specific implication.
- Run the placebo family: placebo-in-time, negative-control outcomes, placebo treatments, placebo units, and the conditional independencies your DAG implies.
- Check balance and overlap (standardized mean differences, love plots, propensity overlap), then re-estimate the headline across a grid of defensible specifications.
- Probe what you can’t see with sensitivity analysis for unobserved confounding.
Why it matters: A free-shipping banner that looks like R$7.86 per user falls to a focal observational contrast of R$1.87 standardized to targeted users, yet remains above the targeted-user ATT truth of R$0.99. Standardizing the contrast defines the target population; it does not identify the ATT. Nor does a targeted-user result answer whether the banner should roll out to the whole base. A failed placebo is evidence against the implication it tests, while a pass offers only bounded reassurance.
Chapter 14: Translating causal estimates into metrics for decision making
Key message: Statistical significance doesn’t make an investment profitable. Before a go/no-go call, you have to discount a raw estimate down to a realistic financial forecast.
- Pick the estimand that matches the decision (ITT vs. LATE), then scale it from your sample to the full population using real unit economics and cost structure.
- Walk the “projection waterfall”: effect decay over time, adoption ramps, cannibalization, and capacity limits that shrink a naive number into an honest one.
- Report ranges instead of point estimates: best- and worst-case bounds from your confidence interval, plus sensitivity to cost and decay assumptions.
Why it matters: This is where a R$2.34-per-user-per-month lift turns into a real answer to “does the R$4M rollout pay off?” Once you account for decay and adoption, a naive R$140M projection can land closer to R$70M. Finance teams and product leaders decide in those numbers, not in p-values.
Chapter 15: What’s next for your causal journey
Key message: The hard problems still come back to the same three questions: what’s the effect, what am I assuming, and can I defend it? Eight advanced topics extend the toolkit you already have instead of replacing it.
- Heavier machinery: structural causal models and do-calculus (front-door identification), causal discovery, causal ML (DML, meta-learners), and Bayesian causal inference.
- Strained settings: sequential and time-varying treatments (g-methods, marginal structural models), interference and spillovers on networked platforms, and policy learning, which turns “what’s the effect?” into “whom should we treat?”
- The throughline: Bayesian methods don’t fix confounding and ML doesn’t fix omitted-variable bias. Identification still carries the load.
Why it matters: Real platforms break SUTVA, run adaptive treatments, and need targeting rules that respect budget and fairness constraints. When you hit one of these, you’ll at least know what the problem is called and where to read more.
Why this pays off
Estimating an effect is the middle of the job, not the end. The people who get trusted with the next decision are usually the ones who put a number on the upside and tried to break their own result before anyone else did.