13  Falsification and robustness: Stress test your results before someone else does

Published

December 27, 2025

Modified

November 3, 2026

Keywords

A/B Testing, Causal Inference, Causal Time Series Analysis, Data Science, Difference-in-Differences, Directed Acyclic Graphs, Econometrics, Impact Evaluation, Instrumental Variables, Heterogeneous Treatment Effects, Potential Outcomes, Power Analysis, Sample Size Calculation, Python and R Programming, Randomized Experiments, Regression Discontinuity, Treatment Effects

Get news about the book. Leave your email here.
Found an error or have a suggestion? Submit feedback here.
Follow the book’s page for new content and follow the author.

Listen to a podcast version of this chapter, generated by Google’s NotebookLM.1

13.1 Data can debunk, but it cannot confirm

A marketplace launches a free-shipping banner in its app, but it doesn’t show up for every user. A targeting model determines who sees it, selecting users the model expects to buy. A few weeks later, the analytics team compares the groups: users who saw the banner generated far more 30-day profit than those who didn’t. The initial comparison puts the profit difference at R$7.86 per user, and someone pastes it into a slide deck titled “Free shipping works.” The team is ready to keep the current targeting rule.

Then a skeptic asks whether that estimate justifies keeping the current targeting rule, let alone showing the banner to every user: How do you know it’s the banner, not the kind of user the banner gets shown to? The targeting model was built to find high-intent shoppers, and those users were going to spend more anyway. Being likely to buy could explain both why a user saw the banner and why they made a purchase. The R$7.86 difference could reflect the banner’s effect, selection into treatment, or a combination of the two. The initial comparison cannot distinguish these explanations.

A causal estimate becomes more credible when it withstands tests designed to reveal potential problems. These tests can show when the data contradict a design’s assumptions, but even consistently reassuring results cannot prove that an estimate is causal. Without such checks, the estimate provides a weak basis for budget decisions. Part II introduced methods for estimating effects: experiments, regression, instrumental variables, discontinuities, difference-in-differences, and heterogeneous effects. This chapter asks how much causal credibility those estimates deserve. Earlier chapters already used parallel trends, balance, overlap, first stages, and density checks. Here, those diagnostics become parts of one falsification workflow, applied to a single estimate before it is translated into business value (Chapter 14).

Before testing the estimate, we need to be clear about what it represents. An estimand is the population quantity an analysis aims to estimate. Here, our primary estimand is the average treatment effect on the treated (ATT): the banner’s average effect on 30-day profit among the 14,209 users who received it. Because the data are simulated, we know the true ATT is R$0.99. For comparison, the average treatment effect (ATE) across all 30,000 users is R$0.90, while the average treatment effect on the controls (ATC) is R$0.82 for the 15,791 users who did not receive the banner.

These values are close, but they answer different questions. The ATT tells us the effect among today’s targeted users, not what would happen if we also showed the banner to users who currently do not receive it. Rolling out the banner to the entire user base therefore requires evidence about those users too, ideally from a randomized holdout. Table 13.1 compares the starting estimates with the simulated ATT for targeted users:

Table 13.1: Starting estimates and simulated ATT among users shown the free-shipping banner. Both intervals use classical (homoskedastic) standard errors. The chapter states the covariance type wherever it differs.
Quantity Value
Raw treated-versus-untreated comparison R$7.86 [R$7.59, R$8.12]
Initial adjusted model R$2.19 [R$1.93, R$2.45]
Targeted-user ATT truth (simulation) R$0.99

The raw estimate is precise, but precision does not ensure that it is close to the true effect. The chapter first adjusts for prior orders, prior profit, and pre-period sessions, reducing R$7.86 to the initial adjusted estimate of R$2.19. The first set of placebo checks assesses the initial adjusted model. We then add eligibility as an effect modifier and average the adjusted profit differences for eligible and ineligible users in their proportions among currently targeted users (some call it standardization), producing an estimate of R$1.87 from the interaction model. We rerun every applicable diagnostic using the interaction model before using sensitivity analysis to quantify how vulnerable the estimate remains to unobserved confounding. The simulated truth lets us compare full-sample targeted-user estimates with R$0.99, while a separate callout compares eligible-user estimates with their population-specific truth of R$1.50.

Data: All data we’ll use are available in the repository. The free-shipping dataset for this chapter is data/free_shipping_banner.csv. You can download it from the repository or load it directly by passing the raw URL to read.csv() in R or pd.read_csv() in Python.2

Packages: The block below loads the packages needed to run the code. You only need to install these packages once, but you must load them every time you start a new session in R or Python.

# If you haven't already, run this once to install the packages:
# install.packages(c("tidyverse", "fixest", "lmtest", "sandwich", "MatchIt", "WeightIt", "cobalt", "sensemakr", "dagitty"))

# You must run the lines below at the start of every new R session.
library(tidyverse)   # data wrangling and plots
library(fixest)      # fast regression with robust standard errors
library(lmtest)      # coefficient tests used with robust covariance matrices
library(sandwich)    # HC1 and HC3 covariance matrices for lm()
library(MatchIt)     # nearest-neighbour and other matching estimators
library(WeightIt)    # inverse-probability and balancing weights
library(cobalt)      # balance diagnostics and love plots
library(sensemakr)   # sensitivity analysis for unobserved confounding
library(dagitty)     # draw DAGs and read off testable implications
# From the repository root, run this once to install the pinned packages:
# python -m pip install -r requirements.txt

# You must run the lines below at the start of every new Python session.
import pandas as pd
import numpy as np
import networkx as nx                             # DAG d-separation checks
import matplotlib.pyplot as plt                   # balance and overlap plots
import statsmodels.formula.api as smf             # OLS with a formula interface
import statsmodels.api as sm                      # formula-based binomial GLM
import sensemakr                                  # sensitivity analysis (PySensemakr)
from scipy.stats import norm, t as student_t      # Fisher and Student-t intervals
from sklearn.neighbors import NearestNeighbors    # nearest-neighbour matching

13.2 Why causal inference is not just estimation

Suppose the team reruns the banner regression across regions, devices, and months and finds a positive effect every time. These results are not independent confirmations if they rely on the same data and assumptions. Finding an effect where none should exist may tell us more than another positive estimate. However, how much we learn depends on the test’s false-positive rate, its ability to detect a plausible violation, and how closely its results are related to those of other tests. Likewise, finding no effect where none is expected provides meaningful reassurance only if the test was capable of detecting a relevant violation. Even then, it does not prove that the assumptions hold (Eggers et al. 2024).

Karl Popper’s account of conjectures and refutations offers a useful way to think about this (Popper 1963). Rather than trying to accumulate evidence that the banner works, you deliberately look for evidence that challenges that conclusion and report what you find: placebo tests, validation checks, and sensitivity analyses are ways to assess the credibility of a primary estimate (Athey and Imbens 2017). The observed free-shipping banner data do not record the intent that drove targeting. To distinguish the banner’s effect from selection into treatment, design a check that should produce different results depending on which explanation is correct.

If you explain every outcome only after seeing it (lift goes up, the banner worked; lift goes down, the market was soft; lift is flat, it takes time to build habit), you have not tested a prediction. State a prediction that could fail before running the check. Two possible predictions are that the banner has no adjusted association with an outcome it cannot affect, and that relevant pre-treatment covariates are comparable between groups. Explain why the assignment design or adjustment justifies each prediction and why the suspected confounding would violate it.

The most useful check is the one aimed at the design’s weakest assumption, because it has a real chance to overturn the result. A check that gives the same reassuring result whether the assumption holds or not tells you nothing. Decide what result would convince you that the causal claim is wrong, then look for it.

You have used falsification tests throughout the book under other names (pre-trends, placebo tests, balance checks, etc.). This chapter names these checks, explains how they relate, and adds tools for assessing the credibility of one estimate.

Key takeaway. Estimating an effect gives you a number, but interpreting that number as causal requires an identification argument. Falsification checks test predictions that follow from that argument, so report which predictions the data contradicted and how much reassurance the remaining checks provide.

13.3 What falsification is

Direct placebo tests follow the same three-step procedure (Eggers et al. 2024): name the assumption, build a check whose true effect is zero under the stated assumptions, and preserve the features of the main analysis through which the problem could affect the result. The chapter also uses related diagnostics — balance, overlap, first stages, and d-separation — that test observable implications without following that exact zero-effect template.

First, name the assumption each check targets and state what its failure would affect. Some failures bias the estimate. The initial adjusted banner regression, for example, assumes that treated and untreated users are comparable after adjustment; otherwise R$2.19 is not the banner effect. Other failures leave the point estimate unchanged but make its confidence interval unreliable. Users in the same group may share unmodeled behavior, making the confidence interval too narrow. The remedies differ: better standard errors do not correct confounding, and adding a control does not fix dependent errors. State the failure before designing the check so you know what the observed result implies for the estimate or its associated uncertainty.

The second step is to build a check where the true effect is zero by construction. Alter one of three things — the outcome, treatment, or population — so that the quantity being tested must be zero if the stated design and measurement assumptions hold. This is the no-effect-by-construction condition: those assumptions predict no effect, so the check can contradict that prediction. Change the outcome to something the banner cannot plausibly affect, like profit in a product category free shipping does not apply to. Change the treatment to a fake exposure with no causal path to profit. Change the population to users for whom the treatment has no effect, such as shoppers who were shown the banner but cannot redeem free shipping at all. A non-zero result is evidence against the maintained implication, but the source may be confounding, functional form, measurement, or another assumption the check relies on. Eggers, Tuñón, and Dafoe call this the No Average Treatment Effect, or NATE, assumption (Eggers et al. 2024). Here, I use the plainer phrase no effect by construction.

Third, preserve the features of the main analysis that could reproduce the problem the placebo is meant to detect. Reuse the estimator, controls, and uncertainty procedure when that keeps the main and placebo assumptions aligned. Make any change required for a valid placebo, such as conditioning on the real treatment in a placebo-exposure test. The analyses need not be mechanically identical. The placebo is informative when its effect is plausibly zero, it behaves as expected when the main assumptions hold, and it reproduces the relevant bias or inference failure when they do not (Eggers et al. 2024).

Before calling a near-zero placebo estimate reassuring, judge its precision against a pre-specified materiality band. Before inspecting the result, define which departures from zero would be negligible for this particular placebo, in outcome units, and justify those bounds. For the interval rule used here, treat the result as evidence against material departures only when its entire confidence interval lies strictly inside that band; merely including zero is not enough. If no defensible band exists, report the estimate and interval without calling the check resolved. A bound on a placebo association does not automatically bound bias in the main effect. Balance, overlap, effective sample size, robustness ranges, and the permutation calibration check introduced below use their appropriate descriptive summaries rather than this interval rule (Hartman and Hidalgo 2018).

Figure 13.1 separates three questions: where is the estimate, how wide is its interval, and which values does the interval exclude? The horizontal axis shows a placebo difference in 30-day profit, measured in R$ per user. The four rows are invented examples, not estimates from the banner data.

Four hypothetical placebo estimates with 95% confidence intervals, compared with a shaded negligible-effect band from minus 0.50 to plus 0.50 reais per user. A includes zero and extends beyond both boundaries. B includes zero and lies inside the band. C excludes zero but lies inside the band. D lies entirely above the upper boundary.
Figure 13.1: Four hypothetical placebo results illustrate how uncertainty and materiality differ. Dots show estimates; horizontal bars show 95% confidence intervals.3 The shaded band, from R$-0.50 to R$0.50 per user, is an invented negligible-effect range used only for teaching. None of these values comes from the banner analysis. What counts as “precise” or “imprecise,” and “small” or “material,” depends on the outcome, units, inferential goal, and justified bounds for the specific check. These amounts are not general cutoffs. The criteria below explain how to make those judgments.

Rows A and B have the same estimate, but A leaves material departures compatible with the data, whereas B places the entire interval inside the illustrative band. Row C excludes zero while remaining inside the band: there is evidence against the exact-zero prediction, although all values in the interval are negligible under the stated bounds. Row D has a narrow interval entirely beyond the upper bound, providing evidence of a material departure. A narrow interval can therefore sharpen a warning. An interval that touches or overlaps a band boundary leaves materiality unresolved under this rule. These interpretations require a suitable uncertainty procedure and a valid placebo; none establishes that the main effect is unbiased.

NoteHow to assess precision and materiality

Precision describes uncertainty around an estimate. Materiality describes whether the magnitude matters for the question being asked. Use different criteria for these different judgments:

  1. Standard error and confidence-interval width. The standard error estimates the sampling variability of an estimator. For an interval with lower endpoint \(L\) and upper endpoint \(U\), its width is \(U-L\). Compare widths at the same confidence level and in comparable units. Whether that width is sufficiently narrow depends on the distinctions the analysis needs to make (Lakens 2022).

  2. A justified materiality band. Choose bounds using substantive knowledge, relevant previous evidence, and the consequences of the departure being tested. Do not choose them to fit the observed result. For a symmetric band \([-\Delta,\Delta]\), where \(\Delta\) is the largest negligible magnitude, the interval must satisfy \(L>-\Delta\) and \(U<\Delta\). Its half-width, \(h=(U-L)/2\), can be compared with \(\Delta\), but the interval’s location matters too: a narrow interval can lie entirely outside the band. Bounds need not be symmetric when the context justifies different limits in the two directions (Hartman and Hidalgo 2018).

  3. A standardized estimate and interval. Expressing an estimate and its interval in units of a stated reference standard deviation can help readers compare magnitudes across different measurement scales. Identify the reference population and period. For a descriptive rescaling, divide the estimate and both interval endpoints by the same reference standard deviation. This changes the scale; it does not establish that an effect is important or that an interval is precise enough (Lakens 2022).

  4. Coefficient of variation and relative standard error. The coefficient of variation of observations, \(CV=s/\bar{y}\), compares their standard deviation with their mean; it describes dispersion, not directly the precision of a regression coefficient. Survey publications also use “CV of an estimate” for its standard error divided by the estimate. A related percentage measure is the relative standard error, \(100\times SE(\hat{\theta})/|\hat{\theta}|\), where \(\hat{\theta}\) is the estimate. These relative measures become unstable near zero and are undefined at zero. A placebo is often expected to be zero, so assess its absolute uncertainty and its interval relative to justified bounds instead (National Institute of Standards and Technology 2017; Statistics Canada 2020).

  5. Planned power and minimum detectable effect. Power is the probability that a specified test rejects its null hypothesis under a specified alternative. The minimum detectable effect is the effect size a design can detect at chosen significance and power levels. These planning quantities depend on the design, sample size, outcome variability, and dependence between observations. They do not determine which effects matter. If the goal is to place an entire interval inside a negligible-effect band, plan the probability of achieving that containment; power for rejecting zero answers a different question (Abdul Latif Jameel Poverty Action Lab 2021; Lakens 2017).

The chapter’s central rule is: a result inconsistent with a claimed implication raises doubts about it; a result consistent with that implication offers limited reassurance only if the check could detect a relevant violation, never proof. The two directions are not symmetric. When a valid, well-powered placebo that should be zero comes back clearly non-zero, the result contradicts that implication. When its interval includes zero, you have not detected the problem that particular check targets, given its power. Other problems may remain undetected. How much the result reduces concern about the targeted problem depends on the interval’s precision; it does not establish that the problem is absent (Eggers et al. 2024). The checks in one analysis are often correlated, so their agreement should not be treated as independent evidence.

You have used this logic throughout the book: each check asks whether the data fit a particular requirement of the design. Pre-treatment trend checks look for differences in how the groups’ outcomes evolved before treatment, which would cast doubt on parallel trends (Chapter 9, Section 9.5.5). A balance table asks whether pre-treatment characteristics differ enough to threaten comparability (Table 3.1, Section 4.2.5). Live experiments have their own checks: sample-ratio mismatch asks whether group sizes agree with the planned allocation, while Twyman’s Law tells you to investigate results that look unusually different before believing them (Section 4.2.6). The expected pattern depends on the requirement being checked. An instrument must predict treatment: the first-stage check assesses the strength of that relationship after accounting for any included controls. A weak first stage undermines ordinary IV estimates and confidence intervals (Section 7.4). A pre-period outcome placebo instead looks for an association that should be absent: the instrument predicting an outcome measured before treatment. Such an association raises doubts about independence because the instrument and outcome may share prior causes. Neither check establishes the exclusion restriction, which is not directly testable in a just-identified IV model (Section 7.4). The next section applies this reasoning to the banner data.

13.4 The falsification tests you already run

Each subsection applies a diagnostic to the free-shipping banner data and points back to the earlier design where it first appeared. Because the data are simulated, the known data-generating process (DGP) helps explain why the data contradict a particular prediction.

Every regression-based check reports an estimate and interval. The initial checks use a model adjusted for prior orders, prior profit, and pre-period sessions. We call this the initial adjusted model. The robustness section later adds shipping eligibility and its interaction with treatment. We then repeat the earlier falsification checks and related diagnostics using this interaction model and compare the results in one table.

13.4.1 Placebo in time: a “fake effect” before the banner existed

Event-study leads are placebo estimates. Under no anticipation and the design’s pre-period trend restrictions, their population values should be zero. A nonzero lead shows that the design or specification conflicts with those restrictions. Possible explanations include differential pre-trends, anticipation, time-varying confounding, functional-form misspecification, contamination from heterogeneous post-treatment effects in a conventional staggered-TWFE event study, and Ashenfelter’s dip when treatment follows a temporary pre-treatment outcome decline. The lead coefficients alone do not identify which explanation applies (Section 9.5.3.2, Section 9.5.5).

The same logic appears in two earlier checks: a fake intervention date in time-series analysis (Section 11.2.4, Section 11.3.5.1) and an A/A test before an experiment (Section 4.2.5). Roth’s point is narrower than “clean leads prove parallel trends”: pre-period coefficients test a prediction related to parallel trends, often with low power. Proceeding only when the test finds no statistically significant departure of those coefficients from zero distorts later inference (Roth et al. 2023). The TWFE chapter uses a joint F/Wald test of all pre-treatment coefficients; Section 9.5.3.2 provides the R and Python implementation.

The banner launched on a specific day, so it cannot have changed anything that happened before that day. For this placebo in time, use profit_pre as the outcome and keep prior_profit among the initial adjusted model’s controls.4 Both measure pre-treatment profit, but they are distinct variables: in the simulation, profit_pre depends on prior_profit, prior orders, pre-period sessions, unobserved shopping intent, and separate random noise. The check therefore asks whether treated and untreated users differ in profit_pre after adjustment for prior_profit, prior orders, and pre-period sessions. If adjustment removes pre-treatment group differences, the adjusted treated-control gap on profit_pre should be about zero because the banner had not yet appeared.

The initial adjusted model estimates a difference in mean pre-period profit of R$1.59 [R$1.32, R$1.86]. Treated users were already more profitable than controls before the banner appeared. That pattern is evidence against the claim that the initial adjustment makes the groups comparable, and it is consistent with the targeting model selecting users who were already more profitable. Figure 13.2 also shows the placebo-in-time result using the interaction model introduced later; the estimated difference remains R$1.60 [R$1.33, R$1.88].

Figure 13.2: Placebo in time: the horizontal axis shows the adjusted treated–control gap in mean pre-period profit (R$), with the initial adjusted model and interaction model on the vertical axis. Both gaps are far from zero, although the banner cannot have caused either one.
library(tidyverse)
library(lmtest)
library(sandwich)

df <- read.csv("data/free_shipping_banner.csv")

report_regression <- function(label, model, term = "treated") {
  test <- coeftest(model, vcov. = vcovHC(model, type = "HC1"))
  estimate <- test[term, "Estimate"]
  se <- test[term, "Std. Error"]
  critical <- qt(0.975, df = df.residual(model))
  cat(sprintf(
    "%s: estimate = %.4f; HC1 SE = %.4f; 95%% Student-t CI = [%.4f, %.4f]\n",
    label, estimate, se, estimate - critical * se, estimate + critical * se
  ))
}

# Same adjusted spec as the headline, but the OUTCOME is the
# pre-treatment period the banner cannot have touched.
placebo_time <- lm(
  profit_pre ~ treated + prior_orders + prior_profit + sessions_pre,
  data = df
)

# The 'treated' coefficient should be ~0 if the design is clean.
report_regression("initial adjusted placebo in time", placebo_time)
#> initial adjusted placebo in time: estimate = 1.5918; HC1 SE = 0.1366; 95% Student-t CI = [1.3239, 1.8596]
import pandas as pd
import statsmodels.formula.api as smf
from scipy.stats import t as student_t

df = pd.read_csv("data/free_shipping_banner.csv")

def report_regression(label, model, term="treated"):
    estimate = model.params[term]
    se = model.bse[term]
    critical = student_t.ppf(0.975, model.df_resid)
    print(
        f"{label}: estimate = {estimate:.4f}; HC1 SE = {se:.4f}; "
        f"95% Student-t CI = [{estimate - critical * se:.4f}, "
        f"{estimate + critical * se:.4f}]"
    )

# Same adjusted spec as the headline, but the OUTCOME is the
# pre-treatment period the banner cannot have touched.
placebo_time = smf.ols(
    "profit_pre ~ treated + prior_orders + prior_profit + sessions_pre",
    data=df,
).fit(cov_type="HC1")

# The 'treated' coefficient should be ~0 if the design is clean.
report_regression("initial adjusted placebo in time", placebo_time)
# initial adjusted placebo in time: estimate = 1.5918; HC1 SE = 0.1366; 95% Student-t CI = [1.3239, 1.8596]

Interpret the nonzero placebo. A non-zero placebo-in-time estimate does not show that the banner has no effect. It shows that treated and control users already differed before treatment, so the post-treatment gap begins from an unequal baseline. Conditioning on the pre-treatment behavior used by the targeting model is the same adjustment that shrank the raw estimate. Because the placebo remains nonzero after that adjustment, the controls did not fully close the pre-treatment gap.

13.4.2 Negative-control outcomes: an outcome the treatment cannot affect

You have run this check before as covariate balance in regression discontinuity. There, you applied the RDD to a variable the cutoff should not affect (age, when the threshold concerns something else) and treated a jump as a warning about the comparison (Section 8.2.1, Figure 8.4). RDD adds another falsification in the density-at-the-cutoff test, which looks for bunching that suggests units manipulated their position relative to the threshold (Figure 8.5). Time-series analysis uses the same logic: a control or auxiliary series used for prediction must be unaffected by the intervention, or the comparison becomes circular (Section 11.3.1). Each check tests a prediction that follows from the design’s assumptions.

The logic generalizes to the idea of a negative-control outcome (also called a null outcome) (Lipsitch et al. 2010). Choose an outcome that shares the confounding path with the outcome of interest but not the causal path. Here, shopping intent is the shared cause: users who are more likely to buy are more likely to see the banner and also spend more across product categories. So use an outcome that shopping intent affects but the banner cannot affect: noneligible_category_profit, profit from product categories excluded from the free-shipping offer. In this simulation, the DGP gives the banner no causal path to this outcome. If selection were absent, the adjusted association would be zero.

In an applied analysis, this is a valid negative-control outcome only if the offer cannot change noneligible-category profit directly or indirectly. Basket substitution, complementary purchases, a shared checkout threshold, or reallocating a fixed shopping budget can create a causal path from the banner to this outcome. Document why those paths are absent; otherwise a nonzero result may reflect a real treatment pathway rather than confounding.

The two panels in Figure 13.3 show why noneligible-category profit can reveal shared causes, and how a treatment pathway would invalidate that interpretation.

Two vertically stacked causal-path diagrams. In Panel A, unobserved shopping intent causes the targeting score, banner exposure, and both category-profit outcomes. The targeting score causes banner exposure, which can affect eligible-category profit but has no directed path to noneligible-category profit. In Panel B, a hypothetical route from banner exposure through cross-category purchases to noneligible-category profit makes that outcome unsuitable as a negative control. Other variables are omitted.
Figure 13.3: A negative-control outcome must share relevant causes with the outcome of interest while remaining unaffected by treatment. In Panel A, shopping intent affects the targeting score, banner exposure, and both category-profit outcomes, but the banner has no causal path to noneligible-category profit. In Panel B, a hypothetical change in cross-category purchases creates such a path. That outcome would then be unsuitable as a negative control for this argument. The diagrams show selected pathways, not the complete simulation process.

The negative-control association from the initial adjusted model is not zero: R$1.11 [R$0.95, R$1.28]. That is evidence against the claim that the core controls make treated and untreated users comparable. Residual selection is a plausible explanation, but the diagnostic alone does not identify its source. As Figure 13.4 shows, the negative-control association using the interaction model is nearly unchanged at R$1.13 [R$0.96, R$1.29].

Figure 13.4: Negative-control outcome: the horizontal axis shows the adjusted association between the banner and noneligible-category profit (R$), with the initial adjusted model and interaction model on the vertical axis. In both models, the banner remains associated with profit it cannot influence, providing evidence against the claim that adjustment makes treated and untreated users comparable.
library(tidyverse)
library(lmtest)
library(sandwich)

df <- read.csv("data/free_shipping_banner.csv")

report_regression <- function(label, model, term = "treated") {
  test <- coeftest(model, vcov. = vcovHC(model, type = "HC1"))
  estimate <- test[term, "Estimate"]
  se <- test[term, "Std. Error"]
  critical <- qt(0.975, df = df.residual(model))
  cat(sprintf(
    "%s: estimate = %.4f; HC1 SE = %.4f; 95%% Student-t CI = [%.4f, %.4f]\n",
    label, estimate, se, estimate - critical * se, estimate + critical * se
  ))
}

# Same adjusted spec, but the OUTCOME is profit from categories
# the free-shipping offer does not apply to. The banner cannot move it.
negctrl <- lm(
  noneligible_category_profit ~ treated + prior_orders + prior_profit + sessions_pre,
  data = df
)

# The 'treated' coefficient should be 0 if selection is absent.
report_regression("initial adjusted negative control", negctrl)
#> initial adjusted negative control: estimate = 1.1148; HC1 SE = 0.0838; 95% Student-t CI = [0.9505, 1.2791]
import pandas as pd
import statsmodels.formula.api as smf
from scipy.stats import t as student_t

df = pd.read_csv("data/free_shipping_banner.csv")

def report_regression(label, model, term="treated"):
    estimate = model.params[term]
    se = model.bse[term]
    critical = student_t.ppf(0.975, model.df_resid)
    print(
        f"{label}: estimate = {estimate:.4f}; HC1 SE = {se:.4f}; "
        f"95% Student-t CI = [{estimate - critical * se:.4f}, "
        f"{estimate + critical * se:.4f}]"
    )

# Same adjusted spec, but the OUTCOME is profit from categories
# the free-shipping offer does not apply to. The banner cannot move it.
negctrl = smf.ols(
    "noneligible_category_profit ~ treated + prior_orders + prior_profit + sessions_pre",
    data=df,
).fit(cov_type="HC1")

# The 'treated' coefficient should be 0 if selection is absent.
report_regression("initial adjusted negative control", negctrl)
# initial adjusted negative control: estimate = 1.1148; HC1 SE = 0.0838; 95% Student-t CI = [0.9505, 1.2791]

Balance tables also assess comparability by comparing pre-treatment covariates; even random assignment can produce chance imbalances in a particular sample. The balance-and-overlap section applies this comparison to the banner data (Section 13.4.5).

A real effect tends to be specific: concentrated where the mechanism acts and absent where it does not. In this simulation, free shipping changes profit in eligible categories and leaves noneligible-category profit unchanged; an association with an outcome the banner cannot affect is consistent with a shared cause or another violation of the model’s assumptions (Lipsitch et al. 2010). We return to this specificity pattern later in the chapter when we read the family of checks together.

Key takeaway. A negative-control outcome shares plausible causes with the outcome of interest but has no causal path from the treatment. Using the initial adjusted model, both the placebo in time (R$1.59 [R$1.32, R$1.86]) and the negative-control association (R$1.11 [R$0.95, R$1.28]) are inconsistent with the prediction that these adjusted associations should be zero. They are related diagnostics from one observational design, not independent confirmations.

13.4.3 Placebo treatments: an effect from an assignment that did nothing

You have done a version of this in the heterogeneous-effects chapter, where the forest is refitted after shuffling treatment labels (Section 12.7). A deterministic null permutation gives an estimate and interval after one fixed shuffle of the treatment labels. Permutation calibration instead uses repeated random permutations to check how often the regression test rejects the null when the true effect is zero. A negative-control exposure should preserve a cause of real assignment while having no causal path to the outcome (Lipsitch et al. 2010).

The code constructs one deterministic null permutation and one score-preserving fake exposure. The deterministic null permutation shuffles the treatment labels across the full sample by a fixed rule rather than a random draw, so R and Python assign identical placebo labels and report the same number. The score-preserving fake exposure instead takes the untreated users the targeting model rated highest and labels them as if they had been exposed. It labels 7,479 of them, the count that gives the fake exposure the same share of the untreated pool that real exposure has of the whole base: floor(14,209 / 30,000 × 15,791). They never saw the banner, but the fake label retains score-based selection among untreated users. It does not reproduce the real assignment process or establish the same degree of confounding. Both regressions keep the real treated variable in the model to test whether the fake label still predicts profit after actual exposure is held fixed. Otherwise, the fake label could stand in for actual exposure.

library(tidyverse)
library(lmtest)
library(sandwich)

df <- read.csv("data/free_shipping_banner.csv")

report_regression <- function(label, model, term) {
  test <- coeftest(model, vcov. = vcovHC(model, type = "HC1"))
  estimate <- test[term, "Estimate"]
  se <- test[term, "Std. Error"]
  critical <- qt(0.975, df = df.residual(model))
  cat(sprintf(
    "%s: estimate = %.4f; HC1 SE = %.4f; 95%% Student-t CI = [%.4f, %.4f]\n",
    label, estimate, se, estimate - critical * se, estimate + critical * se
  ))
}

# fake_exposure_n gives the fake label the same share of the untreated pool
# that real exposure has of the whole base: floor(14209 / 30000 * 15791).
cfg <- list(
  lcg_multiplier = 104729,
  lcg_offset = 12345,
  fake_exposure_n = 7479
)
n <- nrow(df)

# A deterministic null permutation, identical in R and Python.
i0 <- 0:(n - 1)
idx <- ((cfg$lcg_multiplier * i0 + cfg$lcg_offset) %% n) + 1
stopifnot(length(unique(idx)) == n)
df$placebo_null <- df$treated[idx]

# A score-preserving fake exposure among users who were not treated.
untreated <- which(df$treated == 0)
ordered <- untreated[order(-df$predicted_purchase_score[untreated],
                           df$user_id[untreated])]
df$fake_score_exposure <- 0L
df$fake_score_exposure[head(ordered, cfg$fake_exposure_n)] <- 1L
stopifnot(sum(df$fake_score_exposure) == cfg$fake_exposure_n)

null_fit <- lm(
  profit_30d ~ placebo_null + treated +
    prior_orders + prior_profit + sessions_pre,
  data = df
)
score_fit <- lm(
  profit_30d ~ fake_score_exposure + treated +
    prior_orders + prior_profit + sessions_pre,
  data = df
)

report_regression("deterministic null permutation", null_fit, "placebo_null")
report_regression("score-preserving fake exposure", score_fit, "fake_score_exposure")
#> deterministic null permutation: estimate = -0.0626; HC1 SE = 0.1197; 95% Student-t CI = [-0.2972, 0.1720]
#> score-preserving fake exposure: estimate = 1.5868; HC1 SE = 0.1838; 95% Student-t CI = [1.2266, 1.9471]
import pandas as pd
import numpy as np
import statsmodels.formula.api as smf
from scipy.stats import t as student_t

df = pd.read_csv("data/free_shipping_banner.csv")

def report_regression(label, model, term):
    estimate = model.params[term]
    se = model.bse[term]
    critical = student_t.ppf(0.975, model.df_resid)
    print(
        f"{label}: estimate = {estimate:.4f}; HC1 SE = {se:.4f}; "
        f"95% Student-t CI = [{estimate - critical * se:.4f}, "
        f"{estimate + critical * se:.4f}]"
    )

LCG_MULTIPLIER = 104729
LCG_OFFSET = 12345
# FAKE_EXPOSURE_N gives the fake label the same share of the untreated pool
# that real exposure has of the whole base: floor(14209 / 30000 * 15791).
FAKE_EXPOSURE_N = 7479
n = len(df)

# A deterministic null permutation, identical in R and Python.
i0 = np.arange(n, dtype=np.int64)
idx = (LCG_MULTIPLIER * i0 + LCG_OFFSET) % n
assert np.unique(idx).size == n
df["placebo_null"] = df["treated"].to_numpy()[idx]

# Preserve the score-based selection pattern among untreated users.
ordered = (
    df[df["treated"] == 0]
    .sort_values(["predicted_purchase_score", "user_id"],
                 ascending=[False, True], kind="mergesort")
)
df["fake_score_exposure"] = 0
df.loc[ordered.head(FAKE_EXPOSURE_N).index, "fake_score_exposure"] = 1
assert df["fake_score_exposure"].sum() == FAKE_EXPOSURE_N

null_fit = smf.ols(
    "profit_30d ~ placebo_null + treated "
    "+ prior_orders + prior_profit + sessions_pre",
    data=df,
).fit(cov_type="HC1")
score_fit = smf.ols(
    "profit_30d ~ fake_score_exposure + treated "
    "+ prior_orders + prior_profit + sessions_pre",
    data=df,
).fit(cov_type="HC1")

report_regression("deterministic null permutation", null_fit, "placebo_null")
report_regression("score-preserving fake exposure", score_fit, "fake_score_exposure")
# deterministic null permutation: estimate = -0.0626; HC1 SE = 0.1197; 95% Student-t CI = [-0.2972, 0.1720]
# score-preserving fake exposure: estimate = 1.5868; HC1 SE = 0.1838; 95% Student-t CI = [1.2266, 1.9471]

The two checks answer different questions. The deterministic null permutation returns R$-0.06 [R$-0.30, R$0.17] for one fixed shuffle of the treatment labels. Because no materiality band was pre-specified, report the estimate and interval without concluding that a meaningful false effect has been ruled out. One permutation cannot establish the regression test’s false-positive rate. The permutation calibration check among ineligible users below estimates that rate using repeated random permutations. The score-preserving fake exposure returns R$1.59 [R$1.23, R$1.95]. This shows that score-based selection still predicts profit after holding the real treatment and core controls fixed. With the treatment-by-eligibility interaction, the estimates become R$-0.05 [R$-0.28, R$0.18] and R$1.38 [R$1.02, R$1.73]. The interaction model reduces the score-preserving fake exposure’s association with profit but does not remove it.

Read these intervals using the distinction illustrated in Figure 13.1: including zero and excluding material departures are different findings. The figure’s invented band must not be applied retrospectively to these banner results.

13.4.4 Placebo units: groups the banner could not have helped

Changing the comparison group in staggered difference-in-differences is a robustness check (Chapter 10, Section 10.5.1). A placebo-population check instead applies the estimator to units for which the treatment has no effect under the maintained assumptions. In synthetic control, placebo-in-space estimates from untreated units provide a reference against which to assess the treated unit’s estimated effect, taking pre-treatment fit into account.

The simulation provides a natural set of placebo units. Its DGP sets the true banner effect to zero for users with shipping_eligible == 0. Among the 4,842 treated-ineligible users, the simulated truth is R$0.00. We compare these users with untreated users who are also ineligible. A design that removes selection should therefore return roughly zero for the adjusted profit difference between these two groups.

This guarantee comes from the simulation’s DGP, not from the shipping_eligible == 0 label in real data. Seeing an unusable offer can still change clicks, cart abandonment, or later purchases, and eligibility can be mismeasured. Verify that no behavioral or measurement path remains before treating ineligible users as placebo units; otherwise the test mixes a real exposure effect with selection bias.

Under the actual targeting assignment, the initial adjusted estimate on ineligible users is R$1.12 [R$0.70, R$1.54], not zero. Selection still operates among the ineligible: the targeting model favored higher-intent users whether or not they could redeem the offer. Writing the interaction model without centering gives an adjusted profit difference among ineligible users of R$1.25 [R$0.86, R$1.64]. Both are evidence against treating the observational comparison as a causal effect for this placebo population.

A separate check, permutation calibration, uses repeated random permutations to estimate how often the regression test produces a false positive under random labels. Within the ineligible group, shuffle treatment labels while keeping each user’s profit and other characteristics unchanged, re-estimate the regression, and count how often the test rejects the null. Because the true effect there is zero, the rejection rate should be close to the chosen significance level, or one time in twenty at the 5% threshold. A much higher rate means that the standard errors or model specification lead the test to report an effect too often when none exists.

library(tidyverse)
library(lmtest)
library(sandwich)

df <- read.csv("data/free_shipping_banner.csv")

# Placebo population: users who cannot redeem free shipping.
# Their true banner effect is R$0.00 by construction.
ineligible <- df |> filter(shipping_eligible == 0)

# 1) The observational estimator on the placebo units.
#    Need NOT be zero: selection still operates among the ineligible.
obs_placebo <- lm(
  profit_30d ~ treated + prior_orders + prior_profit + sessions_pre,
  data = ineligible
)
test <- coeftest(obs_placebo, vcov. = vcovHC(obs_placebo, type = "HC1"))
estimate <- test["treated", "Estimate"]
se <- test["treated", "Std. Error"]
critical <- qt(0.975, df = df.residual(obs_placebo))
cat(sprintf(
  "adjusted profit difference among ineligible users (initial adjusted model): estimate = %.4f; HC1 SE = %.4f; 95%% Student-t CI = [%.4f, %.4f]\n",
  estimate, se, estimate - critical * se, estimate + critical * se
))
#> adjusted profit difference among ineligible users (initial adjusted model): estimate = 1.1231; HC1 SE = 0.2146; 95% Student-t CI = [0.7025, 1.5437]

# 2) Permutation calibration: fake the assignment within a group
#    that cannot benefit, and see how often we reject the null.
set.seed(42)
reject <- replicate(1000, {
  ineligible$fake <- sample(ineligible$treated)
  m <- lm(profit_30d ~ fake + prior_orders + prior_profit + sessions_pre,
          data = ineligible)
  test <- coeftest(m, vcov. = vcovHC(m, type = "HC1"))
  abs(test["fake", "t value"]) > qt(0.975, df = m$df.residual)
})
mean(reject)   # 0.046: near the 0.05 we want; R's random stream
import pandas as pd
import numpy as np
import statsmodels.formula.api as smf
from scipy.stats import t as student_t

df = pd.read_csv("data/free_shipping_banner.csv")

# Placebo population: users who cannot redeem free shipping.
# Their true banner effect is R$0.00 by construction.
ineligible = df[df["shipping_eligible"] == 0].copy()

# 1) The observational estimator on the placebo units.
#    Need NOT be zero: selection still operates among the ineligible.
obs_placebo = smf.ols(
    "profit_30d ~ treated + prior_orders + prior_profit + sessions_pre",
    data=ineligible,
).fit(cov_type="HC1")
estimate = obs_placebo.params["treated"]
se = obs_placebo.bse["treated"]
critical = student_t.ppf(0.975, obs_placebo.df_resid)
print(
    "adjusted profit difference among ineligible users (initial adjusted model): "
    f"estimate = {estimate:.4f}; HC1 SE = {se:.4f}; "
    f"95% Student-t CI = [{estimate - critical * se:.4f}, "
    f"{estimate + critical * se:.4f}]"
)
# adjusted profit difference among ineligible users (initial adjusted model): estimate = 1.1231; HC1 SE = 0.2146; 95% Student-t CI = [0.7025, 1.5437]

# 2) Permutation calibration: fake the assignment within a group
#    that cannot benefit, and see how often we reject the null.
rng = np.random.default_rng(42)
reject = []
for _ in range(1000):
    ineligible["fake"] = rng.permutation(ineligible["treated"].values)
    m = smf.ols(
        "profit_30d ~ fake + prior_orders + prior_profit + sessions_pre",
        data=ineligible,
    ).fit(cov_type="HC1")
    reject.append(abs(m.tvalues["fake"]) > student_t.ppf(0.975, m.df_resid))
print(np.mean(reject))   # 0.056: a different random stream, also near 0.05

Key takeaway. In the simulation, shipping-ineligible users have a true effect of R$0.00, yet the estimate from the initial adjusted model is R$1.12 [R$0.70, R$1.54]. In the permutation calibration check, the regression test rejects the null in 4.6%-5.6% of repetitions, close to the chosen 5% significance level. Shuffling treatment labels removes the actual score-based assignment mechanism, so the check estimates the regression test’s false-positive rate under random labels. It does not validate the observational coefficient’s bias or false-positive behavior under the real targeting process. The nonzero profit difference among ineligible users is the relevant warning about that process.

13.4.5 Balance and overlap: are we comparing comparable users?

Balance and overlap answer different questions. A balance table compares pre-treatment covariates between treated and untreated groups (Table 3.1, Section 3.4.3). Under a specified assignment design, those covariates have a placebo-like zero implication. Overlap or positivity asks whether each type of user in the target population appears in both treatment groups so that a comparison is possible (Chapter 6). It is a support condition, not a placebo test.

The banner cannot change a pre-treatment covariate, but targeting can still make the groups differ on it. With completely random assignment, the group means are equal in expectation; here, matching instead aims to balance the measured covariates. Standardized mean differences (SMDs) put covariate gaps on a common scale; a love plot shows those gaps before and after adjustment. We use the common \(|\text{SMD}| < 0.10\) threshold as a descriptive screen, not as one hypothesis test per covariate. The propensity score estimates a user’s probability of seeing the banner; the targeting score (the targeting model’s predicted-purchase score) predicts how likely the user is to buy. Propensity-score overlap compares estimated treatment probabilities across groups and exposes regions where one group has no credible counterpart. Inverse-probability weighting (IPW) turns those probabilities into weights for a target-population comparison (Hirano and Imbens 2001). Propensity-score matching works in two stages: it first estimates each user’s probability of treatment from observed pre-treatment covariates, then matches treated and untreated users with similar scores.5

Figure 13.5: Overlap and balance: estimated treatment propensities differ across targeted and untargeted users (left); 1:1 matching with replacement reduces every ATT-standardized SMD below 0.10 (right).

In the left panel of Figure 13.5, the horizontal axis shows estimated probability of targeting and the vertical axis shows density for targeted and untargeted users. Their distributions overlap but do not coincide. The right panel compares balance before and after matching, with absolute ATT-standardized SMDs on the horizontal axis and the propensity score and covariates listed vertically. Matching uses all 14,209 treated users but only 6,017 distinct control users, because matching with replacement lets one control stand in for several treated users. With those reuse weights, the matched controls have an effective sample size (ESS) of 2,727 equally weighted controls. The largest absolute ATT-standardized SMD is 0.026, well below the 0.10 diagnostic threshold. The observed covariates are well balanced, but unobserved shopping intent may still differ between the groups.

library(tidyverse)
library(cobalt)
library(MatchIt)

df <- read.csv("data/free_shipping_banner.csv")

# Unpenalized propensity model and 1:1 ATT matching with replacement.
propensity <- treated ~ prior_orders + prior_profit + sessions_pre +
  predicted_purchase_score + tenure_days + shipping_eligible
ps_model <- glm(propensity, data = df, family = binomial())
matched <- matchit(
  propensity,
  data = df,
  method = "nearest",
  distance = predict(ps_model, type = "response"),
  estimand = "ATT",
  replace = TRUE,
  ratio = 1,
  normalize = FALSE
)

# Overlap: the estimated propensity distribution in each group (left panel).
df$propensity_score <- predict(ps_model, type = "response")
ggplot(df, aes(x = propensity_score, fill = factor(treated))) +
  geom_density(alpha = 0.5, color = NA) +
  labs(x = "Estimated probability of targeting", y = "Density",
       fill = "Targeted")

# SMDs use the treated group's SD because the target is the ATT (right panel).
bal.tab(matched, un = TRUE, binary = "std", s.d.denom = "treated")
love.plot(matched, abs = TRUE, binary = "std", s.d.denom = "treated",
          thresholds = c(m = 0.10))

matched_df <- match_data(matched, data = df)
control_w <- matched_df$weights[matched_df$treated == 0]
sum(matched_df$treated == 0)            # 6017 unique controls
sum(control_w)^2 / sum(control_w^2)     # control ESS = 2727.325
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
import statsmodels.formula.api as smf
from sklearn.neighbors import NearestNeighbors

df = pd.read_csv("data/free_shipping_banner.csv")
PROPENSITY = (
    "treated ~ prior_orders + prior_profit + sessions_pre + "
    "predicted_purchase_score + tenure_days + shipping_eligible"
)
VARS = ["prior_orders", "prior_profit", "sessions_pre",
        "predicted_purchase_score", "tenure_days", "shipping_eligible"]

# Formula-based, unpenalized binomial GLM, matching the R propensity model.
ps_model = smf.glm(PROPENSITY, data=df,
                   family=sm.families.Binomial()).fit()
df["ps"] = ps_model.predict(df)
treated, controls = df[df.treated == 1].copy(), df[df.treated == 0].copy()
idx = (NearestNeighbors(n_neighbors=1)
       .fit(controls[["ps"]])
       .kneighbors(treated[["ps"]], return_distance=False)
       .ravel())

# Keep each unique control once and weight it by its raw reuse count.
control_rows, reuse = np.unique(controls.iloc[idx].index, return_counts=True)
matched_controls = controls.loc[control_rows].copy()
matched_controls["match_weight"] = reuse
treated["match_weight"] = 1
assert len(matched_controls) == 6017
ess = reuse.sum() ** 2 / np.square(reuse).sum()  # 2727.325

# Overlap: the estimated propensity distribution in each group (left panel).
fig, ax = plt.subplots()
for label, group in df.groupby("treated"):
    group["ps"].plot.density(ax=ax, label=f"treated={label}")
ax.set(xlabel="Estimated probability of targeting", ylabel="Density")
ax.legend()

before_smd, after_smd = {}, {}
for col in VARS:
    treated_sd = treated[col].std(ddof=1)
    before_smd[col] = (treated[col].mean() - controls[col].mean()) / treated_sd
    matched_control_mean = np.average(
        matched_controls[col], weights=matched_controls["match_weight"]
    )
    after_smd[col] = (treated[col].mean() - matched_control_mean) / treated_sd

balance = pd.DataFrame({
    "Unmatched": before_smd,
    "Matched (ATT)": after_smd,
})
print(balance)
ax = balance.abs().plot.barh()
ax.axvline(0.10, color="black", linestyle="--")
ax.set(xlabel="Absolute standardized mean difference", ylabel="")
max(balance["Matched (ATT)"].abs())             # 0.026281

First, good observed balance is necessary but not sufficient. Unobserved shopping intent can still differ between treated and untreated users even when every absolute SMD is below the threshold. Second, an SMD table is a threshold diagnostic, not a batch of null-hypothesis tests. If you need a formal claim that differences are negligibly small, pre-specify equivalence margins and treat those tests as a correlated family (Hartman and Hidalgo 2018). Do not retrofit one p-value per covariate onto a love plot.

Key takeaway. Matching brings the largest observed SMD down to 0.026, using 6,017 unique controls with control ESS about 2,727. Those are strong descriptive balance diagnostics. They do not address the unobserved intent that drives both targeting and profit.

13.4.6 d-separation: testing a DAG’s observable implications

The causal-assumptions chapter left this test open (Section 6.4.2). d-separation is the graphical rule for deciding whether a conditioning set blocks every path connecting two variables in a DAG. When all paths are blocked, the graph implies a conditional independence that can sometimes be checked in observed data. If the data contradict that prediction, they provide evidence against the specific independence claim without establishing which part of the causal explanation is wrong.

Here the primary test asks whether the targeting score and 30-day profit have zero residual correlation. The candidate graph described below implies that these two variables are independent after conditioning on exposure, prior orders, prior profit, and pre-period sessions. We regress each variable on that set and correlate the residuals, or what remains after adjustment; this assumes the relationships are linear. The test checks the graph’s implication and the linear adjustment together: a nonzero residual correlation tells you that at least one is wrong, not which one. A missing arrow and a curved relationship that a straight-line fit misses can both produce that result.

The candidate graph says that pre-treatment behavior drives both the score and profit and directly affects exposure; the score also drives exposure, and exposure drives profit. It omits shipping eligibility and unobserved shopping intent. The graph implies three marginal independencies among prior_orders, prior_profit, and sessions_pre, plus score-profit independence after conditioning on exposure and the core controls. We report all four, using the adjusted score-profit correlation as the primary test because it is closest to the identification question. Figure 13.6 compares the candidate graph with a corrected graph that agrees with the process used to simulate the data.

Figure 13.6: Candidate and corrected DAGs. The corrected graph adds observed eligibility and unobserved shopping intent as common causes and retains the arrows from score to exposure and from exposure to profit, with no direct arrow from score to profit. The arrows show which variables affect which others without specifying the form of those effects (Nilsson et al. 2021). In this simulation, the banner effect depends on shipping eligibility, even though the interaction is not drawn as a separate node. Both panels collapse prior_orders, prior_profit, and sessions_pre into one pre-treatment behavior node so the figure stays readable; the code keeps the three variables separate, which is what makes the three marginal independencies testable.
library(tidyverse)
library(dagitty)

df <- read.csv("data/free_shipping_banner.csv")

report_correlation <- function(label, x, y) {
  keep <- complete.cases(x, y)
  r <- cor(x[keep], y[keep], method = "pearson")
  delta <- qnorm(0.975) / sqrt(sum(keep) - 3)
  lower <- tanh(atanh(r) - delta)
  upper <- tanh(atanh(r) + delta)
  cat(sprintf(
    "%s: Pearson r = %.4f; 95%% Fisher CI = [%.4f, %.4f]\n",
    label, r, lower, upper
  ))
}

# Candidate graph under test.
g <- dagitty('dag {
  prior_orders -> predicted_purchase_score
  prior_profit -> predicted_purchase_score
  sessions_pre -> predicted_purchase_score
  predicted_purchase_score -> treated
  prior_orders -> treated;  prior_profit -> treated;  sessions_pre -> treated
  prior_orders -> profit_30d;  prior_profit -> profit_30d;  sessions_pre -> profit_30d
  treated -> profit_30d
}')

# The DAG IMPLIES a list of conditional independencies. Read them off.
impliedConditionalIndependencies(g)

# Report all four candidate-graph implications.
report_correlation("prior_orders _||_ prior_profit",
                   df$prior_orders, df$prior_profit)
report_correlation("prior_orders _||_ sessions_pre",
                   df$prior_orders, df$sessions_pre)
report_correlation("prior_profit _||_ sessions_pre",
                   df$prior_profit, df$sessions_pre)

# The primary test residualizes score and profit on exposure and core controls.
r_score  <- resid(lm(predicted_purchase_score ~ treated + prior_orders +
                       prior_profit + sessions_pre, data = df))
r_profit <- resid(lm(profit_30d ~ treated + prior_orders +
                       prior_profit + sessions_pre, data = df))
report_correlation("predicted_purchase_score _||_ profit_30d after adjustment for the initial model's regressors",
                   r_score, r_profit)
#> prior_orders _||_ prior_profit: Pearson r = 0.7343; 95% Fisher CI = [0.7290, 0.7395]
#> prior_orders _||_ sessions_pre: Pearson r = 0.2166; 95% Fisher CI = [0.2058, 0.2274]
#> prior_profit _||_ sessions_pre: Pearson r = 0.3655; 95% Fisher CI = [0.3557, 0.3753]
#> predicted_purchase_score _||_ profit_30d after adjustment for the initial model's regressors: Pearson r = 0.1188; 95% Fisher CI = [0.1076, 0.1300]
import pandas as pd
import numpy as np
import networkx as nx
import statsmodels.formula.api as smf
from scipy.stats import norm

df = pd.read_csv("data/free_shipping_banner.csv")

def report_correlation(label, x, y):
    keep = x.notna() & y.notna()
    x, y = x[keep], y[keep]
    r = x.corr(y, method="pearson")
    delta = norm.ppf(0.975) / (len(x) - 3) ** 0.5
    lower = np.tanh(np.arctanh(r) - delta)
    upper = np.tanh(np.arctanh(r) + delta)
    print(
        f"{label}: Pearson r = {r:.4f}; "
        f"95% Fisher CI = [{lower:.4f}, {upper:.4f}]"
    )

# networkx renamed d_separated to is_d_separator in 3.3; support both.
# Note: the `or` form matters — getattr(nx, "is_d_separator", nx.d_separated)
# evaluates the default eagerly and breaks on 3.5+, where d_separated is gone.
is_d_separator = getattr(nx, "is_d_separator", None) or nx.d_separated

# Candidate graph under test; verify its four implications before testing them.
roots = ["prior_orders", "prior_profit", "sessions_pre"]
g = nx.DiGraph()
g.add_edges_from(
    [(x, "predicted_purchase_score") for x in roots]
    + [(x, "treated") for x in roots]
    + [(x, "profit_30d") for x in roots]
    + [("predicted_purchase_score", "treated"),
       ("treated", "profit_30d")]
)
implications = [
    ("prior_orders _||_ prior_profit", "prior_orders", "prior_profit", set()),
    ("prior_orders _||_ sessions_pre", "prior_orders", "sessions_pre", set()),
    ("prior_profit _||_ sessions_pre", "prior_profit", "sessions_pre", set()),
    ("predicted_purchase_score _||_ profit_30d after adjustment for the initial model's regressors",
     "predicted_purchase_score", "profit_30d", {"treated", *roots}),
]
for label, left, right, given in implications:
    assert is_d_separator(g, {left}, {right}, given), label

# Test the three marginal implications.
report_correlation("prior_orders _||_ prior_profit",
                   df["prior_orders"], df["prior_profit"])
report_correlation("prior_orders _||_ sessions_pre",
                   df["prior_orders"], df["sessions_pre"])
report_correlation("prior_profit _||_ sessions_pre",
                   df["prior_profit"], df["sessions_pre"])

# Test the primary conditional implication.
ctrl = "treated + prior_orders + prior_profit + sessions_pre"
r_score = smf.ols(f"predicted_purchase_score ~ {ctrl}", data=df).fit().resid
r_profit = smf.ols(f"profit_30d ~ {ctrl}", data=df).fit().resid
report_correlation("predicted_purchase_score _||_ profit_30d after adjustment for the initial model's regressors",
                   r_score, r_profit)
# prior_orders _||_ prior_profit: Pearson r = 0.7343; 95% Fisher CI = [0.7290, 0.7395]
# prior_orders _||_ sessions_pre: Pearson r = 0.2166; 95% Fisher CI = [0.2058, 0.2274]
# prior_profit _||_ sessions_pre: Pearson r = 0.3655; 95% Fisher CI = [0.3557, 0.3753]
# predicted_purchase_score _||_ profit_30d after adjustment for the initial model's regressors: Pearson r = 0.1188; 95% Fisher CI = [0.1076, 0.1300]

All four observed correlations contradict the candidate graph’s independence predictions. The three marginal correlations are 0.73 for prior orders and prior profit, 0.22 for prior orders and pre-period sessions, and 0.37 for prior profit and pre-period sessions, none of them close to the zero the graph claims. The primary adjusted score-profit correlation is 0.12 [0.11, 0.13]. After shipping eligibility and its interaction with treatment are added, that last correlation falls to 0.11 [0.09, 0.12], with an interval that still excludes zero.

In the known DGP, score and profit are associated through shared observed and unobserved causes and through score -> treated -> profit; there is no direct score-to-profit arrow. After adjusting for the interaction model’s regressors, the remaining correlation is consistent with unobserved shopping intent affecting both score and profit. The tests alone do not identify that cause: omitted eligibility relationships, an incorrectly specified functional form, and measurement problems remain possible explanations in an applied analysis.

13.4.7 Two habits that keep placebo tests honest

Define the set of placebo checks before looking at the results. The banner checks draw on overlapping users and related covariates and models, so their results need not be independent. Do not treat their agreement as independent confirmation. Read their estimates and intervals together with what each check should find if its assumptions hold. If you use the checks to make a formal set of statistical claims, use a joint test or a procedure that accounts for multiple testing, chosen for that pre-specified set of claims (Section 5.5.2). The event-study leads discussed earlier belong to one model and test one joint null hypothesis (Section 9.5.3.2). The banner checks instead fit separate models to overlapping users. Adjusting for multiple testing does not make their results independent or remove any bias they share. The balance SMDs remain descriptive comparisons against a threshold; do not include them as one hypothesis test per covariate.

The second habit is to decide in advance what will count as evidence. If you choose which placebo matters after seeing the results, you can always protect your claim: ignore the checks that detect an effect and highlight those that do not. Write down the tests, outcomes, and decision criteria before you look at the data, just as you would define the causal question in advance (Section 3.1). Pre-registration records those choices. A result that agrees with the specified prediction is informative only if the same test could have produced evidence against the claim.

Report which implication the data contradicted, what uncertainty surrounds the estimate, how the checks relate to one another, and what alternative explanations remain.

13.5 Robustness and specification checks

Falsification checks ask where the effect should be zero, but every analysis involves choices about controls, estimators, and samples. More than one choice may be defensible, so a colleague repeating the analysis could reasonably make different choices. A robustness check asks whether the conclusion changes under those alternatives. If the estimate changes little across defensible choices, you have evidence that the result does not depend heavily on one analytical decision; a causal interpretation still depends on the identifying assumptions shared by those analyses.

Compare the specifications for effect-size stability rather than statistical significance. A row of stars across six specifications indicates statistical significance under each model’s assumptions; it does not show that the effect size is stable. For a decision, ask whether the estimated effects are similar enough to support the same action across defensible choices. Large differences between estimates can matter even when every estimate is statistically different from zero.

Use this template for every robustness check (Huntington-Klein, n.d.):

  1. Name the baseline assumption.
  2. Predict the direction of bias if it fails.
  3. Give a dataset-specific reason it may fail.
  4. State the alternative specification.
  5. Before running it, decide how to interpret a change in the estimate.

Take the banner’s initial adjusted model through that template once. The initial adjusted model assumes the pre-treatment behavior we control for (orders, profit, sessions) captures the selection the targeting model acted on. If it does not, the bias points up, because the targeting model favored high-intent users who spend more for reasons we have not fully measured. The placebo-in-time and negative-control checks indicate that the controls helped but did not fully close the gap. The targeting score can help because it reflects some of the unobserved shopping intent that the other controls miss. A large change after adding the score shows that the estimate is sensitive to adjusting for the targeting process; the change alone neither proves bias nor identifies its direction. In another setting, adding a variable that predicts treatment but not the outcome can amplify residual confounding rather than reduce it (Cinelli, Forney, et al. 2024).

Choose each robustness specification to address a specific threat you can name, not because it’s easy to run a lot of them (Neumayer and Plümper 2017). As we discussed in previous chapters, choosing one preferred estimate from twenty unjustified regressions is specification fishing. Explain which assumption each specification examines and what its result would mean, regardless of how many you run.

13.5.1 Add the eligibility interaction and average the adjusted profit difference over targeted users

Before adding the targeting score to the profit regression, we address another limitation of the initial adjusted model: it uses one treatment coefficient for both eligible and ineligible users. Adding shipping eligibility and its interaction with treatment allows the adjusted difference in mean profit between treated and untreated users to differ by eligibility. We call this the interaction model and use it as the main specification for the remaining checks. We center eligibility at its observed share among treated users, 0.659 (65.9% of them are eligible):

\[ \begin{aligned} \text{eligibility\_centered}_i &= \text{shipping\_eligible}_i \\[0.35em] &\quad - \widehat{\mathbb E}[\text{shipping\_eligible}_i \mid \text{treated}_i = 1]. \end{aligned} \]

That centered variable enters the interaction model:

\[ \begin{aligned} \text{profit\_30d}_i =\ & \beta_0 + \beta_1\,\text{treated}_i + \beta_2\,\text{eligibility\_centered}_i \\[0.35em] & + \beta_3\,\bigl(\text{treated}_i \times \text{eligibility\_centered}_i\bigr) \\[0.35em] & + \beta_4\,\text{prior\_orders}_i + \beta_5\,\text{prior\_profit}_i + \beta_6\,\text{sessions\_pre}_i + \varepsilon_i \end{aligned} \tag{13.1}\]

With this centering, \(\beta_1\) is the model’s fitted difference in mean 30-day profit between treated and untreated users, averaged across eligible and ineligible users in the proportions observed among currently targeted users. It is R$1.87 [R$1.60, R$2.13] with HC1. This averaging follows from centering, even if the model is misspecified. A causal interpretation depends on the treatment-effect function being represented by the model and on the identifying assumptions. If the score, prior behavior, region, device, or another covariate modifies the effect, the model needs to represent that heterogeneity. Centering makes the treatment coefficient equal this weighted average; it does not account for treatment-effect variation omitted from the model or identify the ATT.

Without centering, the same fitted model gives R$1.25 for ineligible users, R$2.18 for eligible users, and R$0.94 for the gap between them. Both parameterizations predict identical profit for every user; only the coefficient’s interpretation changes. Adding the interaction changes the specification. Neither the interaction nor the centering establishes causal identification.

The interval for \(\beta_1\) treats the centering constant as known, so it omits uncertainty from estimating that constant.

library(lmtest)
library(sandwich)

df <- read.csv("data/free_shipping_banner.csv")

report_regression <- function(label, model, term, covariance = "HC1") {
  test <- coeftest(model, vcov. = vcovHC(model, type = covariance))
  estimate <- test[term, "Estimate"]
  se <- test[term, "Std. Error"]
  critical <- qt(0.975, df = df.residual(model))
  cat(sprintf(
    "%s: estimate = %.4f; %s SE = %.4f; 95%% Student-t CI = [%.4f, %.4f]\n",
    label, estimate, covariance, se,
    estimate - critical * se, estimate + critical * se
  ))
}

treated_eligibility_share <- mean(df$shipping_eligible[df$treated == 1])
df$eligibility_centered <-
  df$shipping_eligible - treated_eligibility_share

interaction_model <- lm(
  profit_30d ~ treated * eligibility_centered +
    prior_orders + prior_profit + sessions_pre,
  data = df
)
report_regression("Interaction model", interaction_model, "treated")
#> Interaction model: estimate = 1.8657; HC1 SE = 0.1341; 95% Student-t CI = [1.6028, 2.1285]

uncentered_model <- lm(
  profit_30d ~ treated * shipping_eligible +
    prior_orders + prior_profit + sessions_pre,
  data = df
)
# Both parameterizations are the same fit with different labels.
stopifnot(max(abs(fitted(interaction_model) - fitted(uncentered_model))) < 1e-9)
df$ineligibility <- 1 - df$shipping_eligible
eligible_parameterization_model <- lm(
  profit_30d ~ treated * ineligibility +
    prior_orders + prior_profit + sessions_pre,
  data = df
)
report_regression("adjusted profit difference among ineligible users (interaction model)", uncentered_model, "treated")
report_regression("uncentered treatment-by-eligibility interaction", uncentered_model,
                  "treated:shipping_eligible")
report_regression("adjusted profit difference among eligible users", eligible_parameterization_model, "treated")
import pandas as pd
import statsmodels.formula.api as smf
from scipy.stats import t as student_t

df = pd.read_csv("data/free_shipping_banner.csv")

def report_regression(label, model, term, covariance="HC1"):
    estimate = model.params[term]
    se = model.bse[term]
    critical = student_t.ppf(0.975, model.df_resid)
    print(
        f"{label}: estimate = {estimate:.4f}; {covariance} SE = {se:.4f}; "
        f"95% Student-t CI = [{estimate - critical * se:.4f}, "
        f"{estimate + critical * se:.4f}]"
    )

treated_eligibility_share = df.loc[
    df["treated"] == 1, "shipping_eligible"
].mean()
df["eligibility_centered"] = (
    df["shipping_eligible"] - treated_eligibility_share
)

interaction_model = smf.ols(
    "profit_30d ~ treated * eligibility_centered "
    "+ prior_orders + prior_profit + sessions_pre",
    data=df,
).fit(cov_type="HC1")
report_regression("Interaction model", interaction_model, "treated")
# Interaction model: estimate = 1.8657; HC1 SE = 0.1341; 95% Student-t CI = [1.6028, 2.1285]

uncentered_model = smf.ols(
    "profit_30d ~ treated * shipping_eligible "
    "+ prior_orders + prior_profit + sessions_pre",
    data=df,
).fit(cov_type="HC1")
# Both parameterizations are the same fit with different labels.
assert (interaction_model.fittedvalues - uncentered_model.fittedvalues).abs().max() < 1e-9
df["ineligibility"] = 1 - df["shipping_eligible"]
eligible_parameterization_model = smf.ols(
    "profit_30d ~ treated * ineligibility "
    "+ prior_orders + prior_profit + sessions_pre",
    data=df,
).fit(cov_type="HC1")
report_regression("adjusted profit difference among ineligible users (interaction model)", uncentered_model, "treated")
report_regression(
    "uncentered treatment-by-eligibility interaction",
    uncentered_model,
    "treated:shipping_eligible",
)
report_regression("adjusted profit difference among eligible users", eligible_parameterization_model, "treated")

The main warnings remain after adding the eligibility interaction. Table 13.2 compares the applicable diagnostics from both specifications; the remaining analysis uses the results from the interaction model.

Table 13.2: Diagnostic results using the initial adjusted model and the interaction model.
Diagnostic Initial adjusted model Interaction model
Placebo in time R$1.59 [R$1.32, R$1.86] R$1.60 [R$1.33, R$1.88]
Negative-control association R$1.11 [R$0.95, R$1.28] R$1.13 [R$0.96, R$1.29]
Deterministic null permutation R$-0.06 [R$-0.30, R$0.17] R$-0.05 [R$-0.28, R$0.18]
Score-preserving fake exposure R$1.59 [R$1.23, R$1.95] R$1.38 [R$1.02, R$1.73]
Adjusted profit difference among ineligible users R$1.12 [R$0.70, R$1.54] R$1.25 [R$0.86, R$1.64]
Residual score-profit correlation 0.1188 [0.1076, 0.1300] 0.1058 [0.0946, 0.1170]

The rows for the deterministic null permutation and score-preserving fake exposure each add their placebo indicator as one more regressor on the right-hand side of Equation 13.1. The placebo-in-time and negative-control rows reuse the right-hand-side terms of Equation 13.1 in separate fits and swap only the outcome. For the residual score-profit correlation, regress each variable on the interaction model’s predictors and correlate the residuals, instead of adjusting for the initial model’s predictors. The ineligible placebo-unit row is the one exception: it uses the treated coefficient of the same fitted model, written with uncentered shipping_eligible in place of the centered version. This coefficient gives the adjusted profit difference for ineligible users rather than an average of the eligible and ineligible groups’ differences using their proportions among treated users.

13.5.2 Check the interaction model across defensible choices

Figure 13.7 plots the adjusted profit difference (in R$) on the horizontal axis and six defensible analytic choices on the vertical axis: the interaction model, the model with additional covariates (tenure, loyalty tier, region, and device), the model that also includes the targeting score, common-support trimming, IPW for the ATT, and 1:1 nearest-neighbor matching with replacement. All six keep the treatment-by-eligibility interaction fixed. The rows form two clusters. The estimates from the interaction model, the model with additional covariates, and common-support trimming remain near R$1.87. Adjustment for the targeting score, IPW, and matching give estimates between R$1.08 and R$1.22. The estimate changes when the analysis accounts more directly for the targeting process. The raw R$7.86 comparison stays outside the six adjusted specifications as a baseline.

Every row centers shipping eligibility at the share among treated users in the full sample. Each reported adjusted profit difference therefore averages the fitted differences for eligible and ineligible users using their proportions among those treated users. The estimation samples and weights still differ. Common-support trimming drops 18 users, including 10 treated users; the targeted-user truth in that trimmed sample is essentially unchanged at R$0.99 after rounding. IPW changes the weights, and matching replaces the full control sample with reused nearest neighbors.

Common-support trimming here keeps observations inside the overlap of the observed targeting-score ranges. Overlapping ranges do not establish practical positivity: some parts of that range may contain few users from one treatment group. Before judging overlap adequate, show the score or propensity density by treatment group, the largest weights, and ESS.

Read the IPW estimate of R$1.22 [R$0.88, R$1.56] only as a check of how adjustment for observed targeting variables changes the result. It is closer to the simulated truth but still 23% high, and both tested placebos still give positive estimates with intervals that exclude zero. The IPW placebo-in-time estimate is R$0.63 [R$0.29, R$0.97]. The IPW noneligible-category placebo estimate is R$0.39 [R$0.18, R$0.60]. No other IPW placebo is tested here, and this row is not a preferred causal estimate.

The reported IPW interval treats the estimated weights as fixed: HC1 from the weighted regression does not include uncertainty from fitting the propensity model. Before using the IPW result for inference, report the treated and control weight distributions and ESS, check practical positivity, and use a variance estimator or bootstrap that accounts for estimated weights. Apply the same rule in R and Python.

Figure 13.7: Robustness comparison: six adjusted specifications with the eligibility interaction range from R$1.08 to R$1.87, with one R$0.99 targeted-user ATT reference line. The raw R$7.86 comparison is a separate baseline.
library(tidyverse)
library(lmtest)
library(sandwich)
library(WeightIt)
library(MatchIt)

df <- read.csv("data/free_shipping_banner.csv")

report_regression <- function(label, model, term = "treated", covariance = "HC1") {
  test <- coeftest(model, vcov. = vcovHC(model, type = covariance))
  estimate <- test[term, "Estimate"]
  se <- test[term, "Std. Error"]
  critical <- qt(0.975, df = df.residual(model))
  cat(sprintf(
    "%s: estimate = %.4f; %s SE = %.4f; 95%% Student-t CI = [%.4f, %.4f]\n",
    label, estimate, covariance, se,
    estimate - critical * se, estimate + critical * se
  ))
}

treated_eligibility_share <- mean(df$shipping_eligible[df$treated == 1])
df$eligibility_centered <-
  df$shipping_eligible - treated_eligibility_share
interaction_formula <- profit_30d ~ treated * eligibility_centered +
  prior_orders + prior_profit + sessions_pre
propensity <- treated ~ prior_orders + prior_profit + sessions_pre +
  predicted_purchase_score + tenure_days + shipping_eligible

# Same treatment-by-eligibility interaction and fixed centering in every adjusted row.
interaction_model <- lm(interaction_formula, data = df)
context <- lm(
  update(interaction_formula, . ~ . + tenure_days + factor(loyalty_tier) +
           factor(region) + factor(device)),
  data = df
)
score <- lm(update(formula(context), . ~ . + predicted_purchase_score), data = df)

ps_t <- range(df$predicted_purchase_score[df$treated == 1])
ps_c <- range(df$predicted_purchase_score[df$treated == 0])
cs_lo <- max(ps_t[1], ps_c[1]); cs_hi <- min(ps_t[2], ps_c[2])
df_trim <- df |> filter(predicted_purchase_score >= cs_lo,
                        predicted_purchase_score <= cs_hi)
trimmed <- lm(interaction_formula, data = df_trim)

# IPW for the ATT and 1:1 nearest-neighbour ATT matching with replacement.
w <- weightit(propensity, data = df, method = "glm", estimand = "ATT")
ipw <- lm(interaction_formula, data = df, weights = w$weights)
ipw_placebo_time <- lm(
  update(interaction_formula, profit_pre ~ .), data = df, weights = w$weights
)
ipw_negative_control <- lm(
  update(interaction_formula, noneligible_category_profit ~ .),
  data = df, weights = w$weights
)
ps_model <- glm(propensity, data = df, family = binomial())
mm <- matchit(
  propensity, data = df, method = "nearest",
  distance = predict(ps_model, type = "response"), estimand = "ATT",
  replace = TRUE, ratio = 1, normalize = FALSE
)
matched <- match_data(mm, data = df)
nn <- lm(interaction_formula, data = matched, weights = weights)

# HC1 for five rows; HC3 for the fitted match. Every row prints the
# estimate, robust SE, and a Student-t interval using its residual df.
models <- list("Interaction model" = interaction_model, "Additional covariates" = context, score = score,
               trimmed = trimmed, ipw = ipw, matching = nn)
walk(names(models), function(name) {
  covariance <- if (name == "matching") "HC3" else "HC1"
  report_regression(name, models[[name]], covariance = covariance)
})
report_regression("IPW placebo in time", ipw_placebo_time)
report_regression("IPW noneligible-category placebo", ipw_negative_control)
cat(sprintf(
  "common-support trim: %d rows removed; %d treated removed; targeted truth = %.6f\n",
  nrow(df) - nrow(df_trim),
  sum(df$treated == 1) - sum(df_trim$treated == 1),
  mean(df_trim$true_effect_proxy[df_trim$treated == 1])
))
# adjusted estimates: 1.87, 1.86, 1.21, 1.86, 1.22, 1.08
# IPW placebo in time: estimate = 0.6296; HC1 SE = 0.1733; 95% Student-t CI = [0.2900, 0.9692]
# IPW noneligible-category placebo: estimate = 0.3902; HC1 SE = 0.1047; 95% Student-t CI = [0.1849, 0.5955]
# common-support trim: 18 rows removed; 10 treated removed; targeted truth = 0.987911
import pandas as pd
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf
from scipy.stats import t as student_t
from sklearn.neighbors import NearestNeighbors

df = pd.read_csv("data/free_shipping_banner.csv")

def report_regression(label, model, term="treated", covariance="HC1"):
    estimate = model.params[term]
    se = model.bse[term]
    critical = student_t.ppf(0.975, model.df_resid)
    print(
        f"{label}: estimate = {estimate:.4f}; {covariance} SE = {se:.4f}; "
        f"95% Student-t CI = [{estimate - critical * se:.4f}, "
        f"{estimate + critical * se:.4f}]"
    )

treated_eligibility_share = df.loc[
    df.treated == 1, "shipping_eligible"
].mean()
df["eligibility_centered"] = (
    df["shipping_eligible"] - treated_eligibility_share
)
interaction_rhs = "treated * eligibility_centered + prior_orders + prior_profit + sessions_pre"
ctx = interaction_rhs + " + tenure_days + C(loyalty_tier) + C(region) + C(device)"
propensity = (
    "treated ~ prior_orders + prior_profit + sessions_pre + "
    "predicted_purchase_score + tenure_days + shipping_eligible"
)

interaction_model = smf.ols(f"profit_30d ~ {interaction_rhs}", df).fit(cov_type="HC1")
context  = smf.ols(f"profit_30d ~ {ctx}", df).fit(cov_type="HC1")
score    = smf.ols(f"profit_30d ~ {ctx} + predicted_purchase_score", df).fit(cov_type="HC1")

# Trim to common support on the targeting score, then re-fit the interaction model.
score_t, score_c = df.loc[df.treated == 1, "predicted_purchase_score"], \
                   df.loc[df.treated == 0, "predicted_purchase_score"]
cs_lo, cs_hi = max(score_t.min(), score_c.min()), min(score_t.max(), score_c.max())
df_trim = df[df.predicted_purchase_score.between(cs_lo, cs_hi)]
trimmed = smf.ols(f"profit_30d ~ {interaction_rhs}", df_trim).fit(cov_type="HC1")

# Formula-based unpenalized binomial GLM for IPW and matching.
ps_model = smf.glm(propensity, data=df,
                   family=sm.families.Binomial()).fit()
ps = ps_model.predict(df)
wt = np.where(df["treated"] == 1, 1.0, ps / (1 - ps))
ipw = smf.wls(f"profit_30d ~ {interaction_rhs}", df, weights=wt).fit(cov_type="HC1")
ipw_placebo_time = smf.wls(
    f"profit_pre ~ {interaction_rhs}", df, weights=wt
).fit(cov_type="HC1")
ipw_negative_control = smf.wls(
    f"noneligible_category_profit ~ {interaction_rhs}", df, weights=wt
).fit(cov_type="HC1")

# Match with replacement; keep each control once with its raw reuse count.
df = df.assign(ps=ps)
t, c = df[df.treated == 1], df[df.treated == 0]
idx = (NearestNeighbors(n_neighbors=1).fit(c[["ps"]])
       .kneighbors(t[["ps"]], return_distance=False).ravel())
control_rows, reuse = np.unique(c.iloc[idx].index, return_counts=True)
matched_controls = c.loc[control_rows].copy()
matched_controls["match_weight"] = reuse
t = t.copy(); t["match_weight"] = 1
matched = pd.concat([t, matched_controls])
nn = smf.wls(f"profit_30d ~ {interaction_rhs}", matched,
             weights=matched["match_weight"]).fit(cov_type="HC3")

# Read the six adjusted estimates; never duplicate reused controls.
for name, m in [("Interaction model", interaction_model), ("Additional covariates", context),
                ("score", score), ("trimmed", trimmed), ("ipw", ipw),
                ("matching", nn)]:
    covariance = "HC3" if name == "matching" else "HC1"
    report_regression(name, m, covariance=covariance)
report_regression("IPW placebo in time", ipw_placebo_time)
report_regression("IPW noneligible-category placebo", ipw_negative_control)
print(
    "common-support trim: "
    f"{len(df) - len(df_trim)} rows removed; "
    f"{df.treated.sum() - df_trim.treated.sum()} treated removed; "
    f"targeted truth = {df_trim.loc[df_trim.treated == 1, 'true_effect_proxy'].mean():.6f}"
)
# adjusted estimates: 1.87, 1.86, 1.21, 1.86, 1.22, 1.08
# IPW placebo in time: estimate = 0.6296; HC1 SE = 0.1733; 95% Student-t CI = [0.2900, 0.9692]
# IPW noneligible-category placebo: estimate = 0.3902; HC1 SE = 0.1047; 95% Student-t CI = [0.1849, 0.5955]
# common-support trim: 18 rows removed; 10 treated removed; targeted truth = 0.987911

Report each specification’s estimate rather than averaging across specifications. The R$1.08–R$1.87 range shows substantial variation: estimates fall when the analysis uses the targeting process more directly. The matching coefficient is R$1.08, with a conditional interval of [R$0.65, R$1.51] computed with HC3 standard errors. Treat that interval as descriptive. HC3 from the weighted outcome regression is not a matching-specific variance estimator and does not account for estimating the propensity score or selecting and reusing controls. For inference, use matching-aware uncertainty estimation and report match-distance and reuse diagnostics.

These specifications make different choices about the observed data but share the risk of confounding from unobserved shopping intent. Every point estimate remains above the targeted-user truth R$0.99. Their agreement on a positive sign does not remove that shared bias, and the variation in their magnitudes shows that the estimated effect size still depends on analytical choices.

Changing the population changes the question. Restricting the initial adjusted model to shipping-eligible users produces R$2.27 [R$1.93, R$2.60]. Compare this estimate with the eligible-targeted ATT of R$1.50, rather than the R$0.99 truth for all targeted users. This eligible-only result is excluded from the six-specification comparison, which averages the eligible and ineligible groups’ adjusted profit differences using their proportions among treated users in the full sample.

Consensus is not correctness. When every specification relies on the same identifying assumption (here, that the observed covariates capture selection), the estimates can agree while sharing the same bias. Similar estimates provide reassurance about the choices tested, but sensitivity to other specifications and shared bias remain possible. The sensitivity analysis ahead quantifies how strong an omitted confounder would have to be to overturn the estimate, while a randomized design would change the assignment mechanism that creates the shared risk of confounding.

Robustness checks are method-specific. In regression discontinuity, report the effect across defensible bandwidths and kernels (Section 8.2.3, Table 8.1). In difference-in-differences, a heterogeneity-robust estimator addresses staggered timing, but it does not repair violations of parallel trends (Section 10.6). Treat those as separate threats: compare estimators for treatment-effect heterogeneity, then use a sensitivity analysis that bounds how far post-treatment trends may depart from the pre-period pattern (Rambachan and Roth 2023).

When there are many specifications to compare, a specification curve plots the estimate across every pre-declared combination of analytic choices (Simonsohn et al. 2020). Multiverse analysis extends the same idea through the full data-cleaning and coding process (Steegen et al. 2016). Both extend the six-specification comparison used here.

13.5.3 Check robustness to excluding individual regions

Before trusting the full-sample estimate, check whether the result depends on a single region. Refit the interaction model after excluding each region in turn. If excluding one region changes the estimate substantially, explain why the estimate depends on that region. Slicing the data until something is significant manufactures false positives; sample-splitting is the structural cure when the goal is subgroup discovery (Chapter 12). This check has a narrower purpose: determine whether omitting any region materially changes the full-sample estimate.

library(tidyverse)
library(fixest)

df <- read.csv("data/free_shipping_banner.csv")
treated_eligibility_share <- mean(df$shipping_eligible[df$treated == 1])
df$eligibility_centered <-
  df$shipping_eligible - treated_eligibility_share
interaction_rhs <- "treated * eligibility_centered + prior_orders + prior_profit + sessions_pre"

# Leave-one-out over region: does any single region carry the result?
loo <- map_dfr(unique(df$region), function(r) {
  m <- feols(as.formula(paste("profit_30d ~", interaction_rhs)),
             data = filter(df, region != r), vcov = "hetero")
  tibble(dropped = r, treated = coef(m)["treated"])
})

# Full-sample estimate for reference, then the leave-one-out spread.
full <- coef(feols(as.formula(paste("profit_30d ~", interaction_rhs)),
                   data = df, vcov = "hetero"))["treated"]
loo |> mutate(full = full)   # if one row swings far from `full`, be suspicious
import pandas as pd
import statsmodels.formula.api as smf

df = pd.read_csv("data/free_shipping_banner.csv")
treated_eligibility_share = df.loc[
    df.treated == 1, "shipping_eligible"
].mean()
df["eligibility_centered"] = (
    df["shipping_eligible"] - treated_eligibility_share
)
interaction_rhs = "treated * eligibility_centered + prior_orders + prior_profit + sessions_pre"

# Leave-one-out over region: does any single region carry the result?
full = smf.ols(f"profit_30d ~ {interaction_rhs}", df).fit().params["treated"]
for r in df["region"].unique():
    m = smf.ols(f"profit_30d ~ {interaction_rhs}", df[df.region != r]).fit(cov_type="HC1")
    print(r, round(m.params["treated"], 2), "vs full", round(full, 2))
# range R$1.72-R$1.95; spread R$0.23

Excluding one region at a time leaves the estimate from the interaction model between R$1.72 and R$1.95, a spread of R$0.23 (Table 13.3):

Table 13.3: Estimates from the interaction model for the full sample and after excluding each region in turn.
Sample Estimate from interaction model
Full sample R$1.87
Exclude east R$1.95
Exclude north R$1.92
Exclude south R$1.72
Exclude west R$1.88

Compare each row with the full-sample R$1.87 estimate. Excluding the south gives the low endpoint, and excluding the east gives the high endpoint; the other exclusions are closer to the full-sample result. No single region determines the estimate. This check still reuses the same observational design.

13.6 Sensitivity to unobserved confounding

The placebo-in-time, negative-control, and score-preserving fake-exposure checks still show positive associations under the interaction model (Table 13.2). The model nevertheless rests on the assumption that observed controls account for how the targeting system selected users. An unobserved confounder cannot be inspected directly.

So the question changes from whether hidden confounding is absent to how strong it would have to be to overturn the estimate from the interaction model. Sensitivity analysis does not remove bias. It quantifies that required confounder strength so you can compare it with observed covariates as benchmarks (Cinelli and Hazlett 2020).

The partial-\(R^2\) sensitivity framework describes an omitted confounder’s strength using linear regressions. For treatment, first account for the interaction model’s other predictors, then ask what share of the remaining variation in banner exposure the confounder could explain. For the outcome, first account for treatment and those predictors, then ask what share of the remaining variation in profit the confounder could explain. Those shares are the two partial \(R^2\) values; the variation left after adjustment is called residual variation. We apply it to \(\beta_1\) from Equation 13.1 using ordinary least squares with classical, homoskedastic standard errors. This differs from the HC1 robust standard errors reported for the estimate from the interaction model and its diagnostics.6

The Robustness Value (RV) turns these two relationships into a single threshold. For a hypothetical confounder with equal partial \(R^2\) values on both sides, it is the minimum common value that could reduce the treatment coefficient to zero, with the direction of confounding chosen to reduce the estimate. Here, the RV is 0.077: explaining about 7.7% of the residual variation on each side could account for the entire coefficient estimate. The setting q = 1 specifies this target: a 100% reduction in the estimate. The RV for statistical significance, labeled RV-alpha in the code output, is 0.067 at \(\alpha = 0.05\). Using the same equal-strength comparison and classical OLS inference, confounding of about 6.7% on each side could make the adjusted 95% confidence interval include zero. The adjusted point estimate can still be positive. The two RVs are close because the classical estimate is precise. Judge sensitivity by comparing their values with substantively defensible benchmarks, rather than by the gap between them.

The treatment’s partial \(R^2\) with the outcome is 0.0065. This is the share of residual outcome variation associated with treatment after accounting for the interaction model’s other terms. It describes the remaining association between treatment and the outcome; it is not, by itself, a measure of causal strength. The same calculation also reports an extreme scenario: a confounder that explained all residual outcome variation would need to explain only 0.65% of residual treatment variation to account for the estimate. That is a low threshold, not reassurance.

Use prior_profit to interpret the sensitivity thresholds. The formal 1× bound uses prior_profit as the benchmark for how strongly an omitted confounder could be related to treatment and the outcome. Under the benchmarking assumptions, the calculation gives partial-\(R^2\) bounds of 0.0946 with treatment and 0.1666 with the outcome. These are calculated bounds for the hypothetical confounder, not the observed partial-\(R^2\) values of prior_profit itself. At those bounds, choosing the direction of confounding that most reduces the banner coefficient gives R$-1.19. This is a worst-case sensitivity bound, not a predicted corrected effect.

prior_profit is a calibration benchmark, not evidence that an omitted confounder with those exact relationships exists. In this simulation, we know the DGP and have access to the otherwise unobserved shopping_intent, so we can adjust for the hidden confounder itself. Analysts do not have that information in an applied analysis, so they cannot make this direct adjustment.

This direct adjustment is an oracle-adjusted audit. Adding the actual unobserved intent variable to the interaction regression gives a treated coefficient of R$0.91 [R$0.64, R$1.17]. The confounder’s raw partial \(R^2\) is 0.041 with treatment and 0.040 with the outcome. This coefficient is not the truth itself: the targeted-user ATT truth is a DGP average, while R$0.91 is a regression coefficient adjusted with information an analyst would not observe. It is also not the formal 1× bound, which asks a worst-case calibration question.

Sensitivity contour with residual treatment association on the horizontal partial-R-squared axis and residual outcome association on the vertical axis. A dotted equal-strength line connects points with the same value on both axes. A diamond marks the 6.69% interval threshold, where the estimate remains positive. A cross marks the 7.74% zero-estimate threshold on the red zero contour. Separate markers retain the estimate before omitted-confounder adjustment, the simulated hidden confounder, and the formal prior-profit bound.
Figure 13.8: Sensitivity contour for the interaction model using classical OLS inference. The horizontal axis measures a hypothetical confounder’s partial R-squared with treatment; the vertical axis measures its partial R-squared with the outcome. Separate markers show the simulated hidden confounder and the formal 1× prior-profit bound. Along the equal-strength line, the adjusted 95% interval first includes zero at 6.69% on each axis, and the adjusted estimate reaches zero at 7.74% on each axis.

In Figure 13.8, the horizontal axis is a hypothetical confounder’s residual association with treatment and the vertical axis is its residual association with the outcome. Each contour shows the interaction model’s treated coefficient after adjustment; the red zero contour marks where the coefficient changes sign. The oracle-confounder point uses its raw partial-\(R^2\) coordinates and reproduces the fitted R$0.91 coefficient. The formal 1× prior_profit point lies beyond the zero contour and gives a worst-case bound of R$-1.19. The two markers share the same axes but represent different calculations.

Along the equal-strength line, the confounder explains the same share of residual variation on both axes. The RV-alpha threshold is 6.69% on each axis: the adjusted 95% interval first includes zero while the adjusted estimate remains positive. The RV threshold is 7.74% on each axis: the adjusted estimate itself reaches zero. These are the same thresholds described earlier, expressed as percentages to two decimal places. They use classical OLS inference and the direction of confounding that reduces the estimate.

library(tidyverse)
library(sensemakr)

df <- read.csv("data/free_shipping_banner.csv")
treated_eligibility_share <- mean(df$shipping_eligible[df$treated == 1])
df$eligibility_centered <-
  df$shipping_eligible - treated_eligibility_share

# The interaction lm fit with classical inference required by the package interface.
model <- lm(
  profit_30d ~ treated * eligibility_centered +
    prior_orders + prior_profit + sessions_pre,
  data = df
)

# Sensitivity of the `treated` coefficient, benchmarked against the
# strongest observed control, prior_profit.
sens <- sensemakr(
  model, treatment = "treated",
  benchmark_covariates = "prior_profit", kd = 1
)
summary(sens)
# RV = 0.077; RV-alpha = 0.067; treatment partial R2 = 0.0065
# formal 1× prior-profit bound: R$-1.19 [R$-1.44, R$-0.94]
plot(sens)
import pandas as pd
import statsmodels.formula.api as smf
import sensemakr                       # PySensemakr

df = pd.read_csv("data/free_shipping_banner.csv")
treated_eligibility_share = df.loc[
    df["treated"] == 1, "shipping_eligible"
].mean()
df["eligibility_centered"] = (
    df["shipping_eligible"] - treated_eligibility_share
)

# The interaction OLS fit with classical inference required by the package interface.
model = smf.ols(
    "profit_30d ~ treated * eligibility_centered "
    "+ prior_orders + prior_profit + sessions_pre",
    data=df,
).fit()

# Sensitivity of the `treated` coefficient, benchmarked against the
# strongest observed control, prior_profit.
sens = sensemakr.Sensemakr(
    model, treatment="treated",
    benchmark_covariates="prior_profit", kd=[1]
)
sens.summary()
# RV = 0.077; RV-alpha = 0.067; treatment partial R2 = 0.0065
# formal 1× prior-profit bound: R$-1.19 [R$-1.44, R$-0.94]

The partial-\(R^2\) bound is one of several formal sensitivity tools suited to different settings. We introduce three alternatives here. Oster’s \(\delta\) uses changes in the coefficient and \(R^2\) as controls are added to assess how strong selection on unobservables would have to be relative to selection on the observables you already control for (Oster 2019). The E-value measures sensitivity to unobserved confounding on the risk-ratio scale common in epidemiology, reporting the minimum confounder–exposure and confounder–outcome association needed to explain away an effect (VanderWeele and Ding 2017). The Masten–Poirier breakdown point does not require a linear model and asks, nonparametrically, how much the as-good-as-random assumption can be relaxed before the conclusion no longer holds (Masten and Poirier 2018, 2020), with nonparametric estimation and inference developed in Masten et al. (2024). Each parameterization asks how robust the result is to an assumption that cannot be tested directly.

There is no universal “safe” robustness value. The RV for the treatment coefficient of 0.077 becomes informative only when paired with a substantively defensible benchmark and knowledge of the assignment process. Critics must describe a confounder of a specific strength, and proponents must acknowledge that the benchmark calibrates sensitivity rather than proving that the design removes confounding. We first pointed to this kind of sensitivity bound in the causal-assumptions chapter (Chapter 6).

Key takeaway. The sensitivity thresholds, oracle audit, and nonzero placebo estimates from the interaction model lead to the same decision: the regression estimate is too vulnerable to support a confident causal claim. This does not imply that the banner’s effect is zero. The next section considers what additional designs could help address the remaining uncertainty.

13.7 Competing causes, mechanisms, and triangulation

The placebos, robustness checks, and sensitivity bound examine different weaknesses in the observational claim. The next checks ask whether another change affected the same users at the same time and whether the effect appears only where the banner could produce it. The triangulation discussion then considers how RDD, DiD, or a randomized holdout could add evidence, and what each would require.

13.7.1 Competing causes and compound treatments

Other changes can produce the same treated–control gap as the banner: a category launch, payday cycle, regional campaign, price change on a popular stock keeping unit (SKU),7 or novelty. If the banner and other changes always reach the same users at the same time, this comparison cannot separate their effects. A causal interpretation of the bundle’s estimate still depends on the identifying assumptions. This is the compound-treatment problem from two-way fixed effects (Chapter 9) and the reason an RDD requires no other policy to jump at its cutoff (Section 8.2.1).

Before trusting an estimate, write down everything that happened to the treated group around the time of treatment and ask whether each change could produce the observed pattern. Did a promotion reach the same users the banner targeted? Did the banner go live in regions that also got a seasonal campaign? Is the lift concentrated in the first week or sustained over time? Specify what novelty and the shipping incentive would each predict in this setting; an early spike can arise from either. The list makes you consider each competing explanation before attributing the estimated effect to the banner alone.

13.7.2 Mechanism and specificity: does the effect show up only where it can?

The shipping mechanism predicts a positive causal effect on eligible-category profit and no causal effect on noneligible-category profit. The adjusted profit differences can follow a different ordering if unobserved shopping intent biases the two comparisons by different amounts. We therefore reuse the interaction model’s regressors in separate regressions for eligible-category profit, noneligible-category profit, and their per-user difference. This holds the sample, the controls, the centering, and the standard-error type fixed across all three (Lipsitch et al. 2010).

The adjusted difference in eligible-category profit is R$1.49 [R$1.33, R$1.66]. The adjusted difference in noneligible-category profit is R$1.13 [R$0.96, R$1.29]. The adjusted treated–untreated gap in profit per user is R$0.37 [R$0.15, R$0.58] larger in eligible categories than in noneligible categories. The positive difference is consistent with the banner’s mechanism, but the nonzero negative-control association still warns that a common cause lifts both categories. The category comparison helps assess the mechanism; it does not establish that the mechanism is correct.

library(tidyverse)
library(lmtest)
library(sandwich)

df <- read.csv("data/free_shipping_banner.csv")

report_regression <- function(label, model) {
  test <- coeftest(model, vcov. = vcovHC(model, type = "HC1"))
  estimate <- test["treated", "Estimate"]
  se <- test["treated", "Std. Error"]
  critical <- qt(0.975, df = df.residual(model))
  cat(sprintf(
    "%s: estimate = %.4f; HC1 SE = %.4f; 95%% Student-t CI = [%.4f, %.4f]\n",
    label, estimate, se, estimate - critical * se, estimate + critical * se
  ))
}

treated_eligibility_share <- mean(df$shipping_eligible[df$treated == 1])
df$eligibility_centered <-
  df$shipping_eligible - treated_eligibility_share
interaction_rhs <- "treated * eligibility_centered + prior_orders + prior_profit + sessions_pre"
df$category_difference <-
  df$eligible_category_profit - df$noneligible_category_profit

elig <- lm(as.formula(paste("eligible_category_profit ~", interaction_rhs)), data = df)
nonelig <- lm(as.formula(paste("noneligible_category_profit ~", interaction_rhs)), data = df)
difference <- lm(as.formula(paste("category_difference ~", interaction_rhs)), data = df)

walk2(
  c("eligible-category profit", "noneligible-category profit", "difference"),
  list(elig, nonelig, difference),
  report_regression
)
#> eligible-category profit: estimate = 1.4938; HC1 SE = 0.0824; 95% Student-t CI = [1.3324, 1.6552]
#> noneligible-category profit: estimate = 1.1267; HC1 SE = 0.0849; 95% Student-t CI = [0.9604, 1.2931]
#> difference: estimate = 0.3671; HC1 SE = 0.1109; 95% Student-t CI = [0.1497, 0.5845]
import pandas as pd
import statsmodels.formula.api as smf
from scipy.stats import t as student_t

df = pd.read_csv("data/free_shipping_banner.csv")

def report_regression(label, model):
    estimate = model.params["treated"]
    se = model.bse["treated"]
    critical = student_t.ppf(0.975, model.df_resid)
    print(
        f"{label}: estimate = {estimate:.4f}; HC1 SE = {se:.4f}; "
        f"95% Student-t CI = [{estimate - critical * se:.4f}, "
        f"{estimate + critical * se:.4f}]"
    )

treated_eligibility_share = df.loc[
    df.treated == 1, "shipping_eligible"
].mean()
df["eligibility_centered"] = (
    df["shipping_eligible"] - treated_eligibility_share
)
interaction_rhs = "treated * eligibility_centered + prior_orders + prior_profit + sessions_pre"
df["category_difference"] = (
    df["eligible_category_profit"] - df["noneligible_category_profit"]
)

elig = smf.ols(f"eligible_category_profit ~ {interaction_rhs}", df).fit(cov_type="HC1")
nonelig = smf.ols(f"noneligible_category_profit ~ {interaction_rhs}", df).fit(cov_type="HC1")
difference = smf.ols(f"category_difference ~ {interaction_rhs}", df).fit(cov_type="HC1")

for name, model in [("eligible-category profit", elig), ("noneligible-category profit", nonelig),
                    ("difference", difference)]:
    report_regression(name, model)
# eligible-category profit: estimate = 1.4938; HC1 SE = 0.0824; 95% Student-t CI = [1.3324, 1.6552]
# noneligible-category profit: estimate = 1.1267; HC1 SE = 0.0849; 95% Student-t CI = [0.9604, 1.2931]
# difference: estimate = 0.3671; HC1 SE = 0.1109; 95% Student-t CI = [0.1497, 0.5845]

The category comparison can still reflect targeting. Users with stronger shopping intent may spend more in both categories, but by different amounts. If the model does not fully account for shopping intent, that bias will not cancel when we subtract the two adjusted profit gaps between treated and untreated users. The larger gap in eligible categories may therefore reflect both the banner and who was selected to see it.

13.7.3 Triangulation

Triangulation brings together evidence that addresses the same causal question under meaningfully different identifying assumptions (Lawlor et al. 2016). Its value comes from approaches with different important sources of potential bias. The placebos, alternative specifications, and sensitivity analysis above remain complementary checks of one observational design: all can share the problem of unobserved shopping intent. A second design can use some of the same data, but changing the estimator alone does not supply a new identification argument.

For the banner, three additional designs could help, depending on the assignment rules and data available. These are possibilities to investigate, not results obtained in this chapter.

RDD, if targeting uses a real cutoff. Suppose the system assigns the banner using a pre-treatment score with a documented cutoff. Compare 30-day profit for users just above and below it. A causal interpretation requires expected profit with and without the banner to vary smoothly through the cutoff, no precise sorting across it, and no other offer changing there (Section 8.2). If exposure only becomes more likely at the cutoff, a fuzzy RDD needs the additional instrumental-variable assumptions discussed in Section 7.4. Its effect then concerns users at the cutoff whose exposure changes because of the rule. Even a credible RDD would not automatically identify the effect for all targeted users (Section 8.1.1). The current simulation has smooth exposure probabilities, so it supplies no such cutoff.

DiD, if suitable user histories are available. Suppose we observe the same users’ profit over comparable periods before and after the banner launch, with records of when exposure begins. Compare profit changes for users receiving the banner with changes for a group that remains unexposed. The key requirement is parallel trends: without the banner, the groups’ average profit would have changed similarly. The comparison also requires no anticipation, spillovers into the comparison group, or simultaneous changes that affect only one group (Chapter 9). DiD can allow persistent differences in profit levels; it does not remove confounding that changes over time. More pre-launch periods help assess plausibility without proving parallel trends. The existing pre/post columns alone do not justify that assumption; hidden intent influences the simulated periods differently.

A randomized holdout, if the business can run one. Randomly withhold the banner from some users whom the targeting rule would otherwise select, while the rest continue under the policy. Comparing 30-day profit by assigned group would estimate the effect of offering the banner under that policy for the population randomized. Check allocation, outcome measurement, and spillovers (Section 4.2.5, Section 4.1.1). If some assigned users never see the banner, distinguish the effect of assignment from the effect of actual exposure. To inform expansion, include currently untargeted users in the randomization; a trial confined to current targets does not establish the effect on those other users.

Compare the questions before comparing the estimates. Keep the banner policy, profit measure, and follow-up window comparable, and state which users each estimate represents. If a credible RDD and DiD both supported a positive effect, they would strengthen evidence that the banner helps within the populations studied, while leaving its magnitude elsewhere unresolved. A larger regression estimate could still reflect the targeting bias revealed above. Compare effect sizes and uncertainty; a statistically significant estimate and an imprecise estimate are not automatically in conflict. If findings disagree, investigate differences in populations, timing, implementation, and identifying assumptions. Agreement does not make the original biased regression a valid estimate for all targeted users.

If a well-executed randomized holdout directly answers the same policy question with adequate precision, use it as the primary evidence. Observational analyses can still help investigate differences across users or periods, provided their assumptions are defensible. Choose the conclusion according to the credibility and relevance of the evidence, rather than counting positive estimates.

Key takeaway. The adjusted profit difference is R$0.37 [R$0.15, R$0.58] larger in eligible than in noneligible categories. The negative-control association remains R$1.13 [R$0.96, R$1.29]. The earlier regional robustness check gives an estimate range of R$1.72–R$1.95 (Table 13.3). These are complementary checks within one observational design. RDD, DiD, or a randomized holdout could add evidence under different identifying assumptions, if the assignment rules, data, and design assumptions support them. None has been carried out here.

13.8 Putting it together: the credibility memo

Summarize the checks in a credibility memo: one page covering the estimate, assumptions, falsification attempts, remaining threats, and decision. The protocol follows the checklist discipline used throughout the book (Section 10.7) and in the placebo literature (Eggers et al. 2024):

  1. Name the assumption, and the direction the bias would take if it fails. “Treated and control users are comparable after I adjust for pre-treatment behavior” is the banner’s assumption; if it fails because high-intent users were targeted, the bias is upward. Knowing the sign before you test tells you to look for positive adjusted gaps in pre-treatment profit or a negative-control outcome.
  2. Pick the placebo. Alter the outcome, the treatment, or the population so that the true effect is zero by construction — a pre-treatment outcome, a category the mechanism cannot reach, a group for whom the treatment has no effect. A placebo is informative only when you can defend the expectation that its effect “should be zero”.
  3. Verify the no-effect-by-construction condition, then preserve the relevant analysis. Reuse the estimator, controls, and uncertainty procedure when that keeps the main and placebo assumptions aligned, and make any test-specific change needed for a valid placebo. Preserve the parts of the analysis that reproduce the suspected problem; removing them changes the design being tested. Mechanical duplication is not the goal.
  4. Plan power and judge observed precision separately. Before running the check, assess its ability to detect the relevant departure or establish equivalence within justified bounds. Afterward, compare the confidence interval with the pre-specified materiality band, as illustrated in Figure 13.1. Including zero does not establish negligibility, and excluding zero does not establish materiality. If no defensible band was defined in advance, report the estimate and interval and leave materiality unresolved.
  5. Run robustness one defensible change at a time. Swap the estimator, the control set, the sample — but change one thing per specification, and report the results specification by specification. Report the differences between specifications rather than combining their estimates into an average.
  6. Run a sensitivity analysis for possible unobserved confounding. Placebos can reveal signs of hidden confounding, but they cannot establish that it is absent. Sensitivity analysis asks how strong a hidden driver would have to be to overturn the result, and whether a confounder that strong is plausible here (Cinelli and Hazlett 2020).
  7. Investigate competing causes and mechanism specificity. List other changes that coincided with treatment, and compare outcomes where the mechanism can and cannot act (Lipsitch et al. 2010).
  8. Pre-specify, and report what each result does and does not rule out. Choose the checks before seeing their results, then report what each finding means: a placebo estimate clearly inconsistent with zero is evidence against the prediction that the placebo effect is zero; a near-zero estimate offers limited reassurance only if its interval excludes a material violation, but does not prove that the assumptions hold.
Figure 13.9: Three analysis stages: the treated–untreated profit difference falls from R$7.86 to the initial adjusted R$2.19 and then to an estimate of R$1.87 from the interaction model, but remains above the targeted-user ATT truth R$0.99.

Each step can challenge an implication, but none can certify the full design. Figure 13.9 summarizes the progression from the raw comparison to the estimate from the interaction model. The horizontal axis shows the treated–untreated profit difference (in R$), and the vertical axis lists the raw comparison, initial adjusted model, and interaction model. Core controls cut the raw R$7.86 to the initial adjusted R$2.19, leaving a R$1.20 gap from the targeted-user truth. Adding shipping eligibility and its interaction with treatment lowers the estimate to R$1.87 and the gap to R$0.88. The change is informative, but the falsification checks and other diagnostics using the interaction model, together with the sensitivity benchmark, still challenge a causal interpretation.

Put the results into the credibility memo in Table 13.4. It gives stakeholders a one-page summary they can read without rerunning the code. Each row uses the numbers from the checks above to answer a question a careful reader would ask.

Table 13.4: The credibility memo for the free-shipping banner summarizes the estimate and its target population, identifying assumptions, falsification, robustness, and sensitivity results, remaining threats, and recommended decision.
Memo line What the evidence says
Estimate and estimand Estimate from the interaction model for the targeted-user population: R$1.87 [R$1.60, R$2.13]. This regression estimate targets the treated-user population but does not identify the ATT. The targeted-user ATT truth is R$0.99 in this simulation; the raw comparison is R$7.86 and the initial adjusted result is R$2.19.
Identifying assumptions A causal interpretation of the adjusted profit difference for targeted users requires a well-defined and consistently measured exposure, no spillovers, conditional comparability after the eligibility interaction and core controls, adequate overlap over the treated-population support, and correctly specified outcome and treatment-effect models. In this simulation, hidden shopping_intent violates comparability and biases the regression estimate upward. In an applied analysis, unobserved shopping intent would remain a possible source of bias.
Falsification results using the interaction model Placebo in time: R$1.60 [R$1.33, R$1.88]. Negative-control association: R$1.13 [R$0.96, R$1.29]. Score-preserving fake exposure: R$1.38 [R$1.02, R$1.73]. Each interval excludes zero, contradicting the prediction of no placebo effect under the corresponding assumptions.
Robustness results The estimates from the interaction model, the model with additional covariates, and trimming stay near R$1.87. Adjustment for the targeting score, IPW, and matching give estimates between R$1.08 and R$1.22. The estimate changes when the analysis uses the targeting process more directly. The IPW estimate remains 23% above the simulated targeted-user truth of R$0.99, and both tested placebo estimates have intervals that exclude zero.
Sensitivity result Using classical OLS inference for the interaction model, a confounder with a partial \(R^2\) of 7.7% with both residual treatment and residual outcome could reduce the estimate to zero (RV 0.077); 6.7% could make it nonsignificant at the 5% level (RV 0.067). The treatment term has a partial \(R^2\) of 0.0065 with the outcome, so it is associated with 0.65% of residual outcome variation after the other model terms. In this simulation, adjustment for the actual hidden factor gives R$0.91 [R$0.64, R$1.17]. A separate worst-case benchmark based on prior_profit gives R$-1.19 [R$-1.44, R$-0.94]. Neither number is a predicted correction.
Remaining threats Known in this simulation: hidden shopping_intent. In an applied analysis: unmeasured intent, functional-form or measurement error, and the inability of a targeted-user ATT to identify effects for currently untreated users.
Recommendation Do not use this regression estimate to authorize whole-base rollout. Check whether a documented targeting cutoff supports RDD, suitable user histories support DiD, or a randomized holdout is feasible. Assess each design’s assumptions and match its population to the decision: continuing today’s targeting policy or extending the banner to currently untargeted users.

The memo does not say the banner has no effect. In the simulation, the targeted-user truth is R$0.99, the eligible-targeted truth is R$1.50, and the whole-base truth is R$0.90. It says the estimate from the interaction model remains too vulnerable for a confident causal claim. Centering eligibility at its share among targeted users makes the treatment coefficient a weighted average of the model’s adjusted treated–untreated differences in mean profit for eligible and ineligible users. This does not identify the ATT or tell us what would happen if the banner were shown to currently untreated users.

13.9 Wrapping up and next steps

Assess an estimate’s credibility by stating what its assumptions predict and reporting what the checks found. An estimate that has not faced those checks still needs evidence to support its causal interpretation.

Apply the chapter’s checks in this order:

  1. Define the estimand, target population, decision, and identifying assumption before you inspect the diagnostics.
  2. Pre-specify placebo checks, their no-effect-by-construction logic, and the smallest departure from zero that would change the decision.
  3. Keep the target estimand explicit. Preserve the design logic relevant to each check and assess its power. Then vary one defensible choice at a time and benchmark hidden confounding.
  4. Put the estimate, contradicted implications, reassuring findings, unresolved threats, and next action into a one-page credibility memo.

The next step is to investigate which additional design the business can credibly support: RDD around a real targeting cutoff, DiD with suitable user histories and defensible parallel trends, or a randomized holdout when feasible. Compare what each design identifies, for whom, and under which assumptions. Evidence about current targets or users near a cutoff does not, by itself, establish the effect of extending the banner to currently untargeted users.

TipFurther practice

In the R companion script, profit_intent controls how strongly hidden shopping intent affects 30-day profit. Run the simulation with its current value of 4.20, then repeat with 8.40, keeping the random seed and every other setting unchanged. Save each simulated dataset to a separate file and point the R or Python analyses at those copies, leaving the chapter’s original data untouched. For each version, compare the interaction model’s estimated banner effect with the known average effect among targeted users, and calculate the Robustness Value. Does a larger Robustness Value necessarily mean that the estimate is closer to the truth? Explain your answer using the two simulations.

The chapter’s checks connect to two earlier chapters. The DAG in Section 6.4.2 can imply observable restrictions, but each statistical check also carries modeling assumptions. Balance and planning (Table 3.1, Section 3.4.3) apply the same skeptical habit before estimation. Once the diagnostic findings support using an estimate for the decision, the next chapter asks what it is worth: what the effect becomes after you discount it for decay, adoption, and the cost of implementing the change.


  1. As Google advises, “NotebookLM can be inaccurate; please double-check its content”. I recommend reading the chapter first, then listening to the audio to reinforce what you’ve learned.↩︎

  2. To load it directly, use the example URL https://raw.githubusercontent.com/RobsonTigre/everyday-ci/main/data/free_shipping_banner.csv.↩︎

  3. All rows use 95% intervals to match the chapter’s reporting convention. In the usual two one-sided tests (TOST) procedure at the 5% significance level, equivalence is assessed using the corresponding 90% confidence interval. That interval must lie strictly inside the pre-specified bounds. Requiring a 95% interval to lie inside the same bounds is stricter. This figure illustrates the stricter interval rule; it does not report conventional 5% TOST results (Lakens 2017).↩︎

  4. Keeping prior_profit asks whether the adjustment used in the main analysis leaves a pre-treatment difference. Correlation with profit_pre alone is not a reason to remove the control. In real applications however, first establish the measurement periods. The control must be justified for this comparison, and the placebo must plausibly retain information about the suspected confounding after adjustment. Report the unadjusted gap alongside the adjusted gap, and do not treat a small adjusted gap as proof that hidden confounding is absent. See Imbens (2015) and Eggers et al. (2024).↩︎

  5. This book deliberately stops short of teaching propensity-score matching as a complete method. Its causal interpretation requires conditional exchangeability given the observed covariates and adequate overlap. Proper implementation also requires decisions about the covariates and treatment model used to estimate the score; the matching algorithm, caliper, and replacement; the target estimand; balance and overlap diagnostics; outcome analysis and uncertainty; and sensitivity to hidden bias. The book has not developed the treatment-modeling and design-validation methods needed to cover those choices responsibly, so estimating a score and calling a matching function is not enough. Because the method remains widely used, the example below introduces the workflow without serving as an implementation template. For practical guidance, see Caliendo and Kopeinig (2008), Stuart (2010), and Austin (2011).↩︎

  6. The calculation is implemented in sensemakr in R and PySensemakr in Python (Cinelli, Ferwerda, et al. 2024; LaPierre 2024). The Python package installs as PySensemakr but imports as sensemakr. Its own plot() helper breaks on matplotlib 3.10 and later, so the Python companion script draws the contour from the package’s bias functions instead; R’s plot(sens) still works.↩︎

  7. A SKU identifies one sellable product variant, such as a blue running shoe in size 42; size 43 is another. Prices and stock are set at this level, so a change to one popular SKU can affect category sales.↩︎