12  Heterogeneous treatment effects: Different people, different reactions

Published

December 27, 2025

Modified

November 3, 2026

Keywords

A/B Testing, Causal Inference, Causal Time Series Analysis, Data Science, Difference-in-Differences, Directed Acyclic Graphs, Econometrics, Impact Evaluation, Instrumental Variables, Heterogeneous Treatment Effects, Potential Outcomes, Power Analysis, Sample Size Calculation, Python and R Programming, Randomized Experiments, Regression Discontinuity, Treatment Effects

Get news about the book. Leave your email here.
Found an error or have a suggestion? Submit feedback here.
Follow the book’s page for new content and follow the author.

Listen to a podcast version of this chapter, generated by Google’s NotebookLM.1

12.1 When the average hides the decision

A few years ago a doctor prescribed me metoclopramide, a common anti-nausea drug. For most people it does its job, but it left me extremely restless and worse off, a rare but known adverse reaction. The clinical trials are not wrong: on average, the drug probably helps many people. That average does not tell me whether I will be one of them.

That gap between the average and the individual is the subject of this chapter. Up to now we have focused on estimating one number as credibly as possible: the average treatment effect. That number can answer the general “did it work?” — but not the question that drives the business decision: for whom did it work, and what do we do about it?

Imagine a marketing team with a fixed budget for discount coupon campaigns. They can send a discount to only some of their users. Sending a coupon to a customer who would have bought anyway wastes margin. Sending one to a customer who will not convert wastes the coupon too. The extra profit is in finding the users whose behavior actually changes because of the discount — and spending the limited budget on them. DoorDash, for instance, reports using causal inference to “estimate the true incremental effect of a given promotion on each user” and then choosing whom to treat under a fixed budget (Xu et al. 2025).

To make this concrete, I will use one simulated dataset throughout the chapter. Say a company ran a clean experiment that randomly gave a R$5 coupon to half of 20,000 users (D = 1) and nothing to the other half (D = 0).2 Then it measured each user’s observed_profit: the profit they generated over the following weeks, recorded the same way for everyone in the experiment, treated or not.

For every user we also observe four features built from their purchase history, all measured before the experiment started: recency (how long since their last purchase, 0 = just bought, 1 = long lapsed), frequency (how many times they have bought), past_spend, and disc_sens (a discount sensitivity score in \([0, 1]\)).3 Because we simulated the data, we know the true effect for each user, stored in the dataset as true_cate, and can compare our estimates with it. In observed data, we do not know each user’s true effect.

NoteVocabulary

This chapter uses machine-learning (ML) tools, so we first need to match a few ML terms to the terminology used earlier in the book:

  • feature = covariate / control variable / regressor (the predictors \(X\) — here recency, frequency, disc_sens, and past_spend)
  • train / fit a learner = estimate a model
  • (base) learner = model / estimator
  • prediction = fitted value / estimate

The outcome (\(Y\), here observed_profit) keeps its name throughout — I avoid the ML labels target and label for it. The name is deliberate: observed_profit is the profit we observe for every user, treated or not — not the coupon’s “incremental profit” (its uplift), which is the causal effect the rest of the chapter sets out to estimate. Terms with no earlier-chapter equivalent (honesty / sample splitting, out-of-bag, and nuisance functions) are defined where they first appear.

Suppose you run the experiment, estimate the average treatment effect, and obtain the results in Table 12.1:

Table 12.1: Results from the coupon experiment. The average treatment effect is statistically significant but small, even though the coupon is profitable for a minority of users.
Quantity Value
Average treatment effect (difference in means) R$0.30 per user
95% confidence interval [R$0.15, R$0.46]
Share of users whose profit increase exceeds the coupon cost about 11%

The experiment estimates a statistically significant increase of R$0.30 in profit per user. That average could lead the company to cancel the program, even though it answers the wrong question for a targeted campaign. A minority of users generates roughly R$7 each in incremental profit, while the coupon produces a small loss for the majority. Averaging the two groups makes the overall effect look small.

Every effect in this table — and in the rest of the chapter — is measured on profit before subtracting the R$5 the coupon itself costs. For most users the effect is already slightly negative: sending them a coupon lowers their profit by about R$1. A small group responds strongly enough to be worth treating, and that is what “pays off” means in the last row: the coupon lifts a user’s profit by more than the R$5 it costs, not merely by more than zero.

This chapter asks who benefits and by how much, so the budget can be directed toward the 11% of users whose response covers the coupon cost. The methods that follow estimate effects that vary across users, but they still depend on the assumptions that make those effects causal.

Data: All data we’ll use are available in the repository. You can download it there or load it directly by passing the raw URL to read.csv() in R or pd.read_csv() in Python.4

Packages: To run the code yourself, you’ll need a few tools. The block below loads them for you. Quick tip: you only need to install these packages once, but you must load them every time you start a new session in R or Python.

# If you haven't already, run this once to install the packages:
# install.packages(c("tidyverse", "grf", "policytree", "DiagrammeR"))

# You must run the lines below at the start of every new R session.
library(tidyverse)   # data wrangling and plots
library(grf)         # generalized random forests: causal_forest() and friends
library(policytree)  # learned targeting rule (the policy tree in the last section)
# If you haven't already, run this in your terminal to install the packages:
# pip install pandas numpy scipy statsmodels econml scikit-learn mcf (or use "pip3")

# You must run the lines below at the start of every new Python session.
import pandas as pd
import numpy as np
import statsmodels.formula.api as smf            # OLS with a formula interface
import statsmodels.api as sm                     # array-interface OLS for the validation checks
from scipy import stats                          # t-test in the power-problem section
from econml.dml import CausalForestDML           # causal forest estimator
from econml.validate.drtester import DRTester    # RATE / TOC evaluation
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.model_selection import train_test_split   # held-out validation splits
from mcf.optpolicy_functions import OptimalPolicy      # exhaustive policy tree over a candidate split grid

12.2 What we are trying to estimate

Before choosing an algorithm, we need to distinguish three meanings of “the effect of the coupon on each user”, because they differ in what we can estimate.

The effect on this user: the one we cannot have. Recall the potential-outcomes framework from Chapter 1 and the estimands discussion in Section 2.5: each user \(i\) has two potential outcomes, \(Y_i(1)\) if they get the coupon and \(Y_i(0)\) if they do not. The individual treatment effect (ITE) is the difference between the two, \(\tau_i = Y_i(1) - Y_i(0)\): how much this specific user’s profit changes because of the coupon. It is the quantity a business most wants to know, and it is unobservable. Seeing it would require watching the same user live through the same weeks twice, once with the coupon and once without.5 No method in this chapter escapes that, however sophisticated it looks.

The effect on users like this one: the one we can estimate. Take one user from our dataset, call her Maria: lapsed for months, buys often when she is active, high discount sensitivity. We cannot ask “what does the coupon do to Maria?”, but we can ask the next best question: what does the coupon do, on average, to users who look like Maria? That is the conditional average treatment effect, or CATE:

\[ \tau(x) = \mathbb{E}\!\left[\, Y_i(1) - Y_i(0) \mid X_i = x \,\right]. \]

Read it from right to left: gather the users whose features \(X_i\) match the profile \(x\), then average their effects. Conditioning on \(x\) is the formal way to say something intuitive such as “the effect depends on who the user is”. If this sounds like an interaction between treatment and the user’s characteristics, that is the right intuition (Molak 2023). We will begin with interaction models.

A group average is not an individual effect. A CATE is a local average over users similar to Maria. “Users like Maria gain about R$5” does not mean Maria gains R$5: she might gain R$9 while another user with the same observed features gains R$1. For instance, a drug can have a CATE of 0.5 in a group where no individual has an effect of 0.5 — one patient is saved, another is harmed, and the average differs from every patient’s individual effect (Facure 2023). Every per-user number a model gives you describes an average effect for a group of similar users.

So watch the vocabulary too: the literature often calls the CATE the individualized treatment effect, which differs from the individual treatment effect (Molak 2023). When a tool promises “individual-level effects”, read the fine print: it is delivering CATEs.6

Effects for broader groups. A subgroup effect is a CATE defined by a broad version of “like Maria”: the average effect among “iOS users,” or among “users who lapsed more than 60 days ago.” It averages effects within broad groups: useful when you need a rule you can explain in one sentence, but limiting when effects vary within those groups.

Finally, the CATE is a means to an end. The experiment’s ATE answered the average question — did the coupon move profit at all, before its cost? — and the CATE informs the decision this chapter cares about: whom to treat, given that we cannot treat everyone (Facure 2023). The goal is a targeting rule: estimate \(\tau(x)\) credibly, then turn it into a rule for spending the coupon budget on the users where it pays off.

12.3 Before the algorithms: identification still comes first

WarningMachine learning estimates effects; it does not identify them

The methods in this chapter are more flexible than the OLS used so far, but flexibility does not replace identification assumptions. Their output has a causal interpretation only when those assumptions hold. Machine learning methods adapted to causal tasks facilitate estimation, but do not ensure the identification of causal effects (Brand et al. 2023). Machine learning does not fix omitted variable bias; it fits a more elaborate model to the variation available in the data.

Three assumptions support a causal interpretation. The first two are familiar to you and are required for analysis based on selection on observables:

  • Unconfoundedness. Conditional on the observed features \(X_i\), treatment is as good as randomly assigned — there is no unmeasured factor that drives both who gets the coupon and how they respond. This is the conditional independence assumption from Section 6.4.2. In our example this is satisfied by design, because the coupon was randomized. That makes an experiment with rich features the most credible setting for heterogeneity analysis. With purely observational data, a causal interpretation depends on whether you have measured all relevant confounders, which is difficult to establish.

    • One study examined 663 large-scale randomized advertising experiments run on Facebook and asked whether state-of-the-art observational methods, using more than 5,000 user-level features, could reproduce the experimental effects. They could not. The verdict was that “despite having access to large-scale experiments and rich user-level data, we are unable to reliably estimate an ad campaign’s causal effect” (Gordon et al. 2023). More data and more complex models did not recover the effects measured through randomization.
  • Positivity (overlap). Every type of user must have a real chance of being in either arm — the common support requirement we met in Section 6.4.3. Formally, the propensity score \(e(x) = P(D=1 \mid X=x)\) must be strictly between 0 and 1 for all \(x\): within every group of lookalike users, treatment can be likely or unlikely, but never guaranteed or impossible (Chernozhukov et al. 2024). If some kind of user is never treated, no method can tell you their effect; it can only extrapolate, and extrapolation is not estimation.

  • No interference. One user’s coupon must not change another user’s outcome. Randomization does not guarantee this assumption; no interference concerns whether users are separate units, not how treatment was assigned. Coupon sharing within a household, referrals, or treated users competing with untreated ones for the same stock would all violate it in a real campaign, and every estimator in this chapter takes its absence for granted. Designs that measure spillovers directly appear in Chapter 15.

A fourth requirement is specific to heterogeneity work, and just as easy to violate: every feature in \(X\) must be “frozen” at the moment of assignment. Our four features qualify because they are drawn from purchase history that predates the experiment. Many tempting additions do not: whether the user redeemed the coupon, their engagement after the campaign, their follow-up purchases, their post-campaign satisfaction. Anything measured after assignment is potentially affected by the coupon, which makes it a mediator or a collider in disguise — the bad controls we met in Section 6.3.

Split users on a variable like that and you stop asking “for whom does the coupon work?” and start asking “among users who already responded one way, how do the numbers look?” Randomization does not protect you here: it guarantees a clean average effect, not a clean \(\tau(x)\) built on post-treatment features.

Whether a variable is a good, bad, or neutral control depends on its position in the causal graph and the estimand, not only on when it was measured. A safe operational rule is the feature freeze: record the feature table at assignment time and exclude variables measured after that moment.

ImportantThe standing assumption for the rest of this chapter

Every method here assumes you are in one of two settings: a randomized experiment, like our coupon example, or observational data where unconfoundedness and overlap can be defended with evidence. If you are in neither, the fix is identification, not a more flexible estimator. Randomization is the cleanest route to a credible CATE, but it is not the only one. Instrumental variables (Chapter 7), regression discontinuity (Chapter 8), and difference-in-differences (Chapter 9 and Chapter 10) each have heterogeneous-effects variants that inherit their design’s assumptions. If your data fit one of those designs, begin with its identification strategy.

12.4 Manual heterogeneity: useful, but fragile

Long before causal forests, analysts looked for heterogeneity by hand. These methods are transparent and often sufficient; their limitations help explain why we might need more flexible methods. The three common options are subgroup means, OLS interactions, and separate regressions by treatment arm.

Subgroup means. Pick a feature you suspect matters, split users into a subgroup and its complement, and compare the treated-minus-control difference within each. In our coupon data, splitting on discount sensitivity gives a hint: users above the median sensitivity show an effect of about R$0.86, those below about −R$0.25. This difference points in the right direction, but it captures only part of the heterogeneity because no single feature determines the true effect.

OLS with interaction terms. The regression model from Section 1.3 extends naturally: add interactions between treatment and the features, then interpret their coefficients. The limitation is that an OLS interaction can estimate only the heterogeneity represented by the interaction terms you specify. If the true effect depends on recency \(\times\) frequency \(\times\) disc_sens together, a model with only pairwise terms will miss it. On our data the interaction model captures part of the heterogeneity (a positive D:disc_sens coefficient of about 2.96, a D:recency of about 4.57, both highly significant), yet its implied CATE correlates only 0.64 with the truth — it captures variation associated with each feature separately but misses the pattern that depends on several features together.

Separate regressions per arm. Fit one model on the treated users and another on the controls. Then, for every user, ask both models the same question — “how much profit would this person bring?” — and subtract the control answer from the treated one. That difference is the user’s estimated CATE, and it varies across users because the models relate profit to each user’s features (recency, discount sensitivity, and the rest).

A bargain-hunter and a loyal big spender receive different estimates, and the variation across users is the estimated heterogeneity; averaging these estimates gives a single ATE estimate. Each treatment arm has its own model, with no requirement that the two models use the same relationship between features and profit. But each model sees only one treatment arm, so in this experiment it learns from half the users. Estimates become less precise when an arm contains few users of a given type or has few users overall. The method is straightforward to implement and leads to the T-learner introduced later.

library(tidyverse)
df <- read.csv("data/coupon_allocation_experiment.csv")

# 1) Subgroup means: split on one feature you happen to suspect
df |>
  mutate(sensitive = disc_sens > median(disc_sens)) |>
  group_by(sensitive) |>
  summarise(ate = mean(observed_profit[D == 1]) -
                  mean(observed_profit[D == 0]))

# 2) OLS interaction: you only find the heterogeneity you remember to specify
ols <- lm(observed_profit ~ D * (recency + frequency + disc_sens + past_spend),
          data = df)
summary(ols)

# 3) Separate regressions, one per arm (a hand-rolled "two model" estimator)
f  <- observed_profit ~ recency + frequency + disc_sens + past_spend
m1 <- lm(f, data = subset(df, D == 1))
m0 <- lm(f, data = subset(df, D == 0))
df$cate_ols <- predict(m1, df) - predict(m0, df)
import pandas as pd
import statsmodels.formula.api as smf

df = pd.read_csv("data/coupon_allocation_experiment.csv")

# 1) Subgroup means: split on one feature you happen to suspect
df["sensitive"] = df["disc_sens"] > df["disc_sens"].median()
for flag, g in df.groupby("sensitive"):
    ate = g.loc[g.D == 1, "observed_profit"].mean() - \
          g.loc[g.D == 0, "observed_profit"].mean()
    print(flag, round(ate, 2))

# 2) OLS interaction: you only find the heterogeneity you remember to specify
ols = smf.ols("observed_profit ~ D * (recency + frequency + disc_sens + past_spend)",
              data=df).fit()
print(ols.summary())

# 3) Separate regressions, one per arm (a hand-rolled "two model" estimator)
feat = "recency + frequency + disc_sens + past_spend"
m1 = smf.ols(f"observed_profit ~ {feat}", data=df[df.D == 1]).fit()
m0 = smf.ols(f"observed_profit ~ {feat}", data=df[df.D == 0]).fit()
df["cate_ols"] = m1.predict(df) - m0.predict(df)

All three methods share a risk regardless of which features you chose. A subgroup or interaction specified before examining the data can have an ordinary confidence interval for that planned question. But if you compare many subgroups and report only those with apparently unusual effects, you increase the risk of reporting false positives. Test twenty subgroups at the 5% level and, even if the coupon helped no one differently, you expect one “significant” subgroup by luck alone.

This is the multiple testing problem we met in Section 5.5.2: searching many subgroups or metrics inflates the chance of a spurious finding unless the search is planned or evaluated on independent data. Dividing a continuous feature like recency into arbitrary bins makes the problem worse: different cutpoints can produce different estimated patterns, and it is tempting to keep the one that confirms your hypothesis.

Manual methods can miss real heterogeneity or produce apparent differences due to noise. If you discover a subgroup after examining the results but report it as if it had been specified in advance, you are HARKing (hypothesizing after results are known). We need an approach that searches for variation systematically and then lets us check whether the variation is real.

12.5 From manual rules to meta-learners

The first systematic approach is to let a flexible model learn patterns in the data instead of specifying interactions by hand. Meta-learners are recipes that turn any off-the-shelf predictor (a gradient boosting model, a random forest, even a regression) into a CATE estimator by combining outcome predictions (Künzel et al. 2019). Two common versions illustrate the trade-off between using all users to fit one model and fitting a separate model for each treatment arm.

The S-learner (“S” for single) fits a single model for the outcome, using all the features \(X\) together with the treatment indicator \(D\). Once we have that model, we predict each user’s outcome twice: once with the treatment turned on (\(D=1\)) and once with it turned off (\(D=0\)). The difference between those two predictions is our estimate of the treatment effect for that user.

It reuses all the data efficiently, but it has an important limitation. When treatment is one feature among many and its contribution to outcome prediction is small, regularization can shrink its influence toward zero and obscure a real effect. The S-learner can therefore understate both the average effect and its heterogeneity.7

The T-learner (“T” for two) fits two models, one per arm. It generalizes the separate-regressions idea from above to any learner and subtracts the two models’ predictions. Because each arm gets its own model, the relationship between features and outcomes can differ between treated and untreated users. But fitting the models separately increases variance: if one arm contains few users of a given type, or the treated group is small overall, the model has little data to learn from and the difference between predictions becomes less precise.

Both use the potential outcomes framework to predict what a user’s outcome \(Y\) would have been under the other treatment condition. They share the same limitation: their CATE estimates are differences between outcome predictions. If the model misses a response pattern, the estimated heterogeneity misses it too. If the model fits random variation in the sample, that variation can appear as treatment-effect heterogeneity. Neither recipe by itself distinguishes real effect differences from random noise.

A flexible learner can search many patterns in the data: which features interact, where to split a continuous feature, how to rank users, and which response patterns to fit. That flexibility also creates a risk of overfitting: the learner can mistake random noise for signal, making a group of users look unusually responsive because of chance variation in this sample.

A further problem arises when the same data are used both to find a pattern and to estimate its size. If a subgroup was selected because it looked unusually responsive, its estimated effect includes the random variation that made it stand out. Measuring the effect on those same users can therefore overstate the subgroup’s response.

When the same dataset is used to discover a subgroup, ranking, or partition and then estimate its effect, an ordinary confidence interval ignores the uncertainty introduced by that search. It treats the chosen segment as if you had specified it in advance. Accounting for the selection when quantifying uncertainty is the problem of post-selection inference.

Other meta-learners, such as the X-learner and DR-learner, refine these ideas; Molak (2023) provides an accessible overview. The remaining problem is how to search for heterogeneity without overfitting while retaining valid uncertainty estimates. The fix has to separate discovery from measurement, or use methods that reproduce that separation.

Maria is a lapsed user who buys often when she is active and has high discount sensitivity. We want to identify “users like Maria” without choosing arbitrary cutoffs for recency, frequency, and discount sensitivity. We also need uncertainty estimates that account for having discovered the profile in the data. We therefore use a method that uses different observations to discover the group and to estimate its effect.

12.6 Why trees and forests help

Trees help with that task because they partition users into groups and estimate a quantity within each group. For the CATE, that means gathering comparable users and estimating the coupon effect for that profile. The subsections that follow build the method up one idea at a time:

  1. how a single tree carves the data into groups,
  2. how methods built for predicting outcomes can be adapted to estimate causal effects,
  3. how honesty uses separate observations to choose the groups and estimate their effects, and
  4. why a forest gives more stable estimates than a single tree.

The four subsections use words and pictures. If you want the same argument in notation — what each method estimates, what we cannot observe, and how the splitting rule accounts for that missing quantity — Appendix 12.A walks through it.

12.6.1 How a single decision tree is built

Let’s set “treatment” aside for a moment, since trees were built for prediction, not for estimating treatment effects. An ordinary decision tree predicts an outcome we can observe, say how much profit a user brings in over the coming weeks. It asks yes/no questions about the features (for instance, is recency at or below 0.50? Has the user bought at least three times?) and splits the group after each answer until every user lands in a leaf. In this predictive-tree example, the leaves separate high-profit from low-profit users.

The process is recursive: it splits the full dataset once, then asks the same kind of question inside each resulting group. Each leaf holds a single prediction, the average outcome for the users who landed there (in Figure 12.1, their average profit). The result is a flowchart whose successive questions divide users into increasingly similar groups.

Two questions define the whole tree: Where to split? At each step the tree scans every feature and every threshold and keeps the split that makes users within each resulting group most similar in profit — the one that best separates high-profit from low-profit users. This is a greedy search: it chooses the best split at the current step.

And when to stop splitting? Left unchecked, a tree keeps splitting until each leaf holds a single user, memorizing noise instead of signal. We hold it back with a few constraints:

  1. a maximum depth, which caps how many times the tree can split in sequence, so it can’t keep branching indefinitely;
  2. a minimum number of users per leaf, which forces every group to be large enough that its estimate reflects a real pattern rather than a handful of unusual users; and
  3. a minimum improvement required to justify a split, which prevents further splitting unless prediction improves by at least that amount.

Together, these constraints balance two risks: a tree too coarse to capture real structure and one so fine that it overfits. They make the overfitting risk from the previous section explicit and manageable.

Figure 12.1 shows how successive splits build the tree, then presents the finished tree as if/then rules.

Figure 12.1: How a single decision tree is built. Panel A carves the recency × frequency plane one cut at a time in four steps: everyone starts in a single group; the tree splits on the feature that makes users within each box most alike (here, separating higher-profit from lower-profit users); it splits those pieces again; and it labels each final box with the average profit of the users inside. Panel B shows that same finished tree as a rule chart. Each internal node (gray boxes) corresponds to one cut line in Panel A, and each leaf (red-ish boxes) reports the average profit of the users who reach it. Color marks the profit level on a neutral scale. Note: This illustrative tree predicts an outcome (profit), not a treatment effect. A causal tree reuses this partitioning machinery and changes only what each box estimates: the coupon’s effect on profit, not the profit level.

12.6.2 The causal tree: the difference between predicting outcomes and predicting treatment effects

The predictive trees described above use an observed outcome as the label for each user in the training data: the profit they earned, whether they churned, or another outcome we want to predict.

A causal tree needs a different label. The label we want is not the profit a user generated, but how much the coupon changed it: that user’s own incremental response, the counterfactual difference between the profit they’d generate with the coupon and the profit they’d generate without it.

We cannot observe that label. Each user either gets the coupon or doesn’t, never both, so for any single person we observe one side of that difference and never the other. This is the fundamental problem of causal inference: we cannot train a machine-learning model directly on individual treatment effects as we can on observed outcomes. We therefore adapt the tree.

A causal tree is that adaptation. It keeps a similar structure to what we saw in Figure 12.1 but changes the question it asks at every split. The prediction tree looks for cuts that best predict the outcome: how much profit each user generates. The causal tree looks for cuts that best separate users with different treatment effects: those for whom the coupon changes profit by different amounts.

But how can the causal tree separate users by an effect it never observes? It does not start by knowing the groups. It proposes candidate groups using ordinary feature rules: recency above this cutoff, frequency below that cutoff, and so on. For each candidate split, it asks: among the users who would land on each side of this rule, what is the treated-minus-control gap in observed profit?

In our randomized trial, every user has the same chance of receiving the coupon, so that gap estimates the average effect for a fixed candidate group. With observational data, treated users can still differ from controls inside a group, even if treatment is as good as random after accounting for all the features. The comparison may therefore need adjustment within the group. The tree keeps the feature split that makes the group-level effect estimates most different, subject to the usual minimum-size and stopping rules we discussed in the previous subsection. No user gets an observed effect label; candidate groups get scored by estimated effects, and the best-scoring feature rules then define the tree.

In the coupon example, a leaf of the tree would no longer contain “users with predicted profit around R$X over the campaign.” It would gather “users for whom the treated-minus-control coupon effect is around R$Y.” Both tree types divide users into groups. The prediction tree groups users by the profit they generate, the outcome we observe directly. The causal tree groups them by how much the coupon changes that profit.

Mechanically, the tree looks for splits that make the estimated treatment effect as different as possible across leaves. In plain language, it asks: “Where can I cut the data so that the effect in one group looks clearly different from the effect in another?”8 This direct focus on treatment-effect differences fixes one limitation of the S- and T-learners from the last section. Those methods estimate effects indirectly: first predict the outcome with and without the coupon, then subtract the two predictions. As discussed earlier, a regularized S-learner can shrink a real effect toward zero, and a T-learner produces noisy estimates wherever one arm has little data.

12.6.3 Honest causal trees: choosing and measuring on different data

A causal tree does two jobs: first it finds a promising subgroup, then it measures the effect within it. If both use the same rows of data, the estimate includes the noise that made the subgroup look unusually responsive during the search. Selecting the subgroup for its apparent response therefore inflates the estimated effect. This is the post-selection-inference problem described earlier.

The fix is honesty, also called sample splitting. We split the data in two: one part is used by the tree to choose the splits, and the other part is used to estimate the treatment effect inside each leaf (Athey and Imbens 2016). Despite the name, “honesty” here is not about ethics. It is what makes the uncertainty numbers — the standard errors and confidence intervals — trustworthy. Because we no longer estimate a leaf’s effect on the rows that made it look promising, honesty reduces the false-positive risk described in the manual section.

Figure 12.2 illustrates sample splitting without the coupon-specific variables: read Y as the observed outcome, D as the treatment assignment, and the X columns as user features such as recency and frequency. The method randomly assigns each row either to choosing the leaves (yellow cells) or to estimating their effects (orange ones), so the two halves differ only by chance. The held-out sample estimates effects for the subgroups defined by the splitting sample, using different users whose outcomes did not determine the boundaries.

Figure 12.2: Honesty as sample splitting, shown in the abstract. Panel A splits the data at random into a structure half (yellow) and an estimate half (orange). Panel B uses only the structure half to choose the tree: which feature rules define the leaves. Panel C passes the estimate half through those fixed leaves to measure the treated-minus-control effect inside each one, so the same observations do not both discover a promising subgroup and estimate its treatment effect.

Figure 12.3 runs that same structure/estimate split on our coupon data. The structure sample draws four boxes; the estimate sample then measures the treated-minus-control gap inside those fixed boxes. The lapsed, frequent buyers — recency past about 0.6, three or more purchases — are the only group with an estimated effect around the R$5 coupon cost, while the other three groups sit near −R$1. The honest tree separates the search for a promising subgroup from the measurement of that subgroup’s effect. The estimate is still uncertain and still needs the validation checks that come next, but it is less affected by the noise involved in selecting the subgroup.

Figure 12.3: Honesty on the coupon data. The left panel chooses the four boxes on the structure sample; the right panel colors each fixed box by the coupon effect measured on the held-out estimate sample (treated minus control). Lapsed, frequent buyers are the only group near the R$5 coupon-cost threshold (about R$5.4, blue); the other three groups sit near −R$1 (orange). The R$5.4 estimate remains uncertain, but using different rows to select the subgroup and estimate its effect reduces contamination from adaptive search.

12.6.4 From one tree to a forest

A single tree is unstable, even when it is honest. Small changes in the data can move a threshold from 0.6 to 0.55 or reorganize a whole branch. The solution is the one ordinary prediction already uses: grow many trees and average them. A random forest grows hundreds of decision trees, each on its own random draw of the data, then averages their predictions so that no single noisy partition dominates. Classical random forests draw those samples with replacement — that is bagging, short for bootstrap aggregating. The causal forests in this chapter draw theirs without replacement instead, a choice that matters for the confidence intervals we get to later.

But averaging only helps when the trees make different mistakes. If every tree were trained on the same customers and allowed to use the same features at every split, they would tend to draw the same boundaries and repeat the same errors. A random forest avoids that by giving each tree a slightly different view of the coupon data: each tree grows on a different subsample, and each split gets only a random subset of candidate features.

The schematic in Figure 12.4 shows estimated coupon effects across recency and frequency, with slightly different boundaries for each tree. Averaging many trees (the bottom row, right-most panel) makes those boundaries less prominent and shows a clearer pattern of larger coupon effects among lapsed, frequent buyers.

Figure 12.4: From one tree to a forest. Each of the first five panels is one causal tree’s map of the estimated coupon effect across recency × frequency (blue = a large positive effect, near-white = around zero, orange = a negative effect). Grown on a different subsample with a different random subset of features, each tree drops its boundaries somewhere different and is individually blocky and noisy. The last panel averages many such decorrelated trees: the noise cancels and the smooth underlying pattern — a clear payoff for lapsed, frequent buyers — emerges. (Schematic, not fit to the coupon data; the real forest’s output appears in the figures that follow.)

A causal forest grows many honest causal trees on different subsamples instead of asking one tree to decide who looks like Maria. Each tree puts Maria in a leaf with the users its rules consider similar. The original causal forest averages the trees’ effect estimates. The generalized random forest used here combines their comparison neighborhoods into weights, then uses those weights to estimate her effect. Combining information across trees makes the estimate less sensitive to any one set of splits; honesty keeps each tree from using the same outcomes to both choose its splits and estimate effects.

The first benefit is out-of-bag prediction: Maria’s CATE is estimated using only the trees that did not use her data while growing. In practice, use the estimator’s out-of-bag or held-out prediction mode for this step. The methodological requirement is that the CATE used to score a user should come from trees that did not train on that same user.

The second payoff is uncertainty you can use. With enough honest trees, the estimate for a profile like Maria’s stops depending so much on any one random split. It concentrates around the CATE for users with that profile, and its sampling variation becomes regular enough to support confidence intervals. That is the practical meaning of Wager and Athey (2018): the interval is about the true CATE for users like Maria, not Maria’s unknowable personal response. Generalized random forests extend the same idea beyond treatment-effect estimation (Athey et al. 2019). For targeting, this matters because a coupon rule based on “R$5, plus or minus R$1” is a different decision from one based on “R$5, plus or minus R$8.” The interval does not make the decision by itself; the validation checks in the next section supply the rest.

A forest can return different estimates for different feature profiles even though each tree groups users into leaves. Maria’s personal-looking estimate comes from where she lands across the trees. Each tree grew on a different subsample with different split rules, so each tree groups Maria with a somewhat different set of users. A user who shares a leaf with Maria in most trees is someone the forest keeps judging similar to her; a user who almost never shares her leaf is someone the forest keeps judging different. Each tree used for Maria’s prediction divides a total weight of one equally among the users in her leaf whose outcomes were set aside for estimation. Other users get zero weight from that tree. Averaging each user’s weights across those trees gives their final weight: how much their data counts when estimating Maria’s CATE (Athey et al. 2019).

The heavily weighted users form Maria’s comparison neighborhood — the forest’s data-driven answer to “who counts as a user like Maria?” The generalized random forest uses those weights to estimate her CATE from the residualized outcome and treatment. These are the parts left after subtracting what the features predict (Section 12.6.5). Different feature profiles can lead to different comparison neighborhoods and effect estimates. Users can also share a neighborhood and receive the same estimate. Note what the forest is not doing: it is not reading a personal effect off Maria’s own row. It is running the CATE recipe from the start of the chapter:

  1. gather comparable users,
  2. estimate the average coupon effect for that profile, and
  3. let the forest learn the neighborhood instead of drawing the bins by hand.

Figure 12.5 shows that neighborhood on the coupon data. Every user is plotted by recency and frequency; the highlighted points are the ones the forest weights most heavily for Maria, sized by weight. The largest weights are assigned to users with her profile — recency past about 0.6, three or more purchases — in the same region identified by the honest tree in Figure 12.3. The forest assigns different weights within that region instead of treating all users in the group equally.

Figure 12.5: How the forest decides who is “like Maria”. Each point is a user, placed by recency and frequency; the highlighted points are the ones the forest weights most heavily when estimating Maria’s CATE (the star), sized by that weight. The largest weights go to users with her profile — recency past about 0.6 and three or more purchases — showing how the forest learns which users are comparable from the data.

12.6.5 The double machine learning behind the forest

In both Python and R, the forest first removes from the outcome and treatment whatever the user’s features already predict, then estimates the coupon’s effect from what remains. That step has a name, residualizing. For each user we subtract the value the features predicted and keep the leftover, the part the features could not explain, and we do this to the outcome and the treatment alike.

The forest estimates the effect from the relationship between the outcome and treatment residuals for each user profile. Econometricians know this as Robinson’s partialling-out, or the Frisch–Waugh–Lovell logic. It uses two models: one for the outcome and one for the treatment.

  • First, the outcome side. An outcome model \(m(X)=E[Y\mid X]\) predicts a user’s expected outcome from their features \(X\) alone: “what outcome do we expect from a user who looks like this, ignoring the coupon?” Subtract that prediction from what actually happened and you are left with the outcome residual \(\tilde Y = Y-\hat m(X)\), the part of the outcome the features could not explain.

  • Second, the treatment side. A treatment model, the propensity \(e(X)=E[D\mid X]\) — the same propensity score from the positivity assumption — predicts a user’s chance of getting the coupon from the same features: “how likely is a user who looks like this to be treated?” Subtract that from whether they actually got it and you are left with the treatment residual \(\tilde D = D-\hat e(X)\), the part of treatment the features could not explain, the “as good as random” variation.

To estimate the effect, fit \(m\) and \(e\), form \(\tilde Y\) and \(\tilde D\), then regress \(\tilde Y\) on \(\tilde D\). That slope is the coupon’s effect, and the forest estimates how it varies with the user’s features. This residualize-then-estimate step is called orthogonalization. It reduces the estimator’s sensitivity to small errors in \(m\) and \(e\). The terminology differs across implementations: grf describes this step as orthogonalization and relates it to the R-learner, a cousin of the meta-learners from earlier, while EconML makes the double-machine-learning connection explicit in the estimator name CausalForestDML (Chernozhukov et al. 2018).

That is the intuition behind double machine learning. The forest starts from two predictions for each user: their expected outcome and their expected chance of being treated. These predictions must be accurate enough for residualization, but they are not the target of the analysis. That is why they are called nuisance predictions. Flexible machine-learning models estimate these nuisance functions, after which the forest estimates how the coupon effect varies across users (Brand et al. 2023).

The procedure therefore has two stages: residualization removes the variation predicted by the features, and the forest uses the remaining variation to estimate how the treatment effect varies across users. We return to residualizing, with a worked picture, in Chapter 15.9

The code fits the forest, estimates each user’s CATE, and attaches the confidence interval needed for a decision.

library(grf)
df <- read.csv("data/coupon_allocation_experiment.csv")
X  <- as.matrix(df[, c("recency", "frequency", "disc_sens", "past_spend")])

# honest causal forest on the randomized coupon experiment
cf <- causal_forest(X, df$observed_profit, df$D, num.trees = 2000)

# a CATE for every user, with a standard error for a confidence interval
pred        <- predict(cf, estimate.variance = TRUE)
df$cate     <- pred$predictions
df$cate_se  <- sqrt(pred$variance.estimates)
df$cate_lo  <- df$cate - 1.96 * df$cate_se
df$cate_hi  <- df$cate + 1.96 * df$cate_se

average_treatment_effect(cf)   # forest's doubly robust ATE, as a sanity check vs. the simple diff
import pandas as pd
from econml.dml import CausalForestDML
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier

df = pd.read_csv("data/coupon_allocation_experiment.csv")
X  = df[["recency", "frequency", "disc_sens", "past_spend"]].values

# treatment is binary, so the propensity (D | X) model is a classifier
cf = CausalForestDML(
    model_y=RandomForestRegressor(n_estimators=300, min_samples_leaf=20),
    model_t=RandomForestClassifier(n_estimators=300, min_samples_leaf=20),
    discrete_treatment=True, n_estimators=2000,
)
cf.fit(df["observed_profit"].values, df["D"].values, X=X)

df["cate"] = cf.effect(X)
lo, hi = cf.effect_interval(X, alpha=0.05)   # 95% confidence interval per user
df["cate_lo"], df["cate_hi"] = lo, hi

After residualization, the forest estimates a CATE for each user and learns how that effect varies with the features. Unlike the OLS model in Section 12.4, it does not require the analyst to specify every treatment-feature interaction in advance. Figure 12.6 compares the two approaches. It plots each user’s estimated coupon effect (their CATE) against the true effect used to simulate the data, side by side for the forest and the OLS interaction. The forest’s estimates line up closely with the truth, a correlation of about 0.98, and it correctly picks out the small group of high-value users worth R$5 or more in extra profit from a single coupon. The OLS interaction, limited to the few terms we specified, pulls almost everyone toward the average (correlation about 0.64) and misses most of the discount-sensitive users.

Figure 12.6: Estimated and true coupon effects. Each point is a user: estimated CATE (vertical) against the true CATE (horizontal), with the dashed line showing perfect recovery. The causal forest (left) tracks the planted heterogeneity that the OLS interaction (right) averages away.

12.7 How to trust the results before acting

A causal forest can return CATE estimates that vary across users even when the true effect is identical for everyone and the estimated variation is noise. Before using those estimates to allocate a budget, ask two questions in order. First, is there evidence of heterogeneity? Use checks designed to distinguish differences in treatment effects from estimation noise. If out-of-sample calibration and ranking checks provide evidence of differences in treatment effects, ask what the forest found: which users respond, and which features identify them? This section first addresses the false-positive concern from Section 12.4, then interprets the fitted pattern and explains what an inconclusive result means.

12.7.1 Is the heterogeneity real?

Start with calibration, which asks whether differences between users’ estimated effects are supported by out-of-sample data.

The test is a single regression, and it runs on the two residuals the forest was already built from (Section 12.6.5): the outcome residual \(\tilde Y = Y - \hat m(X)\) and the treatment residual \(\tilde D = D - \hat e(X)\). Residualizing lets us assess the forest’s predicted effects. Once the features’ predictions are subtracted, the only systematic thing left that can move a user’s outcome residual is the treatment: what remains is the true effect times the treatment residual, plus noise. That leftover is observable, so the forest can be scored on how well its estimates stand in for the unknown true effect inside it.

Take each user’s out-of-bag CATE — predicted by trees that were not trained on that user — and separate the estimate into two parts: the average predicted effect for everyone (\(\bar\tau\)) and the deviation predicted for this user (\(\hat\tau(X_i) - \bar\tau\)). Regress the outcome residual on both pieces, and two coefficients come back: one grading the average, one grading the spread.10

A coefficient near 1 on the average component indicates that the forest’s overall effect is well calibrated. The coefficient on the deviations is the differential coefficient, and its p-value tests whether the predicted deviations contain out-of-sample information about how effects vary. This coefficient assesses whether the predicted deviations are appropriately scaled, need to be reduced, or contain no useful information. A coefficient near 1 suggests that the predicted differences are on the right scale overall. Around 0.5 suggests halving them: a predicted effect R$2 above the average would be adjusted to R$1 above it. These results describe the overall relationship between predicted and actual effects; they do not confirm the estimate for every user profile or tell us any individual’s personal effect. A coefficient near zero suggests that the forest does not reliably distinguish which users benefit more from the coupon. A statistically significant positive coefficient provides evidence that treatment effects vary and that the forest captures some of those differences. But a nonsignificant coefficient does not prove that effects are equal: as the section on statistical power explains, the test may lack power to detect real differences. On the coupon data both coefficients are close to 1 — the average at 1.00, the differential at 0.99 with a p-value below 0.001 — so the out-of-sample calibration supports using the predicted differences to inform targeting, subject to the remaining checks in this section.

Calibration assesses whether predicted differences are supported out of sample. The best linear projection (BLP) summarizes how treatment effects vary with features, using one regression coefficient per feature. Because each user’s treatment effect is unobservable, the regression uses a doubly robust score: the CATE estimate plus a correction term built from the same outcome and treatment residuals as the calibration test. “Doubly robust” means effect estimates can approach the correct values as the sample grows if either the model for receiving treatment is correct or the outcome models for both treatment groups are correct. Reliable standard errors here also require evaluating each user with predictions from models fitted on other users. Those predictions must be accurate enough, and the standard-error calculation must suit the way the scores were constructed (Semenova and Chernozhukov 2021).

Here the estimated effect rises with recency (4.66), discount sensitivity (3.09), and frequency (0.61), all significant at p < 0.001, while past spend sits at essentially zero. That is a story a marketer can check against what they already believe — lapsed, frequent, discount-sensitive buyers are the ones a coupon moves, and how much someone has spent in the past says nothing about whether it moves them. An implausible projection would not invalidate the model by itself, but it would be a reason to investigate the discrepancy.

Compare these coefficients only after accounting for each feature’s units. Recency’s 4.66 is reais across its whole 0-to-1 scale; frequency’s 0.61 is reais per purchase, and users differ from each other by many purchases. Comparing the raw coefficients makes frequency look less informative than it is. Run the projection again with standardized features so every coefficient is measured in reais per one standard deviation of that feature. After standardization, the order is recency 1.34, frequency 1.05, discount sensitivity 0.70, past spend −0.06 (p = 0.44) — frequency is nearly tied with recency because the raw coefficient sizes reflected different units.11

With either raw or standardized features, a coefficient summarizes how the estimated effect varies with a feature across the user base, not the effect of changing that feature. The projection does not establish that making a user buy more often would increase their coupon response. We return to the discount-sensitivity pattern in Figure 12.9.

Calibration and the projection describe differences in estimated effects. To act on those differences, we need to rank users and treat the highest-ranked ones. The next check evaluates that ranking.

GATES (Grouped Average Treatment Effects) evaluates the ranking using groups: sort users by predicted CATE and divide the list into equal-sized groups — quintiles here, so five groups of 4,000 users each, from lowest predicted effect to highest. Compare each group’s estimated uplift, measured as the treated-minus-control difference in average profit within the group (Chernozhukov et al. 2023). If the top bin’s estimated uplift is no greater than the bottom bin’s, examine the estimates and their uncertainty to determine whether the ranking reflects meaningful differences in treatment effects or merely random noise. On the coupon data the bottom four quintiles sit in a flat band between −R$1.31 and −R$0.73, and the fifth jumps to +R$5.66. Only that top quintile clears the R$5 the coupon costs, which is where the budget should go.

Estimated effects need not increase steadily from one GATES group to the next. In this simulation, users who miss either the recency or frequency threshold have the same true effect of −R$1.00, while effects among users who meet both thresholds increase with discount sensitivity. If the top bucket’s effect could not be statistically distinguished from the rest, GATES would provide no clear evidence that the ranking identifies more responsive users; imprecision could also explain that result. There is no GATES figure in this chapter — the five uplifts are the whole result, and the code block at the end of this section computes them in a few lines.

The TOC curve (Targeting Operator Characteristic) evaluates the ranking across targeting shares (Yadlowsky et al. 2023). While GATES divides users into five groups, the TOC varies the cutoff from the highest predicted CATE to the lowest. In Figure 12.7, the horizontal axis is the share \(q\) of users you would treat, taking them in order of predicted CATE. The vertical axis shows how much the estimated average effect for that top-\(q\) group exceeds the overall average effect, with a 95% confidence band. A curve that starts above zero and stays there is a ranking worth targeting on. Ours stays above zero until the last fifth of users, then falls to zero by construction: by then “the top slice” contains everybody, so its average effect equals the overall average.

A straightforward way to evaluate the ranking is to fit the CATE model on one sample and use it to rank users in an independent sample. Evaluate the ranking on that second sample by checking whether higher-ranked users benefit more on average. When training and evaluation reuse the same data without appropriate safeguards, random patterns learned by the model can be mistaken for real differences in treatment effects. The grf note on evaluating RATEs shows how some shortcuts that reuse data produce too many false positives, detecting differences even when the true effect is the same for everyone.

Figure 12.7: The TOC curve evaluates the ranking across targeting shares: extra uplift over the overall ATE among the top-ranked \(q\) share of users, ranked by predicted CATE, with a 95% band. It stays clear of zero until the final fifth — the same “the top group really does better” verdict as GATES, but without dividing users into fixed groups.

The RATE (Rank-Weighted Average Treatment Effect) summarizes the TOC curve as a weighted area, and the choice of weights affects what the summary detects (Yadlowsky et al. 2023). The AUTOC (Area Under the TOC curve) takes the unweighted area. Top-ranked users are included at every targeting cutoff, so AUTOC emphasizes the top of the ranking and has the most power when only a small group responds. The Qini coefficient weights each targeting share by how many users it includes. It gives more weight to the middle of the ranking and performs better when a weak response is spread across the user base.

Only around a tenth of users are worth treating in this example, so AUTOC is appropriate here and is also grf’s default. The estimate is 2.70, with a 95% confidence interval of [2.50, 2.91]. The interval excludes zero, supporting the conclusion that ranking by CATE improves targeting over targeting users at random. The weighting choice can change the conclusion because the metrics differ in statistical power — their chance of detecting a real advantage from the ranking. Qini can be more powerful when the response is spread across many users, while AUTOC can be more powerful when the response is concentrated in a small group. Compare each estimate with its uncertainty: a smaller but more precise estimate can provide stronger evidence than a larger, noisier one.

The remaining numerical checks cover overlap, placebo behavior, and stability under subsampling.

Overlap (positivity). The forest works by comparing treated and untreated users who look alike, so the comparison only exists where both kinds were actually available. That is the positivity assumption (Section 6.4.3), and you check it with the estimated propensity score — each user’s estimated probability of having been treated, given their features. Look for estimated scores close to 0 or 1. Near 1 means users of that type were almost all treated, so the forest has few untreated lookalikes to compare them against; it extrapolates instead of comparing, and its CATE there deserves a wide interval. Near 0 is the same failure in reverse. Randomization sets the assignment probability at 0.5 by design, but the estimated scores can still vary in groups with few observations. Here they all land between 0.36 and 0.65. No observed region is close to all-treated or all-control, so the CATEs are not leaning on obvious extrapolation.

Placebo (permuted treatment). Shuffle the treatment column at random — so D no longer records who actually received a coupon — and refit the forest. Shuffling breaks the systematic link between treatment labels and outcomes, although chance associations can remain in a particular shuffle. Compare the spread of predicted effects with the original fit and examine the differential calibration result. In the coupon experiment, the standard deviation of the predicted effects drops from R$2.95 to R$0.44, and the differential coefficient falls from 0.99 to 0.03 (p = 0.43). This shuffled fit produces much less apparent heterogeneity and no statistically significant differential calibration signal. That reduces concern about a fitting artifact, although a single shuffled fit cannot establish strong evidence.

Stability on less data. Refit the forest on random subsamples and compare their calibration and estimated effects with the full-sample fit. Similar results suggest stability at that sample size. On 2,000 users — a tenth of the experiment — the differential coefficient is still 1.10 (t = 11.6) and the estimates still correlate 0.95 with the planted truth.12 This is a demanding check: detecting differences in treatment effects often requires much more data than detecting an average effect. The section on statistical power illustrates how much more may be needed. Here, the estimated pattern remains statistically detectable with only one-tenth of the sample, supporting its stability at that sample size.

Two practices that do not return a number still matter: read the confidence intervals and check the fitted pattern against domain knowledge.

Confidence intervals and adaptive search. Every CATE and every segment average arrives with a confidence interval, and the interval is what tells you whether to believe the point estimate: a segment worth “+R$6 per user” whose interval runs from −R$2 to +R$14 is not a segment you fund, it is a segment you are guessing about. But each interval describes uncertainty in a single estimate and does not account for how many estimates you examined before selecting it. A forest examines many candidate subgroups, increasing the chance that one appears unusually responsive by chance. This is the multiple testing problem that affected the manual approach in Section 12.4. Use out-of-sample validation where possible, and build the ranking on data independent of the data used to evaluate it.

Consistency with domain knowledge. Ask someone who understands the business whether the estimated pattern has a plausible mechanism. “Lapsed, frequent, discount-sensitive buyers are the ones a coupon moves” is a sentence a marketer will nod at, because there is a story behind it. “Users who signed up on a Tuesday respond most” is a red flag however clean the statistics look, because there is none — and the likeliest explanation is a coincidence in this particular sample. A finding nobody can explain is a finding you should expect to be fragile.

Table 12.2 summarizes the seven quantitative diagnostics: what each check asks, which finding supports the intended use, what the forest returned, and which decision that result supports. Examining confidence intervals and checking consistency with domain knowledge remain part of the interpretation even though they do not receive rows.

Table 12.2: Diagnostic results for the coupon experiment. All values come from causal forests fit on the full 20,000-user sample, except the RATE row, which uses a train/evaluation split because ranking and evaluation must be independent, and the stability row, which refits on a 2,000-user subsample; the placebo row refits the forest on a permuted treatment column. The table quotes the grf numbers; the chapter’s companion scripts reproduce every row in both R and Python (the Python versions run on held-out splits, so their values differ slightly but support the same diagnostic conclusions). The BLP row projects on standardized covariates so its coefficients compare across features; the forest itself is fitted on the raw features.
Diagnostic What we want Coupon result Decision
Calibration (test_calibration) Differential coefficient near 1 and significant 0.99 (p < 0.001) Predicted spread carries out-of-sample signal — proceed to the remaining checks
Best linear projection (per 1 SD) Associations that match domain knowledge Projected CATE difference per 1 SD: recency +R$1.34, frequency +R$1.05, discount sensitivity +R$0.70 (all p < 0.001); past spend −R$0.06 (p = 0.44) The estimated associations are plausible — proceed
RATE (TOC) Positive, confidence interval clear of 0 AUTOC 2.70, 95% CI [2.50, 2.91] Ranking by CATE beats random targeting
GATES (CATE quintiles) Realized uplift concentrated in the top groups Q1–Q4 between −R$1.31 and −R$0.73; Q5 at +R$5.66 Only the top quintile clears the R$5 cost — aim the budget there
Overlap Propensities far from 0 and 1 Estimated propensities all within [0.36, 0.65] No extrapolation regions
Placebo (permuted treatment) Predicted-effect variation decreases and calibration finds no significant signal SD of predicted effects drops from 2.95 to 0.44; differential coefficient 0.03 (p = 0.43) Less apparent heterogeneity under shuffled treatment reduces concern about a fitting artifact; it does not rule one out
Stability (n = 2,000 subsample) Calibration and the effect pattern remain similar on less data Differential coefficient 1.10 (t = 11.6); correlation with the planted truth still 0.95 This subsample supports stability of the estimated pattern — continue with targeting tests

The code below implements the main diagnostics.

# Built-in validation on the fitted forest `cf` (from the previous section)
# test_calibration prints two rows: `mean.forest.prediction` grades the average
# effect, `differential.forest.prediction` is the differential coefficient.
test_calibration(cf)               # is the heterogeneity real and well-calibrated?
best_linear_projection(cf, X)      # which features move the CATE (raw units)?
best_linear_projection(cf, scale(X))  # same projection, coefficients per 1 SD
variable_importance(cf)            # how often each feature drives a split

# GATES: realized treated-minus-control uplift inside each predicted-CATE
# quintile. grf's predictions are out-of-bag, so bucketing in place is fair.
grp <- cut(df$cate, quantile(df$cate, 0:5 / 5), include.lowest = TRUE, labels = FALSE)
sapply(1:5, function(q) {
  g <- df[grp == q, ]
  mean(g$observed_profit[g$D == 1]) - mean(g$observed_profit[g$D == 0])
})                                 # Q1-Q4 between -1.31 and -0.73, Q5 at +5.66

# RATE / TOC: do users ranked high by CATE actually gain more?
# Rule from the docs: the ranking must come from data the evaluation forest
# never trained on -- rank-forest on one half, RATE evaluated on the other.
train    <- sample(nrow(df), nrow(df) / 2)
cf_rank  <- causal_forest(X[train, ], df$observed_profit[train], df$D[train])
priority <- predict(cf_rank, X[-train, ])$predictions
cf_eval  <- causal_forest(X[-train, ], df$observed_profit[-train], df$D[-train])
# target = "AUTOC" (the default) suits a narrow responsive group; use "QINI"
# when the response is spread across the whole base.
rate <- rank_average_treatment_effect(cf_eval, priority, target = "AUTOC")
print(rate)                        # a positive RATE means the ranking has value
import numpy as np, pandas as pd, statsmodels.api as sm
from sklearn.model_selection import train_test_split

# Average effect with a confidence interval (sanity check against the simple diff)
print(cf.ate_inference(X=X))
print(cf.feature_importances_)     # heterogeneity each feature creates (depth-weighted), not split counts

# Calibration: econml has no test_calibration, but the test is just the one
# regression described above, so we run it ourselves.
def new_forest():
    return CausalForestDML(
        model_y=RandomForestRegressor(n_estimators=300, min_samples_leaf=20),
        model_t=RandomForestClassifier(n_estimators=300, min_samples_leaf=20),
        discrete_treatment=True, n_estimators=2000,
    )

# Grade the forest on users it did NOT train on: effect(X) is in-sample, and
# scoring a forest on its own training users inflates the differential
# coefficient (1.29 in-sample here, against 1.02 held out; grf's out-of-bag
# version reads 0.99). Use the same independent train/evaluation split as
# the RATE, here and in the R tab.
Y, D  = df["observed_profit"].values, df["D"].values
tr, ev = train_test_split(np.arange(len(df)), test_size=0.5)
cf_tr  = new_forest().fit(Y[tr], D[tr], X=X[tr])                    # train-half forest (reused below)
tau_ev = cf_tr.effect(X[ev])                                        # CATEs from the train half
cf_ev  = new_forest().fit(Y[ev], D[ev], X=X[ev], cache_values=True) # residuals from the eval half
y_res, d_res = cf_ev.residuals_[0].ravel(), cf_ev.residuals_[1].ravel()   # Y - m(X), D - e(X)

tau_bar = tau_ev.mean()
A = tau_bar * d_res                # grades the average effect
B = (tau_ev - tau_bar) * d_res     # grades the spread
cal = sm.OLS(y_res, np.column_stack([A, B])).fit(cov_type="HC3")
print(cal.params)   # [1.32, 1.02]: the 2nd is the differential coefficient. The 1st
                    # is imprecise on half the data (SE ~ 0.41, so still consistent
                    # with 1); grf reads 1.00 and 0.99 on the full sample.

# GATES: bucket users by predicted CATE and compare the realized
# treated-minus-control uplift inside each bucket. Use the same independent
# train/evaluation split as above: effect(X) is not out-of-bag, so bucketing the
# forest's own training users would leak prediction noise into the uplift --
# rank the held-out half instead. (grf's predictions ARE out-of-bag, which is
# why the R run behind the validation table can bucket in place.)
y_ev, d_ev = Y[ev], D[ev]
grp = pd.qcut(tau_ev, 5, labels=False)
gates = pd.Series([y_ev[(grp == q) & (d_ev == 1)].mean()
                 - y_ev[(grp == q) & (d_ev == 0)].mean() for q in range(5)])
print(gates)   # Q1-Q4 all negative, Q5 far above -- the top group
               # stands clear, matching grf's [-1.31, -0.73] and +5.66

# Best linear projection (grf's best_linear_projection): regress each user's
# doubly robust score -- the AIPW pseudo-outcome, built from the same residuals
# as the calibration test -- on their features, with robust standard errors.
feats  = ["recency", "frequency", "disc_sens", "past_spend"]
e_ev   = D[ev] - d_res                                   # cross-fitted propensity e(X)
gamma  = tau_ev + d_res / (e_ev * (1 - e_ev)) * (y_res - tau_ev * d_res)
blp = sm.OLS(gamma, sm.add_constant(pd.DataFrame(X[ev], columns=feats))).fit(cov_type="HC3")
print(blp.summary2().tables[1])
# recency 4.98, disc_sens 3.39, frequency 0.62 (all p < 0.001), past_spend ~ 0
# -- the same drivers grf reports (4.66 / 3.09 / 0.61 / ~0)

# Same projection on standardized features: coefficients per 1 SD of each
# feature, comparable across features (the forest inputs stay raw).
X_std = pd.DataFrame(X[ev], columns=feats)
X_std = (X_std - X_std.mean()) / X_std.std(ddof=1)
blp_std = sm.OLS(gamma, sm.add_constant(X_std)).fit(cov_type="HC3")
print(blp_std.summary2().tables[1])
# per 1 SD: recency 1.43, frequency 1.07, disc_sens 0.77, past_spend ~ 0 --
# the same ordering as grf's 1.34 / 1.05 / 0.70 / ~0

# RATE (grf's rank_average_treatment_effect): econml ships it as DRTester.
# Same rule as the R tab: the CATE model comes from the train half only;
# nuisances and evaluation run on the held-out half.
from econml.validate.drtester import DRTester
tester = DRTester(
    model_regression=RandomForestRegressor(n_estimators=300, min_samples_leaf=20),
    model_propensity=RandomForestClassifier(n_estimators=300, min_samples_leaf=20),
    cate=cf_tr,
)
tester.fit_nuisance(X[ev], D[ev], Y[ev], X[tr], D[tr], Y[tr])
autoc = tester.evaluate_uplift(Xval=X[ev], Xtrain=X[tr], metric="toc")
print(autoc.summary())
# AUTOC 2.22 (SE 0.095), p < 0.001: the ranking has value, grf's verdict too
# (2.70, CI [2.50, 2.91]). DRTester integrates the TOC over a 5-95% grid, so
# its area runs a notch below grf's, which includes the very top slice.

12.7.2 What the forest actually found

The checks above are what license acting on the model. The three figures below describe the fitted pattern: which features the forest uses, how estimated effects vary with one of them, and how those estimates are distributed across users. Read them as interpretation, not as further validation. If calibration shows no reliable out-of-sample signal or GATES does not distinguish higher-response groups, these figures do not supply the missing validation evidence.

Start with the features the forest uses most. Its variable importance is a depth-weighted count of how often each feature gets split on — splits near the top of a tree count for more.13 Figure 12.8 shows the highest importance for recency and frequency — the features whose thresholds identify the responsive group — followed by discount sensitivity, with past spend lowest.

The pattern agrees with the business explanation and with the standardized projection, where frequency (1.05 per SD) is close to recency (1.34). The raw coefficients obscured that agreement: frequency’s 0.61 is per purchase, while other coefficients span a whole 0–1 feature. The remaining difference in emphasis concerns the pattern’s shape: the effect increases at a frequency threshold rather than rising smoothly, so the linear projection summarizes that change with a slope while the forest can split at the threshold. Both summaries identify responsive users through their behavior rather than their past spending.

Figure 12.8: Variable importance for the coupon model: grf’s depth-weighted split frequency, higher when the forest splits on a feature often, especially near the top of its trees. Recency and frequency lead, discount sensitivity follows, past spend barely registers. This describes how the fitted forest uses the features, not the size of any effect.

The second figure shows how estimated effects vary with discount sensitivity. Figure 12.9 plots each user’s estimated effect against their discount sensitivity — a companion view of the pattern summarized by the best linear projection, not the projection itself. Each faint dot represents one user, and the solid curve shows the average estimated effect at each level of sensitivity. The shaded band describes uncertainty in that smoothed trend; it does not represent the estimation error in each CATE or a confidence interval for any single user’s effect.

The curve rises steadily with sensitivity but stays below the dotted R$5 cost line even at high sensitivity. The resolution is in the dots: the ones sitting above the cost line are the profitable users, and they cluster at high sensitivity only where recency and frequency line up too — so the curve, an average over everyone at that sensitivity, is pulled back under the cost. That is exactly why a single-feature rule misses them, and why the histogram that follows still shows a profitable tail. A flat or decreasing curve would not disprove the model by itself, but it would be a reason to investigate why the fitted pattern conflicts with the business explanation.

Figure 12.9: Estimated coupon effects by discount sensitivity. Each faint dot is one user’s estimated CATE against their discount sensitivity; the solid curve is the smoothed average of those estimates at each sensitivity level (the shaded band is that trend’s uncertainty, not any user’s confidence interval) and the dotted line is the R$5 cost. The average rises with sensitivity but stays below the cost line — yet the dots above that line are profitable users, because the biggest effects need sensitivity and the recency and frequency gates together.

We compared estimated effects with simulated truth in Figure 12.6; Figure 12.10 now shows how those estimates vary across users. The horizontal axis shows estimated CATEs, and the vertical axis counts users in each range. Most users have estimated effects below the coupon cost, but a small minority have estimates above it — a distinction the chapter-opening average did not show. This is why we examine the distribution rather than rely on a single OLS coefficient: coupons lose money on average, yet can be profitable for a few users, so the business decision depends on whom we target.

Figure 12.10: Distribution of estimated CATEs. The dashed line marks the mean; the shaded band past the dotted cost line is the minority for whom a coupon pays off.

12.7.3 “No heterogeneity detected” may be a power problem

A final question is how to interpret inconclusive checks. Flat calibration and overlapping GATES buckets do not justify the conclusion “everyone responds the same, case closed.” Detecting heterogeneity is a much harder statistical problem than detecting an ATE. The ATE is one difference; heterogeneity is a difference of differences, estimated on slices of the sample — the standard error doubles before the search even starts, and the effect differences worth finding are usually smaller than the ATE itself.

A useful rule of thumb quantifies the difference: detecting an interaction half the size of the main effect takes about sixteen times the sample (Gelman 2018; Gelman et al. 2020, sec. 16.4).14

I made the heterogeneity in this simulation deliberately strong. The planted effects average R$0.38 and have a standard deviation of R$2.8, so the variation is about seven times the average effect. That is why every method in this chapter performs well with 20,000 users and why the forest still detects the pattern in a 2,000-user subsample.

Heterogeneity in real campaigns is often weaker. Rerun the same pipeline with the same average effect and noise, but shrink the planted spread to half the ATE. The 20,000-user experiment still pins down the ATE (a t-statistic above 4). Yet the calibration differential coefficient falls to −0.12 with a p-value of 0.77, and the forest’s correlation with the planted truth drops from nearly 1 to roughly 0.2–0.3 across implementations. Same users, same sample size, genuinely different effects across the base — and every check above reads “nothing here.”

# Only possible in a simulation: shrink the planted heterogeneity around the
# SAME average effect (same noise, same users) and see what survives.
y_base    <- df$observed_profit - df$D * df$true_cate  # outcome without treatment effects
ate_true  <- mean(df$true_cate)
tau_quiet <- ate_true + (df$true_cate - ate_true) / 16    # spread shrunk to ~half the ATE
df$y_quiet <- y_base + df$D * tau_quiet

cf_quiet <- causal_forest(X, df$y_quiet, df$D, num.trees = 2000)
t.test(df$y_quiet[df$D == 1], df$y_quiet[df$D == 0])  # ATE still detected: t above 4
test_calibration(cf_quiet)             # differential coefficient -0.12, p = 0.77: reads flat
cor(predict(cf_quiet)$predictions, tau_quiet)         # truth correlation collapses to ~0.3
# Only possible in a simulation: shrink the planted heterogeneity around the
# SAME average effect (same noise, same users) and see what survives.
from scipy import stats

y_base    = df["observed_profit"] - df["D"] * df["true_cate"]  # outcome without treatment effects
ate_true  = df["true_cate"].mean()
tau_quiet = ate_true + (df["true_cate"] - ate_true) / 16          # spread shrunk to ~half the ATE
y_quiet   = (y_base + df["D"] * tau_quiet).values

cf_quiet = CausalForestDML(
    model_y=RandomForestRegressor(n_estimators=300, min_samples_leaf=20),
    model_t=RandomForestClassifier(n_estimators=300, min_samples_leaf=20),
    discrete_treatment=True, n_estimators=2000,
)
cf_quiet.fit(y_quiet, df["D"].values, X=X)

print(stats.ttest_ind(y_quiet[df.D == 1], y_quiet[df.D == 0]))   # ATE still detected: t above 4
print(np.corrcoef(cf_quiet.effect(X), tau_quiet)[0, 1])          # truth correlation collapses to ~0.2
# Calibration: run the held-out two-column regression from the diagnostics fold on this
# quiet world and the differential coefficient reads -0.22 (p = 0.22) -- flat, matching
# grf's -0.12 (p = 0.77). Both say "nothing here."
# (Bucketing in-sample effect(X) into a GATES table would mislead in this quiet world:
# effect(X) is not out-of-bag, so use a held-out split if you want GATES.)

For a null result in a heterogeneity analysis, follow the guidance in Chapter 5: examine what the confidence interval covers rather than relying only on whether the p-value is below 0.05. An interval on the differential coefficient wide enough to contain both 0 and 1 is absence of evidence, not evidence of absence: the experiment is too imprecise to distinguish no heterogeneity from a well-calibrated CATE model. If targeting is the goal, size the experiment for the smallest effect difference you need to detect, not only for the ATE. A sample with adequate power for the ATE can still be an order of magnitude too small for heterogeneity analysis.

Use the estimated heterogeneity to inform targeting when out-of-sample calibration and ranking results support differences in response, with enough precision to guide the decision.

12.8 From CATE to targeting

A validated CATE helps inform the decision of whom to treat under a limited budget. We turn the per-user estimates into a decision in three steps: rank users by CATE until the budget runs out, compare that ranking with predicted coupon redemption, and learn a targeting rule directly from the estimates.

Uplift, not propensity. Before ranking users, distinguish purchase propensity from treatment effect. A propensity model predicts how likely each user is to buy — marketing’s use of the word, not the propensity score from the validation section, which is the probability of being treated. That likelihood of buying says nothing about the coupon: a user can be almost certain to buy whether or not you send them one. A CATE model — an uplift model, as marketers call it — predicts how much the coupon changes what each user spends. These are different questions, and they rank users in different orders: the users most likely to buy are often not the users the coupon influences most.

A CATE ranking sorts users into four kinds marketing already has names for — though it cannot tell the first two apart, and it does not need to. Sure things would buy with or without the coupon; their CATE is near zero, so a coupon just gives away R$5 on a sale that was going to happen anyway. Lost causes would not buy either way; their CATE is also near zero, and the coupon is wasted on them too. Sleeping dogs are users the coupon actively hurts — their CATE is negative (Gutierrez and Gérardy 2017). For example, a discount email can remind a dormant subscriber to cancel a subscription they had forgotten, or teach a full-price buyer to hold out for the next deal.

That leaves the persuadables, the users with a positive CATE — the only group whose behavior the coupon improves. A positive effect is profitable only when the persuadables’ CATE exceeds the R$5 coupon cost.

Predicting who will buy does not by itself tell us who will benefit most from a coupon. Our simulation makes the distinction using profit: past_spend predicts profit without the coupon, but it does not enter the formula for the coupon’s effect. It also has the lowest forest importance in Figure 12.8. Because past spending was generated independently of the features that determine the effect, ranking users by past spending alone would not identify the profiles that benefit most.

Rank, cut, and compare. A simple fixed-budget targeting policy is to rank users by estimated CATE, walk down the list, and treat people until the budget runs out. To turn “effect” into “money,” subtract the treatment cost — here, the R$5 coupon — from each user’s CATE to get their expected value if treated, \(\text{EV} = \hat{\tau}(x) - \text{cost}\). One caveat before the table: the numbers below are computed from the model’s own CATE estimates, so they say what the model expects each policy to earn — not what a campaign actually earned. The out-of-sample check of the chosen policy comes later.

Policy Model-implied net incremental value
Blanket: give everyone a coupon −R$94,300
Random 20% of users −R$19,000
Target the top 20% by CATE +R$2,400
Treat only users with positive expected value (~11%) +R$5,600

The policies produce different expected results because they treat different users and shares of the base. Giving everyone a coupon loses almost six figures because the typical user is estimated to return about −R$1 in incremental profit against a R$5 cost. A random 20% loses money for the same reason: most users in any random sample are not persuadable. Ranking by CATE turns the program profitable, and stopping where expected value crosses zero — at about 11% of the base — captures essentially all of the gain the model predicts.

Figure 12.11 then scores the CATE-ranked and random-order policies against the truth. Because we simulated the data, we can rank users by the forest’s estimated CATE and still score each policy on the true effects we planted: the vertical axis is cumulative true net value (each treated user’s true CATE minus the R$5 cost), the horizontal axis is the share of users treated, and users enter in order of estimated CATE on one line and in random order on the other. The forest’s estimates determine the CATE-ranked order, while the simulated true effects determine each policy’s net value — so the curve shows what each policy would earn in the simulation, rather than what the model expects.

Read the CATE-ordered line by its shape. It climbs while the budget reaches positive-value users and peaks around the 11% mark, where the marginal user’s true value turns negative. That peak is the most the program can make; beyond it, further coupons go to users who cost more than they return, so the curve declines. At the 20% budget line, cumulative value is below its peak but remains far above the value from random targeting, which declines toward the loss from treating everyone.

A plot like this is called a cumulative-gains curve, or a Qini curve in the uplift-modeling literature, and the vertical gap between the two lines at any budget is the value targeting actually creates over random assignment (Gutierrez and Gérardy 2017).15

On your own data no one hands you the true effects, so you cannot draw this exact curve: uplift models have no per-user ground truth to grade against. The workable substitute is a realized curve evaluated on held-out data, using the same separation between ranking and evaluation as GATES and the TOC. Rank held-out users by predicted CATE; at each budget share, take the top-ranked slice and compute the uplift it actually realized — the treated-minus-control difference in average profit inside the slice, times the number of users you would treat — then subtract the coupon cost and plot that cumulative value against the share treated. That realized curve is how the Qini curve is computed in practice, and its peak picks the cutoff the same way the true-effect peak does here.

Figure 12.11: Targeting, scored against the truth. Cumulative true net value (each treated user’s planted true CATE minus the R$5 cost) as the budget grows, ordering users by the forest’s estimated CATE (treat the most responsive first) versus random order. The forest-ordered line climbs, peaks where the marginal user’s true value turns negative (around the profitable 11%), then declines; the random line declines. At the dashed 20%-budget line, CATE-targeting still produces positive net value while random targeting produces a large loss.
coupon_cost  <- 5
budget_share <- 0.20
n_budget     <- floor(budget_share * nrow(df))

# df$cate came from predict(cf), which is out-of-bag: no user is scored by
# trees that trained on them (Python has no OOB mode -- see its tab)
ranked <- df |>
  mutate(expected_value = cate - coupon_cost) |>   # net value per coupon
  arrange(desc(cate))

value_targeted <- sum(head(ranked$expected_value, n_budget))           # top 20% by CATE
value_random   <- sum(sample(ranked$expected_value, n_budget))         # random 20%
value_blanket  <- sum(ranked$expected_value)                           # treat everyone
value_positive <- sum(ranked$expected_value[ranked$expected_value > 0]) # positive-EV users only (~11%)
c(targeted = value_targeted, random = value_random,
  blanket = value_blanket, positive_ev = value_positive)
import numpy as np
coupon_cost, budget_share = 5, 0.20
n_budget = int(budget_share * len(df))

# econml's effect(X) scores the forest's own training rows in-sample (there is
# no out-of-bag mode like grf's), so build an out-of-fold CATE before money is
# assigned by it: each half is scored only by a forest trained on the other half,
# written back in the original row order.
half_a, half_b = train_test_split(np.arange(len(df)), test_size=0.5, shuffle=True)
cate_xfit = np.empty(len(df))
cate_xfit[half_b] = new_forest().fit(Y[half_a], D[half_a], X=X[half_a]).effect(X[half_b])
cate_xfit[half_a] = new_forest().fit(Y[half_b], D[half_b], X=X[half_b]).effect(X[half_a])
df["cate_xfit"] = cate_xfit

ranked = df.assign(expected_value=df["cate_xfit"] - coupon_cost) \
           .sort_values("cate_xfit", ascending=False)

value_targeted = ranked["expected_value"].head(n_budget).sum()          # top 20% by CATE
value_random   = ranked["expected_value"].sample(n_budget).sum()        # random 20%
value_blanket  = ranked["expected_value"].sum()                         # treat everyone
value_positive = ranked.loc[ranked["expected_value"] > 0, "expected_value"].sum()  # positive-EV users only (~11%)
print(dict(targeted=value_targeted, random=value_random,
           blanket=value_blanket, positive_ev=value_positive))

The targeting matrix. Ranking by CATE answers “who would respond?”, but a real campaign must also ask whether the user will redeem the coupon. The scores can disagree because they come from different models. The CATE averages over a user’s lookalikes, while a redemption model can use pre-treatment signals absent from the CATE model, such as past redemption rates. A user whose lookalike group gains a lot can still be one the offer rarely reaches, and for them the group’s CATE overstates what this campaign will collect. The wrong fix is to multiply the two scores: the treatment here is coupon assignment, so the CATE already averages over redeemers and non-redeemers among users with the same features, and scaling it by a predicted redemption probability double-counts the non-redemption it already contains. If pre-treatment redemption history really carries information about treatment response, add it to the CATE’s feature set and re-estimate the effect instead. Crossing who benefits (high or low CATE) with who acts (high or low predicted redemption) gives the two-by-two strategy map we first sketched in Chapter 14, now recast for coupons:

High CATE (large incremental profit) Low CATE (little to gain)
High redemption Core audience — the CATE-minus-cost rule already says treat them Low-value engagers — stop spending here
Low redemption Missed opportunity — fix the offer, not the product Irrelevant — ignore

Before you build this matrix, make sure the redemption axis uses predicted redemption estimated from pre-treatment history (past redemption rates, for instance). Actual redemption of this campaign’s coupon is a post-treatment outcome, so the feature-freeze rule from the identification section excludes it from the model.

The Missed opportunity quadrant contains users whose predicted effect is large but whose predicted redemption probability is low. For this group, test whether changes to the offer or delivery mechanism, such as a clearer email, one-tap redemption, or a reminder, increase redemption. The matrix alone cannot establish why redemption is low, and it does not replace the decision rule — that stays estimated CATE minus cost.

12.8.1 A learned rule: the policy tree

Ranking by CATE and the targeting matrix both rely on cutoffs you choose by hand. A policy tree automates that choice. You give it a per-user reward for each possible action — here, R$0 for skipping the coupon and the estimated CATE minus the R$5 cost for sending it — and it returns a short set of if-then rules, arranged as a shallow decision tree, that assigns each user to coupon or no coupon so that the total reward is as high as possible. The search over candidate trees is exhaustive, not greedy, so the rule it returns is the best tree of its depth for the rewards you supplied (Zhou et al. 2023).16

In production you would give the tree doubly robust scores rather than the raw forest CATE. A doubly robust score — the same construction behind the best linear projection — still lands on the right answer if either the outcome model or the treatment model is correct, even when the other one is wrong (Brand et al. 2023), which is why it is the standard input for policy learning (Athey and Wager 2021). I use the CATE net of cost here so the path from the forest to the decision stays easy to follow. To switch, policytree::double_robust_scores(cf) returns one score per action: keep both columns and subtract the R$5 cost from the coupon column only — the no-coupon column keeps its own score rather than being set to zero. Expect close but not identical results to the simple version; they are different estimates of the same reward.

Figure 12.12 shows the depth-2 tree learned on our data: send a coupon only to users who buy often (frequency above two) and have lapsed (recency past about 0.6), and skip everyone else. A product team can inspect and test this short rule more easily than the full forest.

Score the rule before trusting it, because it has two valuations and they disagree. It treats 22% of users and, scored by the model’s own out-of-bag estimates, expects about +R$365 net of coupon costs. Scored against the planted true_cate — the check only a simulation allows — the same allocation earns essentially nothing: about R$55 below zero (policytree’s score; mcf’s near-identical rule lands just above zero, the same verdict). The gap is the tree’s depth. The planted rule needs three gates — recency, frequency, and discount sensitivity — and a depth-2 tree can express only two. It uses the recency and frequency thresholds but must treat every user who meets both. The low-sensitivity users inside it have true effects below the R$5 cost and almost exactly cancel the profitable minority. That coarseness is also why the tree’s +R$365 falls so far short of the targeting table’s +R$2,400 for the top 20% by CATE: the ranking picks users one at a time, while the tree can only take or leave its whole box. Compare Figure 12.11, whose truth-scored optimum treats about 11% — half this rule’s reach. This is the best depth-2 rule for the supplied scores and a readable policy hypothesis. The chosen depth limits the rule in this simulation, so the next step is to add one more level and check how the allocation changes.

Figure 12.12: A targeting rule you can read. The depth-2 policy tree assigns each user to coupon or no coupon to maximize estimated value net of the R$5 cost, and it treats only frequent, lapsed buyers. Depth 2 cannot also gate on discount sensitivity, so the rule treats 22% of users where the truth-scored optimum treats about 11%: its model-implied value is about +R$365, but scored against the planted effects it earns essentially nothing (about R$55 below zero); the missing gate is the subject of the next figure.

Given a third level, the tree finds the missing gate (Figure 12.13). Fit on the same reward scores, it keeps the recency and frequency gates and now also gates on discount sensitivity, treating only the frequent, lapsed users whose sensitivity is above about 0.53. The new rule reaches about 10% of users — right where the truth-scored curve in Figure 12.11 peaks — and its two valuations now point the same way: about +R$5,400 by the model’s own estimates and about +R$4,900 against the planted truth, roughly 95% of the most any allocation can earn here (R$5,165, from treating exactly the users whose true effect exceeds the R$5 cost — which on this data is precisely the planted three-gate rule). The two numbers still sit about 10% apart, with the model’s own estimate on top. Out-of-bag estimation protects each user’s score from self-prediction, but the search then picks the rule whose estimated value is highest, so estimation error inflates the winner’s total: the tree is selected and valued on the same scores. The holdout below checks for this upward bias, which can increase as the search considers more rules.

Figure 12.13: One level deeper. The depth-3 policy tree, fit on the same reward scores as the depth-2 tree above, adds the gate depth 2 could not express: it treats only frequent, lapsed users with discount sensitivity above about 0.53. The rule reaches about 10% of users, and its estimated and true net values are now close — about +R$5,400 model-implied against +R$4,900 on the planted truth — unlike the depth-2 tree’s values.

Depth increases computational cost sharply. On 20,000 users, the exact search that makes the depth-2 tree provably optimal at that depth is computationally impractical at depth 3 (Sverdrup et al. 2020). The depth-3 tree above comes from policytree’s hybrid_policy_tree, which runs the exact search one level at a time (Sverdrup et al. 2026): it finished in under a minute, but it is no longer guaranteed to find the best possible depth-3 tree — here it happens to land on the planted rule. On the Python side, mcf searched depth 3 over its candidate split grid (100 evaluation points per feature) and took roughly an hour where depth 2 took twenty seconds, arriving at nearly the same treatment region. The close agreement occurs because the planted rule is exactly three gates deep. In real data, we do not know how many levels the useful targeting rule needs. A deeper tree can fit more noise in the scores, and increasing depth cannot recover an effect pattern absent from the CATE estimates: this tree found discount sensitivity because the forest’s estimates already reflected it. Choose depth by evaluating the rule on held-out data.

library(policytree)
# reward per action: R$0 for no coupon, out-of-bag CATE net of the R$5 cost for
# a coupon. Production alternative: double_robust_scores(cf) returns one score
# per action -- keep BOTH columns, then subtract the cost from the coupon
# column only (Gamma[, 2] <- Gamma[, 2] - 5); never zero the no-coupon column.
reward <- cbind("No coupon" = 0, "Coupon" = df$cate - 5)
ptree  <- policy_tree(X, reward, depth = 2, min.node.size = 1000)
print(ptree)        # the learned if-then rule
plot(ptree)         # draw the tree (needs the DiagrammeR backend)

# one more level: the exact depth-3 search is computationally impractical at
# this n, so hybrid_policy_tree runs the exact lookahead one level at a time
# instead (fast, but not guaranteed to be the best possible depth-3 tree)
ptree3 <- hybrid_policy_tree(X, reward, depth = 3, search.depth = 2,
                             min.node.size = 1000)
from mcf.optpolicy_functions import OptimalPolicy

# policy score per action: R$0 for no coupon, the out-of-fold CATE from the
# targeting section net of the R$5 cost for a coupon
df["score_nocoupon"], df["score_coupon"] = 0.0, df["cate_xfit"] - 5
optp = OptimalPolicy(
    var_polscore_name=["score_nocoupon", "score_coupon"],
    var_x_name_ord=["recency", "frequency", "disc_sens", "past_spend"],
    gen_method="policy tree", pt_depth_tree_1=2,
    pt_depth_tree_2=0,   # optimal depth-2 tree; no second-stage tree inside the leaves
)
alloc, _, _ = optp.solve(df, data_title="coupon")  # alloc["Policy Tree"]: 0 = skip, 1 = coupon

# one more level: mcf searches its candidate split grid (100 points per feature)
# at depth 3 -- expect a long run (roughly an hour at n = 20,000)
optp3 = OptimalPolicy(
    var_polscore_name=["score_nocoupon", "score_coupon"],
    var_x_name_ord=["recency", "frequency", "disc_sens", "past_spend"],
    gen_method="policy tree", pt_depth_tree_1=3, pt_depth_tree_2=0,
    pt_no_of_evalupoints=100,  # the candidate grid per feature (mcf's default, stated)
)
alloc3, _, _ = optp3.solve(df, data_title="coupon_depth3")

A learned rule is a hypothesis. The tree was trained and evaluated on the same experiment, so its estimated profit is an in-sample estimate. Before the rule reaches the full base, measure it out of sample: roll it out while keeping a randomized slice of users on business-as-usual as a holdout, and compare the two groups to measure the policy’s realized incremental value — the same design the original experiment used to measure the coupon.

Then keep watching after launch. The CATEs were estimated on one period’s users, and the rule decays: the customer mix shifts, seasons change, and the policy itself changes behavior (users can learn that lapsing triggers a coupon, which trains them to lapse on purpose). Re-estimate the forest and re-learn the tree on a regular cadence, always against that standing holdout. Finally, specify the constraints before deployment. Fairness requirements, exclusion lists, frequency caps, margin floors, and regulatory limits should be part of the policy rather than added in response to a failure.

The shallow tree here is the simplest case of policy learning, the broader theory of choosing an optimal assignment rule under budget and fairness constraints. That theory — including the doubly robust objective and settings with more than two possible actions — gets a dedicated section in Chapter 15. The manual ROI arithmetic that turns a per-segment CATE into a budget line is developed in Section 14.6.1. Targeting uses the estimated heterogeneity to decide who receives treatment.

12.9 What can go wrong

Even when the validation statistics look sound, acting on the estimate creates ongoing risks. Some have a diagnostic in the validation section. Others cannot be detected by any in-sample check and require standing safeguards: a frozen feature table, a permanent holdout, and a pre-launch audit. Review each risk before rollout.

False positives from slicing. The multiple testing problem from Section 5.5.2 arises when a forest considers thousands of candidate subgroups: some appear responsive by chance, such as a segment with an estimated gain of +R$6 per user whose gain is not reproduced when you refit. Honest estimation and out-of-sample validation reduce this risk but do not eliminate it. Before allocating budget to a segment, reproduce its response in independent data and investigate unexpected shuffled-treatment results, allowing for chance findings.

Hidden confounding. This risk applies only to observational data; the coupon was randomized, which is why the chapter established that scope at the outset. Suppose instead the coupons had been sent by an old marketing rule that targets lapsed users, and lapsed users were already due to bounce back on their own: every CATE absorbs that bias, and the forest reports the biased numbers with tight, confident intervals, because the model cannot warn you about a variable it never saw. No in-sample check detects this problem. Examine how treatment was assigned and, before using the estimates for targeting, run the sensitivity analysis in Chapter 13 to assess how strong an unmeasured confounder would have to be to erase the result.

Post-treatment features. The bad controls described in Section 6.3 can also enter the model as features. Add “redeemed the coupon” to the feature table and the forest will discover that redeemers respond most — but that is the outcome leaking into the segmentation, not heterogeneity, and the same goes for post-campaign engagement and follow-up purchases. The screening question for every candidate feature is “could the treatment have changed this value?” — a timestamp after assignment day is an automatic disqualifier. Freeze the feature table at assignment time, as this chapter’s four features were.

Weak overlap. Where a region of feature space is nearly all-treated or all-control, the forest has no lookalikes to compare, so the CATE there is extrapolation, not estimation (Section 6.4.3). Our randomized data kept every estimated propensity between 0.36 and 0.65; a pilot that only ever couponed high spenders would not — and the forest would still hand you a number for everyone else. The overlap check in the validation section is the detector. Treat a wide interval in a region with few comparable users as a reason to stop: do not fund a segment whose CATE rests on extrapolation.

Model instability. Refit the forest on a new week of data and the ranking may change most in the middle of the distribution, where CATEs differ by only a few centavos. Users near a targeting threshold can move in and out of the campaign from one refit to the next. The stability check detects large changes; to detect smaller changes, compare the rank ordering across refits before using the score in an automated system. Then target broad, stable tiers — treat the top 20%, say — rather than base decisions on small score differences that may change after the next refit.

Over-trusting individual predictions. A CATE is a high-resolution group average — the effect on users like Maria, never on Maria herself, as the estimand section established — and individual effects vary around it. Quote “this user is worth R$7.20” and you have made a promise the method cannot keep; the first spot check that contradicts an individual prediction costs the whole model its credibility. Communicate segment-level averages with their confidence intervals, and avoid presenting a single person’s estimated effect to the nearest cent.

Short-term optimization. A rule that maximizes this month’s incremental profit can base eligibility on behavior users learn to change deliberately: if lapsing triggers a coupon, users learn to lapse. As the targeting section explained, this feedback can reduce long-run margin even while short-run results look favorable. Only a standing holdout, watched over a horizon longer than one campaign, reveals the drift. Define the outcome over the horizon the business actually cares about — Chapter 14 discounts effects that decay for the same reason — and see Tran et al. (2024) for methods that formally connect short-run experiments to long-run outcomes.

Fairness. None of our four features names a protected attribute, yet recency, frequency, and spending patterns correlate with income and geography, so an “optimal” policy can systematically exclude — or exploit — a protected group without any feature admitting it. That is a disparate-impact liability, legal as well as ethical, and no accuracy metric will flag it. Before launch, tabulate the learned rule’s treatment decisions across the groups you are obligated to protect, and enforce the constraints when deploying the policy. Use the exclusion lists and audits from the policy-tree section; constrained policy learning is discussed further in Chapter 15.

These risks do not rule out heterogeneity analysis, but they require safeguards: validate the estimates, freeze features at assignment, prefer stable targeting tiers, retain a randomized holdout, and audit which groups the policy treats and excludes.

12.10 Wrapping up and next steps

The main conclusions are:

  • An average close to zero can still hide the decision. Our coupon’s average effect was R$0.30 per user — small enough for a CFO to round to zero. The average combined about 11% of users who returned roughly R$7 each in extra profit with a majority who cost more in margin than they gave back. The average answers “did the coupon work?” The business question is “who did it work for?”, and an overall average does not show which users benefit.

  • You can estimate the effect on people like a given user, never the effect on that user. Maria either gets the coupon or she doesn’t; we never see both versions of her, so her personal effect is unknowable. What we can estimate is the conditional average treatment effect, the CATE: the average effect among users whose features look like Maria’s. Every tool that promises “individual-level effects” is really handing you a group average with high resolution.

  • The assumptions that make an effect causal do not go away because you switched to machine learning. You still need unconfoundedness (nothing you failed to measure drives both who got treated and how they responded) and overlap (every type of user had a real chance of landing in either group). A causal forest fits a more flexible model to the variation available in your data. It does not create variation that isn’t there, and it will not warn you about a confounder it never saw.

  • Only use features that existed before treatment was assigned. Whether the user redeemed the coupon, how engaged they were afterwards, what they bought next — all of these are affected by the treatment. Split users on them and you are no longer asking “for whom does the coupon work?” but “among users who already responded, how do the numbers look?” Snapshot the feature table on assignment day and let nothing dated after that in.

  • Hand-built heterogeneity is transparent but limited. You can split users into subgroups, add interaction terms to a regression, or fit one model per treatment arm. All three are transparent and sometimes enough. But an interaction term only finds the pattern you thought to write down: on our data, the OLS interaction model’s estimated effects correlated only 0.64 with the truth, while the forest reached 0.98. And when you slice the data twenty ways and keep the slice that looks interesting, you will find a “responsive” segment by luck alone.

  • A causal forest searches systematically for treatment-effect heterogeneity while separating subgroup discovery from effect estimation. It grows hundreds of trees, each grouping users into leaves. For each user’s feature profile, the generalized random forest combines the trees’ comparison neighborhoods into weights, then uses those weights to estimate the effect from the residualized outcome and treatment. Three ingredients make it trustworthy. Honesty: the rows used to choose the splits are separate from the rows used to measure the effect, so the estimate does not reuse the noise that made the group look promising. Averaging: hundreds of trees, each grown on a different subsample, reduce instability from any one set of splits. Residualizing: before looking for heterogeneity, the method subtracts from both the outcome and the treatment whatever the features already predict, then estimates how the effect varies using the remaining variation. What comes out is an effect estimate for every user, with a confidence interval you can use.

  • A forest can produce different estimates even when the true effect is the same for everyone, so you have to test whether the differences are real. The validation checklist assesses whether those differences are supported by the data. Calibration asks whether the users the forest predicted would respond more actually did, and by the predicted amount. The best linear projection asks which features are associated with larger or smaller effects, and in which direction, so you can check the pattern against what the business already believes. GATES sorts users by predicted effect, cuts the list into buckets, and compares the uplift each bucket actually realized. The TOC curve and its summary number, the RATE, ask whether treating the top of the ranking beats treating people at random. Overlap checks that no group of users was almost entirely treated or entirely untreated. A placebo test shuffles the treatment column and confirms the apparent heterogeneity collapses. A stability check refits on a fraction of the data and asks whether the estimated pattern remains similar. And two checks return no number at all: read every confidence interval before you believe a point estimate, and say the finding out loud to someone who knows the business — if nobody can name a mechanism, expect it to be fragile.

  • “We found no heterogeneity” often means “our sample was too small to see it.” An average effect is one difference. Heterogeneity is a difference between differences, estimated on halves of the sample, so its standard error is twice as large before the search even begins. Detecting an interaction half the size of the main effect takes roughly sixteen times the sample size. An experiment with adequate power to measure the average effect can be an order of magnitude too small to detect how effects differ across users.

  • Rank users by how much the coupon changes their behavior, not by how likely they are to buy. Purchase propensity and treatment effect are often confused in targeting decisions. Your best customers buy with or without the discount, so couponing them just burns margin. The users worth paying for are the ones whose behavior actually changes — and they can sit anywhere in a propensity ranking. In our simulation, past spending predicts profit without the coupon but does not enter the formula for the coupon’s effect. It was also the feature the forest relied on least. On the model’s own arithmetic, ranking by estimated effect instead flipped the program from a R$94,300 loss if we couponed everyone to a R$5,600 gain.

  • A learned targeting rule is a hypothesis until a holdout says otherwise. A policy tree turns the forest’s thousand-tree score into a short if-then rule a product team can read and argue with — ours came back as “coupon only frequent buyers who have lapsed.” But a rule’s promise and its performance are different numbers: ours expected about +R$365 from its own estimates and earned essentially nothing against the planted truth (about R$55 below zero), because a depth-2 tree cannot express the third gate (discount sensitivity) the profitable segment needs. One level deeper, the tree recovered that gate and both valuations turned strongly positive (about 10% apart) — in this simulation the depth constraint, not the method, was the shortfall. Roll a rule out against a randomized slice of users kept on business-as-usual, measure what it actually earned, and re-estimate as the user mix drifts.

How to run a heterogeneity analysis on your own data:

  1. Confirm the average effect is credible before you go looking for who it falls on. Randomized assignment is the clean case. With observational data you are betting you measured every confounder — a bet that has failed badly even with thousands of features.

  2. Build the feature table as of assignment day. Purchase history, tenure, and past behavior qualify; information generated after treatment assignment does not.

  3. Start with a regression that interacts treatment with your features. It is transparent, takes one line, and may be enough when the decision needs only broad groups.

  4. Fit an honest causal forest — causal_forest() in R’s grf, CausalForestDML in Python’s econml — and pull an estimated effect and a confidence interval for every user.

  5. Validate the estimates before presenting or using them. The calibration coefficient should sit near 1 and be significant. The best linear projection should show associations with features a domain expert recognizes. The RATE should be positive with a confidence interval excluding zero. Build the ranking on one half of the data and evaluate it on the other so that evaluation does not reuse the data used to choose the ranking. Pick AUTOC when you expect a small group to respond, Qini when you expect a weak response spread across everyone. GATES should show a top bucket standing clear of the rest. Propensities should stay away from 0 and 1. After randomly shuffling treatment labels and refitting the forest, you should find little evidence of differences in treatment effects across users. When you refit using the actual treatment labels on a smaller random sample, check whether the same user profiles still appear to benefit more.

  6. If the checks detect no clear heterogeneity, look at what your confidence interval covers before concluding everyone responds the same. An interval on the calibration coefficient wide enough to include both 0 and 1 means you learned nothing, not that there was nothing to learn.

  7. Turn effects into money. Subtract the cost of treating from each user’s estimated effect. Sort by what’s left. Walk down the list until the budget runs out or the next user costs more than they return.

  8. If you need a rule people can read, learn a shallow policy tree — and check that your package searches every possible tree rather than picking splits greedily one at a time. A greedy search on our data recommended treating nobody.

  9. Launch against a standing holdout. Keep a randomized group on business-as-usual so you can measure the rule’s incremental value, and refit at regular intervals as customers, seasons, and behavior change.

  10. Specify the constraints before deployment: exclusion lists, frequency caps, margin floors, and a fairness audit of which groups the policy excludes.

If you want more practice before moving on, the Mixtape session: Machine learning and heterogeneous effects labs walk through heterogeneous-effects and policy-learning exercises end to end on real data, and the grf package vignettes include hands-on examples of causal_forest, best_linear_projection, and rank_average_treatment_effect on benchmark datasets. Reproduce the coupon analysis, then change the true effect function in the data-generating script and compare how the forest and OLS recover the new pattern. For a formal derivation, Appendix 12.A shows how a splitting rule built for predicting outcomes is rewritten to estimate treatment effects.

This chapter closes Part II. You can now estimate a credible average effect and credible differences in effects across user profiles. Part III covers the checks and decisions that follow estimation. The first chapter examines the credibility of your results using placebo tests, negative controls, balance and overlap checks, and sensitivity analysis for confounders you cannot observe (Chapter 13).

Then we translate a statistically significant effect into a forecast of financial return that accounts for effect decay, gradual adoption, cannibalization, capacity constraints, and uncertainty (Chapter 14). Finally, we examine methods beyond those developed so far: structural causal models, causal discovery, Bayesian methods, treatments that change over time, spillovers between users, and the full theory of policy learning introduced by this chapter’s policy tree (Chapter 15). Across these methods, identification determines whether an estimate is causal; more flexible estimation cannot substitute for it.

Appendix 12.A: From predicting outcomes to predicting treatment effects

Earlier in the chapter I described a prediction tree as looking for cuts that make each box internally similar, and a causal tree as looking for cuts that make the effect estimates as different as possible across boxes. Those read like opposite instructions. For the same target and the same sample, reducing variation within leaves is equivalent to increasing variation between leaf averages. Prediction trees target outcomes, while causal trees target treatment effects. They can therefore favor different cuts.

The derivation has four steps: define the target in each problem, identify the unobserved quantity in the causal problem, replace it with something observable, and rewrite the criterion. It uses standard results (Athey and Imbens 2016; Wager and Athey 2018; Athey et al. 2019) and follows the presentation in the Mixtape session on heterogeneous effects linked in the wrap-up.

Two problems with the same shape

Both problems target a conditional average and use squared error to assess predictions. Prediction targets the expected outcome for a user with features \(x\), while causal estimation targets the expected effect for users with those features. Writing \(m\) for the outcome model the chapter introduced in Section 12.6.5, the two targets are:

\[ \begin{aligned} m(x) &= \mathbb{E}\!\left[\, Y_i \mid X_i = x \,\right] \\[0.5em] \tau(x) &= \mathbb{E}\!\left[\, \tau_i \mid X_i = x \,\right] \end{aligned} \tag{12.1}\]

Each target can also be defined as the value that minimizes squared error, which is the form needed here. Ask what single number you would guess for a whole group of lookalike users if you were graded on squared error. The answer is their average, so each target in Equation 12.1 is also the solution to a minimization:

\[ \begin{aligned} m(x) &= \operatorname*{arg\,min}_{a \in \mathbb{R}} \; \mathbb{E}\!\left[\left(Y_i - a\right)^2 \mid X_i = x\right] \\[0.8em] \tau(x) &= \operatorname*{arg\,min}_{a \in \mathbb{R}} \; \mathbb{E}\!\left[\left(\tau_i - a\right)^2 \mid X_i = x\right] \end{aligned} \tag{12.2}\]

The squared-error framework is the same; only the estimand changes. A tree inherits that framing: every leaf groups lookalike users and stores their average.

The column that does not exist

The trouble is in the training data. To fit the prediction problem you need pairs \(\{Y_i, X_i\}\), and every one of those is sitting in your table. To fit the causal problem you would need pairs \(\{\tau_i, X_i\}\) — and \(\tau_i = Y_i(1) - Y_i(0)\) is the individual treatment effect from Chapter 12’s opening, which is unobserved for every user. Each user is treated or not, never both.

The target and squared-error criterion still apply. Unlike an ordinary prediction tree, however, the causal tree cannot be trained directly because each user’s treatment effect \(\tau_i\) is unobserved.

Splitting the effect into two observable pieces

Instead of trying to observe \(\tau_i\) directly, we estimate two conditional average outcomes. Group users by their features, estimate average outcomes separately among treated and untreated users, and take the difference:

\[ \begin{aligned} \mathbb{E}\!\left[\tau_i \mid X_i\right] &= \mathbb{E}\!\left[Y_i(1) - Y_i(0) \mid X_i\right] \\[0.8em] &= \mathbb{E}\!\left[Y_i \mid X_i, D_i = 1\right] - \mathbb{E}\!\left[Y_i \mid X_i, D_i = 0\right] \end{aligned} \tag{12.3}\]

Both terms on the last line are ordinary conditional means of observed outcomes. The second equality requires three causal assumptions introduced earlier: consistency and no interference, so a treated user’s recorded outcome equals \(Y_i(1)\); conditional exchangeability, \(\left(Y_i(0), Y_i(1)\right) \perp D_i \mid X_i\), the conditional independence assumption from Section 6.4.2; and positivity, so both treatment groups exist within every feature profile (Section 6.4.3).

The independence assumption concerns treatment assignment: the potential outcomes must be independent of \(D_i\) given \(X_i\). They can still depend on \(X_i\), as they do in our coupon simulation, where effects vary with recency, frequency, and discount sensitivity. That variation is what we want to estimate. If the potential outcomes were independent of \(X\), no feature would carry information about the effect and there would be no heterogeneity to find. Conditioning on \(X\) allows us to compare treated and untreated users with the same features while retaining differences across feature profiles.

Why the criterion turns into a maximization

Imagine we could observe each user’s treatment effect. We would predict each effect with the average in its leaf. Minimizing squared prediction error would then be equivalent to maximizing the spread of the leaf averages, with each leaf weighted by its number of users. The equivalence requires two conditions.

First, each leaf’s predicted value is the mean of the effects inside it, \(\bar\tau_\ell = \frac{1}{n_\ell}\sum_{i \,:\, X_i \in \ell} \tau_i\), not some arbitrary constant we are free to choose. Second, we are comparing candidate partitions of the same parent sample, so the number of observations and their overall mean are held fixed while only the cut moves.

Before the algebra, distinguish the tree’s piecewise-constant target from the true CATE. A tree predicts one constant per leaf, not \(\tau(x)\) directly. Write \(\ell(x; \Pi)\) for the leaf that profile \(x\) falls into under partition \(\Pi\), and give the tree’s piecewise-constant target its own symbol:

\[ \tau_\Pi(x) = \mathbb{E}\!\left[\, \tau_i \mid X_i \in \ell(x; \Pi) \,\right] \tag{12.4}\]

The derivation distinguishes four quantities. \(\tau(x)\) is the true CATE. \(\tau_\Pi(x)\) is the population average effect within a leaf — a coarser target than \(\tau(x)\) that changes when the partition changes. The leaf mean \(\bar\tau_{\ell(i)}\) is the finite-sample oracle quantity: the sample analogue of \(\tau_\Pi(x)\). It averages the individual \(\tau_i\), so it is unobservable too.

The fourth quantity is the operational leaf estimate, \(\hat\tau_\ell = \hat\mu_1(\ell) - \hat\mu_0(\ell)\): the average outcome among treated users in the leaf minus the average among controls. For a fixed leaf, the random assignment in our coupon trial makes this difference an estimate of the leaf’s average effect: every user has the same chance of receiving the coupon. In observational data, treated and untreated users can still have different feature profiles inside the same leaf. The assumptions behind Equation 12.3 justify comparisons at the same \(X_i\); pooling different profiles into one leaf may require further adjustment, such as propensity-score weighting (Athey and Imbens 2016). The derivation uses the oracle quantity to show the form of the criterion; the estimator makes the criterion usable.

This is why Athey and Imbens (2016) include \(\Pi\) in their notation, writing \(\mu(x; \Pi)\) and \(\hat\tau(X_i; S^{tr}, \Pi)\) rather than only \(\mu(x)\) and \(\tau(x)\). The tree’s target moves when the cuts move; the true CATE does not. The same distinction applies on the prediction side, where \(m_\Pi(x)\) is the leaf-constant approximation to \(m(x)\).

Expand the squared error one leaf at a time, using the leaf means an oracle tree would store if the individual effects were observable. The cross-term collapses because \(\bar\tau_\ell\) is the leaf mean, and the expansion becomes:

\[ \sum_{i} \left(\bar\tau_{\ell(i)} - \tau_i\right)^2 = \sum_{i} \tau_i^2 \;-\; \sum_{\ell \in \Pi} n_\ell \, \bar\tau_\ell^{\,2} \tag{12.5}\]

The first term on the right does not mention the partition \(\Pi\) at all. Move the cut wherever you like and \(\sum_i \tau_i^2\) is unchanged, so it cannot affect which split wins. Minimizing the left side is therefore the same problem as maximizing \(\sum_\ell n_\ell \bar\tau_\ell^{\,2}\), the leaf-size-weighted sum of squared leaf effects.

The criterion in Athey and Imbens (2016) already subtracts the squared-outcome term:

\[ -EMSE_\mu(\Pi) = -\mathbb{E}\!\left[(Y_i - \mu(X_i;\Pi))^2 - Y_i^2\right] - \mathbb{E}\!\left[(\hat\mu - \mu)^2\right] \]

Because every candidate partition shares the dropped term, removing it does not change their ranking. The same reasoning lets us discard the unobservable \(\sum_i \tau_i^2\) in the treatment-effect case.

Two cautions about Equation 12.5. It is equivalent to maximizing between-leaf dispersion, but the quantity \(\sum_\ell n_\ell \bar\tau_\ell^{\,2}\) is not literally that dispersion: it differs from the centered \(\sum_\ell n_\ell (\bar\tau_\ell - \bar\tau_{\text{parent}})^2\) by the fixed amount \(N \bar\tau_{\text{parent}}^{\,2}\). The two quantities therefore rank the candidate splits in the same order. Wager and Athey (2018) use a related rule based on variance. The equivalence above is exact for oracle leaf means, which average the unobserved individual effects. For estimated treated-minus-control gaps, the overall average can change when the cut moves, even after weighting leaves by size. The weighted sum of squared estimates can then favor a different cut from the weighted variance, which measures spread around that changing average. And when you see this written as \(\max \sum_i \bar\tau_{\ell(i)}^{\,2}\), summing over users rather than leaves, the leaf sizes are already in there — every one of the \(n_\ell\) users in leaf \(\ell\) contributes the same \(\bar\tau_\ell^{\,2}\).

The same argument on the prediction side produces the same criterion shape. Athey and Imbens (2016) write conventional CART as:

\[ -MSE_\mu(S^{tr}, S^{tr}, \Pi) = \frac{1}{N^{tr}}\sum_{i \in S^{tr}} \hat\mu^2(X_i; S^{tr}, \Pi) \]

The prediction criterion has the same form as the causal criterion in Equation 12.6 below. For a fixed sample, total variation is constant, so minimizing variation within leaves is equivalent to maximizing variation between leaves. For the same target, the instructions “Make the boxes internally similar” and “make the boxes differ from each other” therefore lead to the same ranking of splits.

Three criteria, not one

The equivalence above is an oracle result: it still has \(\tau_i\) in it, so it motivates a splitting rule rather than describing one you can run. Athey and Imbens (2016) separate three objects that are easy to blur together.

The infeasible criterion evaluates a partition using individual effects we cannot observe. The left side of Equation 12.5 is a simplified example: a finite-sample oracle loss whose ranking of partitions is unchanged when partition-invariant terms are removed. Athey and Imbens (2016)’s formal criterion, \(MSE_\tau(S^{te}, S^{est}, \Pi)\), keeps test and estimation samples separate, but it is infeasible for the same reason: the individual effects are unobserved. The same argument about removing partition-invariant terms applies. In prediction trees, the corresponding criterion can be computed directly because outcomes are observed; causal trees need a substitute for individual effects. The adaptive criterion replaces the missing effects with estimates and is what a plain causal tree maximizes:

\[ \begin{aligned} &-\widehat{MSE}_\tau(S^{tr}, S^{tr}, \Pi) \\[0.6em] &\qquad = \frac{1}{N^{tr}} \sum_{i \in S^{tr}} \hat\tau^2(X_i; S^{tr}, \Pi) \end{aligned} \tag{12.6}\]

This is the “maximize the sum of squared effect estimates” rule in operational form. But it rewards differences caused by sampling noise as much as differences caused by treatment-effect heterogeneity. The honest criterion subtracts a penalty for the uncertainty in the leaf estimates:

\[ \begin{aligned} &-\widehat{EMSE}_\tau(S^{tr}, N^{est}, \Pi) \\[0.6em] &\qquad = \frac{1}{N^{tr}} \sum_{i \in S^{tr}} \hat\tau^2(X_i; S^{tr}, \Pi) \\[0.6em] &\qquad\quad - \left(\frac{1}{N^{tr}} + \frac{1}{N^{est}}\right) \sum_{\ell \in \Pi} \left( \frac{S^2_{treat}(\ell)}{p} + \frac{S^2_{control}(\ell)}{1 - p} \right) \end{aligned} \tag{12.7}\]

\(S^{tr}\) is the sample used to choose the splits, and \(N^{est}\) is the size of the sample used to estimate the leaf effects. \(S^2_{treat}(\ell)\) and \(S^2_{control}(\ell)\) are the within-leaf variances of the outcome among treated and control users in leaf \(\ell\). They are computed on the splitting sample \(S^{tr}\) so the criterion can anticipate uncertainty in the estimation sample without using its outcomes. The treatment probability \(p\) is constant here because this version assumes a randomized experiment like our coupon trial. The compact penalty treats each leaf as containing about the same share of the splitting sample and the estimation sample. It also treats the fraction of treated users within each leaf as approximately \(p\). Observational applications need the propensity-score adjustment Athey and Imbens (2016) discuss instead of a single \(p\). In their own words, the criteria “reward a partition for finding strong heterogeneity in treatment effects and penalize a partition that creates variance in leaf estimates.”

Note which way the logic runs, because the chapter’s honesty section is easy to read backwards. Honesty is the estimation design — split the rows, choose the tree on one half, measure the leaves on the other. The criterion in Equation 12.7 is then derived under that design, and the penalty is what falls out of doing the accounting properly. The criterion anticipates honesty; it is not forced into existence by it.

The minimum-leaf-size condition also depends on the method. In the double-sample causal trees of Wager and Athey (2018), every leaf must hold at least \(k\) estimation-sample observations from each treatment arm. A prediction tree has no per-arm requirement — it needs enough rows. A causal tree needs enough rows on both sides of the comparison, since a leaf with no treated users has no treated-minus-control gap to report. That is the per-treatment-arm version of the minimum-leaf-size rule from the chapter’s tree-building section.

From a causal partition to a causal forest

A causal forest differs from two outcome forests fit separately to treated and untreated users. The following four constructions make that distinction clear:

  • A T-learner fits two separate outcome models, one per arm, and subtracts their predictions. The two models are fitted independently, so they can learn different partitions of the feature space. This is the estimator the chapter introduced among the meta-learners.
  • A causal tree builds one shared partition. Inside each leaf it estimates one mean outcome among the treated observations and one among the controls, then subtracts them to get that leaf’s effect estimate (Athey and Imbens 2016). One partition, two arm-specific means per leaf — not two trees.
  • The original causal forest grows an ensemble of causal trees and aggregates their treatment-effect estimates, with the treated-minus-control leaf mean as the within-leaf estimator (Wager and Athey 2018).
  • The generalized random forest, used by causal_forest, retains recursive partitioning, subsampling, and random feature selection. It residualizes the outcome and treatment, uses gradient-based pseudo-outcomes computed from those residuals to choose splits, and estimates each user’s CATE by applying forest weights in a local estimating equation rather than taking a plain average of tree-level estimates (Athey et al. 2019).

The generalized random forest combines the chapter’s two explanations. For each tree used to predict Maria’s effect, take the users in her leaf whose outcomes were set aside for estimating effects. Divide a total weight of one equally among them; all other users get zero weight from that tree. Average each user’s weight across those trees to form Maria’s comparison neighborhood. The residualized outcome and treatment from Section 12.6.5 help score possible splits and also enter the final effect calculation. The weights determine how much each user’s data contributes to that calculation.

Both R and Python use the approach just described: grf::causal_forest documents it in its algorithm reference, and Python’s CausalForestDML pairs the same residualization with a generalized-random-forest final stage built on honest trees and local moment equations. Their codebases, APIs, and methods for quantifying uncertainty differ; econml documents a bootstrap-of-little-bags method for its intervals. On this chapter’s simulated data the two reach the same substantive conclusions, which is the claim the validation table makes; their estimates and diagnostics are not interchangeable.

Two qualifications complete the appendix. First, this family of forests draws each tree’s sample without replacement. This differs from classical bagging, which resamples with replacement, and contributes to the justification for confidence intervals — alongside honesty, a condition on how fast the subsample grows with the data, and regularity conditions on the trees themselves (Wager and Athey 2018). Subsampling alone does not guarantee valid intervals. Under the stated assumptions, the inference result concerns one specified feature profile at a time — what statisticians call pointwise inference. It does not guarantee that the intervals for all profiles cover their true CATEs together (Section 12.7). Second, the two terms in Equation 12.7 reward different kinds of feature, and either type can improve the causal tree’s criterion. The first term averages squared effect estimates, so sampling noise can change its value even when the average effect is the same in both child leaves. A feature that predicts profit but not the coupon’s effect adds no real effect differences, yet it can still change this estimated term. The penalty reflects how uncertain the leaf estimates are. A split can lower it by making outcomes more alike among treated users and among controls, but only if that gain outweighs the loss of precision from having fewer users in each leaf. A split on a pure outcome predictor can therefore improve the criterion even when both children have the same treatment effect — Athey and Imbens (2016) make this point directly, and note it is why the gap between the adaptive and honest criteria matters more for treatment effects than it ever did for prediction.

So the two terms can favor overlapping but distinct sets of features. A strong outcome predictor need not be a strong effect predictor, though it can still earn its place by making the effect estimates more precise. In our simulation, past_spend predicts profit without the coupon, but it does not enter the formula that generates the coupon’s effect. It also receives the lowest importance score in Figure 12.8. Predicting outcomes can still help a causal tree estimate effects more precisely.


  1. As Google advises, “NotebookLM can be inaccurate; please double-check its content”. I recommend reading the chapter first, then listening to the audio to reinforce what you’ve learned.↩︎

  2. A heads-up for when you read causal-forest documentation: many implementations call the treatment W, following the notation in that literature. Same variable, but I’ll stick with the \(D\) this book has used since Chapter 2.↩︎

  3. These first three variables form what is commonly called RFM, which stands for Recency, Frequency, and Monetary value. It is widely used in customer segmentation and machine learning to summarize customer behavior: when they last engaged, how often they engage, and how much value they bring.↩︎

  4. To load it directly, use the example URL https://raw.githubusercontent.com/RobsonTigre/everyday-ci/main/data/coupon_allocation_experiment.csv and replace the filename coupon_allocation_experiment.csv with the one you need.↩︎

  5. This is the “fundamental problem of causal inference” from Chapter 1: we see at most one of the two potential outcomes for any user.↩︎

  6. Some people may promise to estimate the effect on this specific user, i.e. the individual treatment effect (ITE). Although I understand why that target is attractive, I have not yet seen compelling reasons to believe it can be estimated reliably in realistic scenarios.↩︎

  7. Regularization penalizes model complexity: a feature’s influence is reduced or removed unless it improves prediction enough to offset the penalty. Ridge regression shrinks coefficients, lasso can set them to zero, and gradient boosting, random forests, and neural networks impose complexity controls in different ways.↩︎

  8. If that sounds like the opposite of what a prediction tree does — earlier I said it keeps the cut that makes each box most internally similar — the connection holds when we compare the same target on the same sample. Reducing variation inside the boxes is then equivalent to increasing variation between their averages. Prediction trees target outcomes, while causal trees target treatment effects, so they can choose different cuts. Appendix 12.A shows the algebra, along with the conditions it needs and the difference between the rule a causal tree would like to use and the one it can actually compute.↩︎

  9. The residualize-then-estimate logic generalizes beyond forests. The DoubleML package, available in R and Python, is a separate implementation of the broader double/debiased machine learning framework — not the CausalForestDML estimator this chapter’s Python code uses — and a natural next stop if you want a DML-centered workflow for ATEs, GATEs, CATEs, and policy rules; see the DoubleML heterogeneity guide.↩︎

  10. In grf, the regression is \(\tilde Y_i = \alpha\,\bar\tau\,\tilde D_i + \beta\,\big(\hat\tau(X_i) - \bar\tau\big)\,\tilde D_i + \varepsilon_i\), fit by OLS with no intercept and heteroskedasticity-robust standard errors, with one-sided p-values. test_calibration prints \(\alpha\) as mean.forest.prediction and \(\beta\) as differential.forest.prediction. This is the best linear predictor of Chernozhukov et al. (2023) — the same paper behind GATES below — written in residualized form: they weight by \(1/\big(\hat e(x)(1-\hat e(x))\big)\) and fit the CATE proxy on a separate split, while here the forest supplies the proxy and residualizing handles the propensity.↩︎

  11. Standardize only the projection covariates — best_linear_projection(cf, scale(X)). The forest’s own features can stay raw: a tree split depends only on how feature values are ordered, so standardizing them would change nothing.↩︎

  12. That correlation is a simulation-only luxury: real data has no planted truth to check against, so you lean on the calibration coefficient, which is computable anywhere.↩︎

  13. That is grf’s statistic, and it is what Figure 12.8 plots. econml’s feature_importances_ measures something related but different: how much treatment-effect heterogeneity each split creates, also depth-weighted. The companion scripts report each measure under its own name and save them as separate figures.↩︎

  14. The arithmetic, for two pre-defined halves of an experiment with \(n\) users: each half’s effect is estimated on \(n/2\) users, so its standard error is \(\sqrt{2}\) times the full-sample ATE’s; differencing the two halves’ effects multiplies it by \(\sqrt{2}\) again — twice the ATE’s standard error in total. Twice the standard error costs four times the sample to undo (Chapter 5), and a target half the size costs another four: \(4 \times 4 = 16\). The 16 is a planning default, not a law: it leans on the half-size assumption — a reasonable starting point, no more — and on the standard coding of the interaction term.↩︎

  15. For budgets with per-user costs, or several competing offers at once, this curve generalizes to constrained assignment: pick the best action per user under a cost constraint, often by solving a small optimization problem. The example here is the simplest case, with a single coupon and a single budget.↩︎

  16. This is also why I reach for mcf rather than the more familiar econml.policy.PolicyTree in Python. A greedy tree-builder fixes one split before looking at the next, so it can miss a rule like the recency-and-frequency rule shown below, where neither feature looks useful until you split on both. On this data, econml’s greedy policy tree recommends treating no one. R’s policytree considers every eligible split and finds the real optimum. Python’s mcf searches exhaustively over a grid of candidate split points per feature — not every raw value — and lands on nearly the same rule. Greedy learners are fine for many problems; just confirm your tool searches exhaustively rather than greedily before trusting a shallow policy rule.↩︎