graph LR
Recs[Recommendations] --> Spend[Total spent]
2 Biases, causal frameworks, and causal estimands
Keywords
A/B Testing, Causal Inference, Causal Time Series Analysis, Data Science, Difference-in-Differences, Directed Acyclic Graphs, Econometrics, Impact Evaluation, Instrumental Variables, Heterogeneous Treatment Effects, Potential Outcomes, Power Analysis, Sample Size Calculation, Python and R Programming, Randomized Experiments, Regression Discontinuity, Treatment Effects
2.1 Data alone isn’t enough for causal questions
Imagine you’re surrounded by data on clicks, sales, and user sessions. It may seem sufficient to answer all sorts of causal questions. But “data itself is profoundly dumb”: it can reveal patterns, not the “why” behind them. No matter how large your dataset, you won’t answer causal business questions without a causal model (Pearl and Mackenzie 2018).
I like to illustrate this limitation with the tale from Taleb (2007). Imagine a turkey fed by a butcher every day for a thousand days. Every single data point reinforces the statistical model that the butcher is a benevolent guardian of the turkey. The trend is undeniable, and the confidence intervals tighten as this conclusion becomes more credible. But on the Wednesday before Christmas2, the turkey undergoes a sharp “revision of belief”.
This story exposes the problem with relying on purely ‘data-driven’ approaches. Data alone cannot distinguish between a caregiver and a predator waiting for the right moment. Without a framework to interpret it, data is only a record of the past; it is blind to what could happen and what could have happened.
Tip💻 Want to run the code? Set up your environment: click here
Data: All data we’ll use are available in the repository. You can download it there or load it directly by passing the raw URL to read.csv() in R or pd.read_csv() in Python.3
Packages: To run the code yourself, you’ll need a few tools. The block below loads them for you. Quick tip: you only need to install these packages once, but you must load them every time you start a new session in R or Python.
# If you haven't already, run this once to install the package
# install.packages("tidyverse")
# You must run the lines below at the start of every new R session.
library(tidyverse) # The "Swiss army knife": loads dplyr, ggplot2, readr, etc.# If you haven't already, run this in your terminal to install the packages:
# pip install pandas numpy statsmodels (or use "pip3")
# You must run the lines below at the start of every new Python session.
import pandas as pd # Data manipulation
import numpy as np # Mathematical computing in Python
import statsmodels.formula.api as smf # Linear regressionConsider a marketplace app that launches a new recommendation system to suggest items based on each user’s preferences. As Figure 2.1 shows, users who see more recommendations also spend more R$ per session. A tempting explanation is that better recommendations help users find what they want and buy it.
Your director asks you to use data from 800 clients to estimate how the new recommendation system affects their total spending. You start with the simple linear regression from Chapter 1, shown in Equation 2.1. The code tab provides the R and Python code.
\[ \text{total spent} = \beta_0 + \beta_1 \times \text{recommendations shown} + \varepsilon \tag{2.1}\]
We find that \(\hat{\beta_0}=80.03\) and \(\hat{\beta}_1 = 6.12\). The intercept \(\hat{\beta_0}\) suggests that a user shown zero recommendations is expected to spend R$80.03 on average. The coefficient \(\hat{\beta}_1\) suggests that each additional recommendation shown increases average spending by R$6.12, a result that is statistically significant at the 1% level. This seems intuitive: the more recommendations we show users, the more they spend, with the former interpreted as the cause of the latter.
But wait, there is more to this story!
Recall from Chapter 1 that this interpretation holds only if our model is close to the true model known to the all-knowing God (i.e., if it captures how the world actually works). A colleague experienced in causal inference suggests plotting recommendations shown against users’ previous engagement.4 Figure 2.2 shows this relationship.
The second plot colors each dot by the user’s engagement before the recommendation system launched: red indicates lower engagement (fewer minutes in the app), and green indicates higher engagement (more minutes in the app).5
Now we begin to suspect that lower previous engagement (represented by red-ish dots) is correlated with lower recommendations shown to these users, and lower total R$ spent by these users, while higher previous engagement (represented by green-ish dots) is correlated with higher recommendations shown to those users, and higher total R$ spent by those users. Let’s run the multiple regression including both variables, as in Equation 2.2, to confirm this:
\[ \text{total spent} = \beta_0 + \beta_1 \times \text{recommendations shown} + \beta_2 \times \text{engagement} + \varepsilon \tag{2.2}\]
Once we control for engagement, the estimated effect of showing an additional recommendation falls from R$ 6.12 to R$ 0.61. Independent of recommendations, each additional minute a user spends on the app is associated with about R$ 14.49 more in total spending. Without that control, the model makes the algorithm look far more effective because it is mostly “riding along” with users who already love your platform.
These kinds of misleading, overly optimistic conclusions like the one you initially reached are common in large datasets. Behavioral economists call this the illusion of causality: our brains are naturally inclined to find order in randomness and to see patterns, even when none truly exist. When two variables that seem plausibly connected move together, it’s all too easy to jump to causal conclusions, sometimes with disastrous business consequences (Kahneman 2011).
This is a textbook case of omitted variable bias (OVB). Leaving engagement out of Equation 2.1 hides its role, which becomes visible when we include it in Equation 2.2. The rest of the book develops methods for estimating causal relationships without falling into this trap.
2.2 The anatomy of omitted variable bias
The statistical concept of bias
In statistics and causal inference, bias is a systematic error that pushes an estimate away from the true effect. Unlike a random mistake, it consistently distorts the result.
These errors may arise from who enters your sample (selection bias), what you fail to measure (omitted variable bias), who drops out (attrition bias), or a mistaken direction of influence (reverse causality and simultaneity). The next sections examine omitted variable bias; Appendix 2.A covers the other biases and misleading relationships.
The anatomy of OVB
A causal diagram, also called a Directed Acyclic Graph (DAG), makes the problem visible. A DAG maps assumptions about causal relationships: variables are nodes, and arrows show the direction of influence. Here, the DAG shows why the correlation between recommendations and spending misled us.6
In Figure 2.3, we see what we thought we were measuring: a direct path from recommendations to spending. But Figure 2.4 shows the reality: engagement is a confounder that creates two pathways to higher spending. Highly engaged users naturally see more recommendations (because they spend more time in the app), and they also spend more money directly (because engaged users simply buy more).
graph LR
Eng[Engagement] --> Recs[Recommendations]
Eng[Engagement] --> Spend[Total spent]
Recs[Recommendations] --> Spend[Total spent]
This creates a misleading correlation. When we ignore engagement in our first regression, we accidentally attribute all of its effect to recommendations. It’s like crediting a rooster’s crow for causing the sunrise — they happen together, but one doesn’t cause the other.
In this example, we could see “engagement” in our data table — we simply forgot to include it in our regression. But omitted variable bias can be much more insidious. The missing variable might be something we can’t directly observe or measure, like a customer’s true satisfaction level, their underlying propensity to spend, or their personal financial situation. These unobservable factors can still create the same misleading patterns, making it appear that recommendations drive spending when the real driver is something we never measured at all.7
Omitted variable bias is particularly dangerous in practice because the confounder may be present but overlooked, or it may be completely unobserved.
WarningDon’t panic: I’ll translate the math
If Greek letters give you flashbacks to high school calculus, don’t worry. We will translate every term below into plain English.
The goal is to understand where the bias comes from and how to remove it, not to master the proof. If you prefer intuition to derivation, you can skip to the “Key insight on OVB” callout.
Suppose the true model is the “long regression” (Equation 2.2), which includes engagement. We instead estimate the “short regression” (Equation 2.1), which omits engagement because it is unavailable or overlooked. Econometric theory gives us a famous result that explains exactly why our result was wrong. I show this in Equation 2.3.8. The symbol \(\mathbb{E}\) stands for “Expected Value” — stat-speak for the average. It indicates that this result holds if we run this regression in several different samples and take the average of the estimated effects we found.
\[ \begin{aligned} &\textcolor{#f77a05}{\text{average values of } \hat{\beta}_1} = \textcolor{#f77a05}{\mathbb{E}[\hat{\beta}_1]} = \textcolor{#0572f7}{\beta_1} + \textcolor{#ff0585}{\text{Bias}} \\[1em] &\textcolor{#f77a05}{\mathbb{E}[\hat{\beta}_1]} = \textcolor{#0572f7}{\beta_1} + \left[ \textcolor{#f70505}{\left( \begin{array}{c} \text{Effect of engagement} \\ \text{on total spent} \end{array} \right)} \times \textcolor{#800080}{ \left( \begin{array}{c} \text{How recommendations and} \\ \text{engagement move together} \end{array} \right)} \right] \\[1em] &\textcolor{#f77a05}{\mathbb{E}[\hat{\beta}_1]} = \textcolor{#0572f7}{\beta_1} + \left[ \textcolor{#f70505}{\beta_2} \times \textcolor{#800080}{\frac{\text{Cov}(\text{recommendations shown}, \text{engagement})}{\text{Var}(\text{recommendations shown})}} \right] \\[1em] \end{aligned} \tag{2.3}\]
Read the equations from top to bottom: each line follows from the one above, and the colors track the same terms throughout. Now we can translate the last row of Equation 2.3 into plain English, term by term:9
\(\mathbb{E}[\hat{\beta}_1]\): This is the expected value, the average, of what our “short” model spits out as a coefficient for recommendations. It’s the average estimated effect we would obtain from running this regression in several different samples.
\(\beta_1\): This is the true causal effect of recommendations on spending. This is known by the gods of causal inference; we want to find it but can’t see it directly.
\(\beta_2\): This is the true effect of the omitted variable (engagement) on spending.
Fraction: This represents the relative size of the relationship between the treatment and the omitted variable. In our case, it measures how strongly “recommendations shown” correlates with “engagement”.10
Here is the punchline: The estimate you get is a mix of the truth that you seek plus a bias term. And this bias term is the product of two things:
- The effect of the omitted variable on the outcome (red).
- The correlation between the omitted variable and your treatment (purple).
Because the bias is a product, it equals zero if either term is zero. Omitted Variable Bias (OVB) does not arise merely because a variable is missing; the omitted variable must be related to both the outcome (“Total Spent”) and the treatment (“Recommendations”).
In our example, both conditions for OVB hold:
- Engagement strongly drives spending, so \(\beta_2\) is positive and large.
- Engagement correlates with recommendations, so the purple term is positive.11
Because both terms were non-zero and positive, the simple regression overstated the effect of recommendations by attributing part of engagement’s effect to the algorithm. If engagement did not affect spending or was not correlated with recommendations, the omitted-variable bias term would be zero.
ImportantKey insight on OVB
OVB only occurs when the variable you leave out of your model is related to both the outcome you’re measuring (“total spent”) and the treatment variable you’re studying (“recommendations shown”). If the omitted variable only affects one or the other — but not both — then leaving it out won’t bias your estimated effect.
This rule has important exceptions, which we cover in Chapter 6. If the covariate is part of the mechanism (what we call a mediator) or an effect of both treatment and outcome (what we call a collider), adding it to the model can introduce bias or remove the effect you are trying to measure.12
When we cannot measure or control for every confounder, we need other ways to estimate causal effects. Part II develops these methods.
The common strategy is to find variation in treatment that is unrelated to the unobserved factors affecting the outcome: assignment that is “as good as random” even when it is not literally random.
Randomized experiments create this variation directly by breaking the link between treatment assignment and unobserved confounders. When experiments are impossible, we look for naturally occurring assignment processes that mimic randomization or use methods designed to isolate the causal effect despite confounding.
2.3 Selection bias: when “who gets what” ruins causal answers
Sometimes the problem starts before we fit the model: the mechanism that determines who receives treatment is itself biased. That is selection bias, a cousin of omitted variable bias.
Both stem from differences between treated and control groups that also affect the outcome. But while OVB often happens because we omit a confounder in our model, selection bias happens upstream — when the process of deciding who gets the treatment effectively rigs the game by favoring those who would perform either better or worse than average anyway. And very often, one leads to the other: selection bias creates the conditions for OVB when you try to estimate causal effects without properly modeling the selection process, or when the selection relies on hidden factors you simply cannot measure.
Example: let’s get back to our new recommendation system example, but this time suppose the product team launches it only for users who have logged in at least 5 times in their first month. They want to “reward” active users and make sure the new feature is seen by people most likely to use it. When your manager asks for a report on the system’s impact, you might be tempted to simply compare total R$ spent between users with and without the new system.
Users with high engagement and lots of logins are selected into the treatment group. They differ from less engaged users, and those same differences — like being more active or more interested in shopping — also make them likely to spend more, regardless of the new system.
Running total spent ~ recommendation system enabled will produce a positive estimate, but that estimate mixes the effect of high engagement with the effect of access to the new feature. The groups differ before treatment, so the comparison reflects selection bias.
Selection bias arises in the data generation process: the treated and untreated groups were not comparable before treatment. Omitted variable bias arises in the statistical model when it fails to adjust for those differences.
2.4 The potential outcome model
In the recommendation example, including engagement in the regression corrected the omitted variable bias. But a confounder may be impossible to observe or measure, such as a customer’s true satisfaction level, underlying propensity to spend, or personal financial situation.
Economists, statisticians, philosophers, and computer scientists developed causal frameworks to reason about a true model we cannot observe. The most widely used framework in my field is the potential outcomes model (Imbens and Rubin 2015), which asks us to consider alternative realities.
The potential outcomes model allows us to compare what would happen to the same unit with and without treatment — a “what if” machine for causal reasoning. The terminology (treatment, control) has roots in medicine and agriculture, but the logic applies universally. For the rest of this chapter, we’ll assume a simple binary treatment: you either get it or you don’t.13 Figure 2.5 illustrates the idea with a marketing campaign example.
Of course, this exercise is purely hypothetical, but it illustrates a key insight: if we could somehow observe both potential outcomes for the same unit, causal inference would be straightforward. We would simply calculate the difference between these two scenarios to determine the true causal effect of the campaign.
However, this is precisely what makes causal inference challenging in practice. In the real world, we face what’s known as the “fundamental problem of causal inference”: for any given unit, we can only observe one potential outcome - either the treated state or the untreated state, but never both simultaneously. When a company implements a global marketing campaign, we cannot simultaneously observe what would have happened to its sales if the campaign was not executed.
The potential outcomes framework makes the missing quantity explicit. Although we cannot observe a unit’s counterfactual directly, the framework guides us toward valid comparison groups that can approximate it.
Once we identify the missing counterfactual as the target, we can design studies and choose methods that construct a credible substitute for it.
NoteModel and notation
The notation may feel abstract at first. Read it as a compact language for the comparisons we will use throughout the book.
Before we go further, let’s lock in the vocabulary we’ll use throughout this book.
Treatment (denoted by the variable names \(T_i\) or \(D_i\)):14 It represents the intervention received or adopted by individuals, such as being exposed to a new recommendation algorithm, a discount coupon campaign, or a new product feature. In the binary treatment framework, it can take on the values \(0\) for those assigned to stay in the control group or \(1\) for those assigned to receive the treatment.
Treatment group (represented by \(D_i=1\)): The individuals assigned to receive the treatment. In the coupon example, this is the group assigned to receive the coupon campaign.15
Control group (represented by \(D_i=0\)): It denotes the group of individuals who did not receive the treatment. In the same coupon campaign example, the control group is the group of customers who did not receive the coupon campaign.
Potential outcome (denoted by \(Y_{i1}\) or \(Y_{i0}\)): It indicates the value of the outcome variable \(Y\) that an individual \(i\) would have in each scenario. For instance, it’s as if God knows beforehand what your outcome would be if you were treated (\(Y_{i1}\)) and what it would be if you were not treated (\(Y_{i0}\)). In the same example, it’s as if God knows beforehand what your total spending would be if you received the coupon campaign (\(Y_{i1}\)) and what it would be if you did not receive the coupon campaign (\(Y_{i0}\)).
Counterfactual (also denoted by \(Y_i(1)\) or \(Y_i(0)\)): It refers to the unobserved potential outcome or to “what would have happened to an individual under the alternative condition”. For example, if a customer received the coupon campaign (\(D_i=1\)), we would observe their outcome as \(Y_{i1}\), but their counterfactual would be \(Y_i(0)\) - what their spending would have been without receiving the coupon campaign. Conversely, if a customer didn’t receive the coupon campaign (\(D_i=0\)), we would observe their outcome as \(Y_{i0}\), but their counterfactual would be \(Y_i(1)\) - what their spending would have been if they had received it. The counterfactual is always the “what if” scenario we can’t observe; the “road not taken” for each individual customer.
Put simply, selection bias means the dice are loaded: who gets the treatment depends on traits that themselves predict the outcome. The assignment isn’t random — it reflects pre-existing differences.
2.4.1 Explaining bias through the potential outcomes framework
Suppose you work on the data team of a food-delivery app. To increase orders, the marketing team creates the “Super Promo Banner”, a large ad at the top of the home screen offering discounts. A business rule shows it only to users who don’t order much and never to users who already order often. According to the team, “this is a business rule” (we’ve all heard that one).
Using the purely hypothetical potential outcomes framework, suppose you can see each user’s orders both with and without the banner. Table 2.1 shows those outcomes:
| João (Low-orders user) |
Maria (High-orders user) |
|
|---|---|---|
| Did this person see the banner? \((D_i\)) | Yes \((D_{João}=1\)) | No \((D_{Maria}=0\)) |
| Orders this month (observed) (\(Y_i\)) | 3 (\(Y_{João}\)) | 8 (\(Y_{Maria}\)) |
| Orders if NOT shown banner (\(Y_{i0}\)) | 2 (\(Y_{João,0}\)) | 8 (\(Y_{Maria,0}\)) |
| Orders if shown banner (\(Y_{i1}\)) | 3 (\(Y_{João,1}\)) | 8 (\(Y_{Maria,1}\)) |
In this hypothetical world, where we could observe both potential outcomes, the true effect of showing the banner would be:
- Effect for João = \(Y_{João,1} - Y_{João,0}\) = 3 (with banner) - 2 (without banner) = 1
- Effect for Maria = \(Y_{Maria,1} - Y_{Maria,0}\) = 8 (with banner) - 8 (without banner) = 0
In practice, we never observe both potential outcomes for the same person. We cannot see what João’s orders would have been without the banner (\(Y_{João,0}\)) or what Maria’s orders would have been with it (\(Y_{Maria,1}\)). We observe only each user’s actual, observed outcome.
Many analysts then make a classic mistake: they compare João’s and Maria’s observed outcomes and call the difference the banner’s “effect”: \(Y_{João} - Y_{Maria} = 3 - 8 = -5\). A naive analyst may conclude that the banner reduces orders and build a persuasive story around the biased result, perhaps arguing that the banner is intrusive and harms the user experience.
Except in randomized experiments, people who get the “treatment” are usually selected for a reason, and that reason is often related to the outcome we’re measuring. That’s what creates confounding and bias in our causal effect estimates.
The negative “effect” is misleading because Maria would have placed more orders than João with or without the banner. João saw the banner precisely because he was a low-order user. This is selection bias: the groups differed before treatment.
Takeaway: Whenever “the treated, if they hadn’t been treated” are different from “the untreated as observed,” any naive difference in averages will mix bias into the true causal effect.
2.5 Main causal estimands
“I learned very early the difference between knowing the name of something and knowing something.”
— Richard P. Feynman (American physicist)
In causal inference, estimands are the core conceptual causal questions we want to answer. They are conceptual because they rely on comparing what would have happened to the same unit with and without treatment - but in reality, we can only observe one of these outcomes for each unit.
Due to the fundamental problem of causal inference presented above, it’s important to keep in mind that these estimands are conceptual in nature: they describe ideal comparisons we’d make if we could observe both potential outcomes for each unit.
Later, when we move from concept to data, our challenge will be to approximate these estimands as closely as possible using the right combination of sample data and research design.
Feynman’s point applies directly here. You’ll meet plenty of data scientists who can rattle off “ATE, ATT, ITT, LATE” like a catchy acronym, yet struggle to explain what each one actually measures or when to use which. That’s knowing names, not knowing things.
The sections below explain the most common estimands and the causal question each answers. Every example uses a new recommendation algorithm for a marketplace app, so you can focus on what each estimand captures and why you would choose it rather than memorizing labels.
Average Treatment Effect (ATE)
The ATE is the average difference between two hypothetical worlds for the population of interest: everyone receives the treatment in one, and no one receives it in the other.
For that reason, this estimand is particularly useful for answering questions like: “Should we roll this feature out to everyone in our user base?”
To make this clear: imagine a marketplace app testing a new recommendation algorithm on 100,000 users. The ATE is the difference between the average total spent if all users had used the new algorithm and the average total spent if all users had remained on the old one:
ATE = (Average total spent if all users used the new algorithm) − (Average total spent if all users used the old algorithm)
Average Treatment Effect on the Treated (ATT)
The ATT focuses on the average effect of the treatment only for those who actually received it - often by their own choice. The conceptual comparison is between the observed outcomes of the treated users and the outcomes they would have had if they had not been treated.
This estimand answers questions such as: “How effective is our feature for users who actually adopt it?” It is especially relevant when adoption is voluntary, as it often is in digital products.
To make this clear: suppose only 60% of users chose to use the new algorithm. The ATT measures the difference between the total spent of those users and the total spent they would have had if they had continued using the old algorithm:
ATT = (Average total spent for algorithm users) − (Average total spent those same users would have had without the algorithm)
Intention-to-Treat Effect (ITT)
The ITT measures the average effect of being assigned to treatment, regardless of whether users actually receive or comply with the treatment. This estimand compares outcomes between those assigned to the treatment group and those assigned to the control group, based purely on the initial randomization.
This estimand is essential for answering questions like: “What is the overall impact of our feature rollout campaign, including both users who adopt it and those who ignore it?”
To make this clear: suppose the marketplace randomly assigns 50,000 users to receive an email promoting the new algorithm, while another 50,000 users receive no email. The ITT measures the difference in average total spent between these two groups, regardless of whether users in the first group actually enabled the new algorithm after receiving the email:
ITT = (Average total spent for users assigned to receive the email) − (Average total spent for users assigned to the control group)
Relationship to ATE: The ITT and ATE are closely related but serve different purposes. While the ATE asks “what if everyone were treated?”, the ITT asks “what if everyone were assigned to treatment?” In scenarios where everyone assigned to treatment actually receives and uses the treatment, ITT equals ATE. However, when this relationship is imperfect — as it often is in real-world applications — the ITT will typically be smaller than the ATE because it includes the “diluting” effect of those who were assigned to treatment but didn’t actually receive or use it. Think of ITT as the “realistic” version of ATE that accounts for the messiness of actual implementation.
Local Average Treatment Effect (LATE)
The LATE captures the effect of the treatment only for users whose behavior changes because of being assigned or encouraged to receive the treatment. These are called compliers: people who adopt the treatment only if prompted, such as by a notification or email.
The relevant comparison here is between compliers who were assigned to treatment and compliers who were assigned to control. This estimand is ideal for answering questions like: “How effective is our email campaign at encouraging feature adoption and improving outcomes for those who respond to it?”
To make this clear: imagine some users only enable the new algorithm after being emailed about it; they would not have used it otherwise. The LATE would then measure the causal effect on these email-persuaded users:
LATE = (Average total spent for users who enabled the algorithm due to the email) − (Average total spent those same users would have had without the email)
We’ll unpack compliers and LATE in much more detail in Chapter 7.
Conditional Average Treatment Effect (CATE)
The CATE shows how the average treatment effect differs across specific subgroups within the population. Instead of one population-wide average, it provides subgroup-specific effects that can inform personalization.
This estimand is essential for answering questions like: “Is this feature more effective for new users than for our loyal subscribers?” or “Does the algorithm’s impact depend on the user’s preferred genre?”
To make this clear: the marketplace might find that the ATE of the new algorithm is a R$ 10 increase in total spent. However, the CATE could reveal that for users who buy electronics, the effect is a R$ 25 increase, while for users who buy books, there is no effect at all. This insight allows the platform to personalize the user experience by rolling out the feature only to the subgroups for whom it is most beneficial.
CATE for electronics buyers = (Average total spent for electronics buyers with the new algorithm) − (Average total spent they would have had with the old algorithm)
Individual Treatment Effect (ITE)
The ITE represents the treatment effect for a specific individual — the most granular level of causal analysis possible. It answers the question: “What would be the exact effect of this treatment on this particular user?”
This estimand would be ideal for answering questions like: “Should we recommend the new algorithm specifically to User Robson based on their individual characteristics and predicted response?”
To make this clear: the ITE for a specific user would be the difference between their total spent with the new algorithm and their total spent with the old algorithm. For instance, User Robson might have an ITE of + R$ 15, while User Cristiane might have an ITE of - R$ 5 (meaning the new algorithm actually decreases her spending).
ITE for UserRobson= (UserRobson’s total spent with new algorithm) − (UserRobson’s total spent with old algorithm)
Feasibility and relationship to CATE: The ITE is fundamentally unobservable because we can never observe the same individual in both treated and untreated states simultaneously. However, the ITE is the theoretical foundation that underlies other estimands.
The CATE, for instance, can be thought of as an aggregation of ITEs across a subgroup of users with similar characteristics. The finer we make these groups, the closer we get to individual-level effects. In practice, modern machine learning techniques for estimating CATE — such as causal forests or meta-learners — are essentially sophisticated ways of approximating individual treatment effects by creating very fine-grained subgroups based on user characteristics.
2.5.1 From counterfactuals to comparison groups: the bloodhound’s job
Every topic in this chapter—omitted variable bias, selection bias, and causal estimands—returns to the same problem: we can never observe what would have happened to a user under the alternative treatment condition. João saw the promo banner, so his orders without it remain unknown. Maria did not see it, so her response to the banner remains unknown.
Causal inference therefore depends on a comparison group that can stand in for the missing counterfactual. The more closely it represents what treated users would have experienced without treatment, the more credible the estimate. Each method in Part II, from randomized experiments to difference-in-differences, constructs that comparison differently.
Randomized experiments are the gold standard because random assignment makes the treatment and control groups comparable on average. If we flip a coin to decide who gets the new recommendation algorithm, the coin breaks any systematic link between treatment and the users’ potential outcomes. The control group can then stand in for the missing counterfactual without systematic selection bias or confounding.
The next chapters turn these concepts into practice: designing experiments, handling non-compliance, and deciding when you can (and can’t) trust a comparison group.
2.6 Wrapping up and next steps
We’ve covered a lot of ground in this chapter. By now you can:
- Recognize that raw data alone can’t prove causality; as the turkey example showed, even a thousand data points can’t distinguish between a caregiver and a butcher without a causal model to interpret them.
- Identify the two main enemies of causal inference: omitted variable bias (when hidden factors like engagement drive both treatment and outcome) and selection bias (when the treated group is fundamentally different from the control group to begin with).
- Understand the fundamental problem of causal inference: we can never observe both potential outcomes for the same unit. This forces us to rely on counterfactual thinking — always asking “what would have happened in the alternative scenario?”—and to search for valid comparison groups.
- Speak the language of causal estimands to match the right metric to the right business question, whether it’s deciding on a global rollout (ATE), measuring impact on actual users (ATT), or personalizing offers (CATE).
- Appreciate why randomized experiments are the gold standard: by deciding who gets treated by a flip of a coin (or a random number generator), we eliminate selection bias and ensure that, on average, our comparison groups are identical.
The next chapter turns these concepts into an analysis plan. You will learn:
- A practical framework to plan and design causal analyses, ensuring you never start a project without a clear path to a valid answer.
- How to translate vague business requests into precise, testable causal hypotheses (and how to perform “business therapy” to get stakeholders on board).
- The difference between design-based (ex-ante) and model-based (ex-post) approaches, and how to choose the right one for your constraints.
- A high-level view of how to execute a complete analysis, from checking assumptions to interpreting the results and pressure-testing your conclusions.
Appendix 2.A: More biases and misleading relations
Attrition bias
Technical explanation: Attrition, also called dropout or non-adherence, occurs when participants leave a study or sample over time. It causes bias when those who leave differ systematically from those who remain.
Researchers call a dataset with a complete history for every participant a “balanced panel”. An unbalanced panel has missing observations and may bias treatment-effect estimates, depending on why the data are missing. Attrition introduces no bias when it is unrelated to treatment assignment and the outcome.
Intuitive explanation: Suppose dissatisfied customers leave a long-term study while satisfied customers remain. The remaining sample can make the intervention look more effective than it is, much like judging a restaurant only from reviews left by happy customers.
Reverse causality
Technical explanation: Reverse causality occurs when the assumed cause-and-effect relationship between two variables is inverted: the observed “effect” actually influences the “cause”. In digital businesses, this often arises when analyzing user behavior or product metrics, where temporal sequencing is misidentified.
Intuitive explanation: A fitness app may claim that its “daily achievement badges” increase activity because badge earners exercise more. But users who already exercise regularly may simply be more likely to unlock badges. The badges may reflect existing habits rather than cause activity.
Interpreting such models causally can misdirect investment. A social media company might prioritize a feature correlated with engagement, such as live streaming, only to discover that the feature attracts existing power users rather than new ones.
Simultaneity
Technical explanation: Simultaneity occurs when two variables affect each other at the same time. This two-way relationship obscures the direction of causality and biases estimates from standard models.
In causal inference, this feedback loop prevents a standard model from separating the effect of X on Y from the effect of Y on X. Ignoring it produces biased causal estimates.
Intuitive explanation: Consider a social media platform analyzing the relationship between ad spending and user growth. Increased ad spending may attract more users, but simultaneously, rapid user growth might prompt the platform to spend more on advertising to maintain momentum. Both variables feed into each other simultaneously, blurring the line between cause and effect.
Consequence on causal interpretation: If simultaneity is ignored and results are interpreted causally, businesses risk misallocating resources based on faulty assumptions about which factors drive outcomes. This could lead a company to overly rely on ad spending, mistakenly believing it’s the sole driver of user growth, without recognizing that user growth itself fuels spending decisions. Properly addressing simultaneity usually requires specialized methods such as instrumental variables or simultaneous equation modeling.
Appendix 2.B: Mathematical demonstration of the OVB
Let’s demonstrate mathematically how omitted variable bias arises. In the beginning of this chapter, we used total_spent as the dependent variable, recommendations_shown as the main variable of interest, and engagement as the omitted variable. To generalize the demonstration below, we will use \(Y\) as the dependent variable, \(X_1\) as the main variable of interest, and \(X_2\) as the omitted variable.
Suppose the true model is:
\[ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \varepsilon \tag{2.4}\]
But we estimate the simpler model, omitting \(X_2\):
\[ Y = \beta_0 + \beta_1 X_1 + \nu \tag{2.5}\]
In a large sample, the estimator \(\hat{\beta}_1\) converges to:
\[ \hat{\beta}_1 = \frac{\text{Cov}(Y, X_1)}{\text{Var}(X_1)} \tag{2.6}\]
Substituting the true model from Equation 2.4:
\[ \hat{\beta}_1 = \frac{\text{Cov}(\beta_0 + \beta_1 X_1 + \beta_2 X_2 + \varepsilon, X_1)}{\text{Var}(X_1)} \tag{2.7}\]
Using the linearity of covariance:
\[ \hat{\beta}_1 = \frac{\text{Cov}(\beta_0, X_1) + \beta_1 \text{Cov}(X_1, X_1) + \beta_2 \text{Cov}(X_1, X_2) + \text{Cov}(\varepsilon, X_1)}{\text{Var}(X_1)} \tag{2.8}\]
Using the properties of covariance — specifically that \(\text{Cov}(\beta_0, X_1) = 0\) (a constant doesn’t vary) and \(\text{Cov}(\varepsilon, X_1) = 0\) (by assumption)—we arrive at:
\[ \mathbb{E}[\hat{\beta}_1] = \frac{0 + \beta_1 \text{Var}(X_1) + \beta_2 \text{Cov}(X_1, X_2) + 0}{\text{Var}(X_1)} \tag{2.9}\]
This simplifies to the omitted variable bias formula:
\[ \mathbb{E}[\hat{\beta}_1] = \beta_1 + \left[ \beta_2 \times \frac{\text{Cov}(X_1, X_2)}{\text{Var}(X_1)} \right] \tag{2.10}\]
The bias term \(\beta_2 \times \frac{\text{Cov}(X_1, X_2)}{\text{Var}(X_1)}\) shows that bias occurs when: 1. The omitted variable \(X_2\) has a non-zero effect on \(Y\) (i.e., \(\beta_2 \neq 0\)) 2. The omitted variable \(X_2\) is correlated with the included variable \(X_1\) (i.e., \(\text{Cov}(X_1, X_2) \neq 0\))
As Google advises, “NotebookLM can be inaccurate; please double-check its content”. I recommend reading the chapter first, then listening to the audio to reinforce what you’ve learned.↩︎
The original passage is about thanksgiving, but I find Christmas more relatable to most readers, including those outside the US.↩︎
To load it directly, use the example URL
https://raw.githubusercontent.com/RobsonTigre/everyday-ci/main/data/advertising_data.csvand replace the filenameadvertising_data.csvwith the one you need.↩︎Common measures of engagement are (i) the average time users spent in the app before the recommendation system was implemented (i.e., before the intervention), (ii) the number of items purchased before the recommendation system was implemented, (iii) the amount spent before the recommendation system was implemented, (iv) a combination of these measures, etc.↩︎
Note that “time spent” isn’t always a good proxy for engagement. In an app with poor UX, a user might spend a long time just trying to figure out how to complete a task! But for established marketplaces, we can generally assume that more time spent browsing correlates with higher interest and more product searches. We use it here for simplicity.↩︎
Follow this example for now. We develop DAG concepts and applications in Chapter 6 after introducing the ideas they require.↩︎
In real life, people tend to control for past measures of user’s Recency, Frequency, Monetary (RFM) variables when studying customer behavior. For sure, this is not enough to account for all the differences between users, but it’s a useful framework for you to know. See more here.↩︎
If you are curious about the derivation of this omitted variable formula, check Appendix 2.B↩︎
Experienced readers will notice I’m trading statistical and mathematical precision for intuition here. Strictly speaking, the purple term in Equation 2.3 represents the covariance of recommendations and engagement scaled by the variance of recommendations (formally, the coefficient \(\alpha_2\) from the auxiliary regression \(\text{engagement} = \alpha_1 + \alpha_2 \times \text{recommendations shown} + \eta\)). But since this book focuses on practitioners solving business problems rather than scholarly proofs, I prioritize simplicity and intuition over precision. If you spot a misleading inaccuracy, please leave a comment with a suggested fix that fits the context of this book.↩︎
Formally, this is the coefficient \(\alpha_2\) from the auxiliary regression \(\text{engagement} = \alpha_1 + \alpha_2 \times \text{recommendations shown} + \eta\).↩︎
You can confirm this positive relationship by looking at Figure 2.2, or by checking the correlation with
cor()(R) ordf.corr()(Python) as shown in Appendix 1.A.↩︎A confounder is a common cause of both treatment and outcome (e.g., Engagement causes both Recommendation and Spending); controlling for it blocks bias. A mediator is a step in the chain (mechanism); controlling for it blocks the effect you want to measure. A collider is a common effect (e.g., both Recommendation and Spending cause Support Tickets); controlling for it creates a spurious correlation. To know what to include, you need a causal story.↩︎
Another framework is the continuous treatment framework, where the treatment can take on different values: for instance, the dosage of a medication in medicine. This is called the “intensity” of the treatment.↩︎
I will use the term \(D_i\) to denote the treatment assignment variable, just because I’m more used to it from my readings. Some other books use \(T_i\).↩︎
Notice I say “assigned to receive” instead of “received”. In real-world settings, users may be assigned to a treatment without receiving or taking it. For example, a patient given medication to take at home may not consume it. We address these distinctions starting in Chapter 4.↩︎