Glossary

Published

December 27, 2025

Modified

August 11, 2026

Keywords

A/B Testing, Causal Inference, Causal Time Series Analysis, Data Science, Difference-in-Differences, Directed Acyclic Graphs, Econometrics, Impact Evaluation, Instrumental Variables, Heterogeneous Treatment Effects, Potential Outcomes, Power Analysis, Sample Size Calculation, Python and R Programming, Randomized Experiments, Regression Discontinuity, Treatment Effects

Get the new chapters straight to your inbox. Leave your address here.
Found an error or have a suggestion? Submit feedback here.
Follow the book’s page for new content and corrections, or follow the author.

A

A/A test: A diagnostic experiment where both groups receive the identical experience (no actual treatment difference). Used to validate that the experimentation infrastructure is working correctly — if you find a statistically significant difference between two identical groups, something may be broken in your randomization or data pipeline.
A/B test: A randomized experiment where two or more variants (A and B) are compared to determine which one performs better. In causal inference, it is the gold standard for estimating treatment effects because randomization eliminates selection bias.
Absorbing treatment: A treatment that, once received, stays “on” for the rest of the study. Most policy changes, infrastructure investments, and feature launches are absorbing; a temporary promotion that turns off after a few weeks is not. Also called irreversible treatment.
Adoption curve (or adoption ramp): The path by which a feature reaches its eligible audience after launch — typically gradual rather than instantaneous. Multiplying a per-user causal effect by the steady-state user count instead of the period-by-period adoption curve overstates the cumulative impact in the early months.
Alternative hypothesis ($H_1$): The hypothesis that there is a real effect or difference (e.g., “the feature generates incremental revenue”). Validated only when data provides strong evidence to reject the null hypothesis.
Always-takers: In instrumental variables settings, people who always take the treatment regardless of their assignment. They get treated whether assigned to control or treatment and contribute zero to the first stage effect.
Anticipation effect: A change in outcomes that occurs before a treatment officially begins, often because units know it is coming and adjust their behavior. Anticipation contaminates the pre-treatment baseline and can bias DiD estimates if not addressed — modern staggered DiD methods carve out contaminated periods through a tunable anticipation window.
ARIMA (AutoRegressive Integrated Moving Average): A forecasting model that predicts future values based on a combination of past values (AR), differencing to remove trends (I), and past forecast errors (MA). The (p,d,q) parameters control how many lags of each component to include.
Ashenfelter’s Dip: A phenomenon where individuals who participate in a program (like job training) experience a temporary decline in their outcome (like earnings) just before the program starts. This pre-program dip is a classic sign of selection bias.
Attrition bias: Bias caused when participants drop out of a study non-randomly (e.g., only unsatisfied users leave), rendering the remaining treatment and control groups incomparable.
AUC (Area Under the Curve): A score from 0.5 to 1 measuring how well a classification model separates two classes — for example, treated from untreated users. 0.5 means the model is guessing; 1 means perfect separation.
Auxiliary series: In CausalImpact, a related time series that was not affected by the intervention. Used to build a more credible counterfactual by capturing external factors that influence the outcome.
Average Treatment Effect (ATE): The conceptual average of the individual treatment effects across the entire population. It answers the question: “What would be the average difference in outcomes if everyone were treated versus if no one were treated?”
Average Treatment Effect on the Treated (ATT): The average causal effect of the treatment specifically for those who actually received it. It answers: “How much did the program help the people who actually participated?”

B

Backdoor path: In a causal diagram (DAG), a non-causal path connecting treatment and outcome through common causes (confounders). These paths must be blocked to identify the true effect.
Bad control: A variable that, when controlled for in a regression, introduces bias rather than removing it. Common examples include mediators (which block the causal path) and colliders (which open spurious paths).
Balanced panel: A panel dataset in which every unit is observed in every time period. Standard DiD estimators assume a balanced panel because each ATT depends on a long difference computed for the same units. The opposite is an unbalanced panel.
Bandwidth: The specific range of the running variable around the cutoff used for analysis in a Regression Discontinuity Design (RDD). Because the assumption of comparability only holds near the threshold, bandwidth selection triggers a trade-off between bias (too wide) and variance (too narrow).
BART (Bayesian Additive Regression Trees): A Bayesian machine learning method that models the outcome with many small decision trees, producing flexible causal estimates with uncertainty intervals built in. Its successor, the Bayesian Causal Forest, extends it to strong confounding and heterogeneous effects.
Bayesian inference: A framework for updating beliefs with data: start from a prior distribution, combine it with the observed data through a likelihood function, and obtain a posterior distribution. It answers questions like “what is the probability the effect exceeds 1 point?” directly, rather than through p-values.
Bias: Systematic error that causes the estimated effect to differ from the true causal effect. Common sources include selection bias, omitted variable bias, and measurement error.
Breakeven effect: The smallest per-user-per-period effect that makes a fixed investment pay for itself. Computed as investment cost divided by (users × effective periods), where “effective periods” is the raw horizon for a flat effect or the discounted sum $\sum_{t=0}^{T-1}\lambda^{t}$ for an effect that decays geometrically. A direct application of the cost-volume-profit break-even formula from managerial accounting (Horngren, Datar, and Rajan 2012, 91). When the MDE from power analysis exceeds the breakeven effect, a statistically significant result implies the effect is large enough to justify the investment.
BSTS (Bayesian Structural Time-Series): A flexible forecasting framework that decomposes a time series into trend, seasonality, and regression components while quantifying uncertainty using Bayesian methods. The foundation of CausalImpact.

C

CAC (Customer Acquisition Cost): The total cost of acquiring one new customer or activating one new user of a feature. Defined as total acquisition spend divided by new customers acquired.
Cannibalization: A form of interference where treating one unit reduces outcomes for another unit. Common in marketplaces (e.g., boosting Seller X’s visibility reduces sales for Seller Y) and product lines (e.g., a new product steals customers from an existing one). Cannibalization violates SUTVA because the control group’s outcomes are affected by the treatment.
Causal ARIMA (C-ARIMA): An extension of ARIMA models designed for causal inference. It uses pre-intervention data to forecast what would have happened without the intervention, then compares this counterfactual to actual outcomes.
Causal discovery: A family of algorithms that attempt to learn the causal graph (DAG) from data rather than assuming it, by testing which variables are independent of which others. Best used to challenge and refine a hand-drawn DAG, not to replace domain knowledge.
Causal forest: A machine learning algorithm used to estimate heterogeneous treatment effects. It improves on standard random forests by using “honest” splitting to maximize the difference in effects across subgroups.
Causal inference: The science of determining whether and how much a specific action (cause) affects an outcome (effect). It goes beyond correlation to understand “what would happen if” we intervened.
Causal machine learning: Using machine learning to estimate causal effects — handling hundreds of covariates and flexible functional forms — while keeping the identification discipline of causal inference. Double Machine Learning, causal forests, and meta-learners are examples.
Causal Markov condition: A causal discovery assumption: once you condition on a variable’s direct causes, it is independent of everything except its own effects. This is what lets an algorithm read independence relationships off a causal graph.
Causal sufficiency: The causal discovery assumption that no unmeasured common cause sits behind any pair of measured variables. If a hidden confounder affects two measured variables, discovery algorithms can confidently draw the wrong graph.
CausalImpact: A Bayesian method developed by Google for estimating causal effects in time series when no control group exists. It uses auxiliary series to build a counterfactual and provides posterior probability distributions for the effect.
Central Limit Theorem (CLT): A statistical theorem stating that the sampling distribution of the sample mean approaches a normal distribution as the sample size gets larger, regardless of the shape of the population distribution.
Cluster randomization: A randomization scheme where entire groups (clusters) — such as cities, stores, or driver pools — are assigned to treatment or control rather than individual units. Used to prevent contamination when interference between units is expected, but dramatically reduces effective sample size.
Clustered standard errors: Standard errors that account for the correlation of observations within groups (clusters). Required when the unit of analysis differs from the unit of randomization, or when errors are correlated within groups over time.
Cohen’s d: A standardized measure of effect size expressing the difference between two means in terms of standard deviations. In experiments, it helps compare effects across studies and determine practical significance beyond statistical significance.
Cohort: In staggered designs, the set of units that first receive treatment in the same period. “Group 7” or “cohort 7” refers to the units first treated in period 7. Used interchangeably with group.
Collider: A variable that is caused by both the treatment and the outcome (or by two other variables in a path). Controlling for a collider creates a spurious association between the variables that cause it, leading to bias (often called M-bias or selection bias).
Common support (overlap): The assumption that every unit in the population has a non-zero probability of receiving either the treatment or the control. Without common support, we cannot reliably estimate causal effects for the entire population.
Compliance rate: The percentage of units assigned to a treatment group who actually receive or consume the treatment. Low compliance dilutes the intention-to-treat estimate.
Compliers: In instrumental variables settings, people who follow their assignment: they take the treatment if assigned to it and skip it if assigned to control. The LATE estimates the treatment effect specifically for this persuadable segment.
Compound treatment: A nominal treatment that arrives bundled with other simultaneous changes, making their separate effects impossible to attribute without additional design or variation. The estimate identifies the bundle’s effect, not necessarily the named component’s effect.
Conditional Average Treatment Effect (CATE): The average treatment effect for a specific subgroup defined by observed characteristics (e.g., the effect for “mobile users” vs. “desktop users”).
Conditional Independence Assumption (CIA): The assumption that, after controlling for a set of observed covariates, the treatment assignment is effectively random (i.e., independent of potential outcomes). Also known as “selection on observables” or “unconfoundedness.”
Conditional parallel trends: A more flexible version of the parallel trends assumption: after conditioning on a set of observed covariates, the treated and control groups would have followed parallel paths in the absence of treatment. Useful when treated and control units differ in observable ways that affect outcome trends.
Confidence interval (CI): A range of values derived from sample data that is likely to contain the true population parameter with a certain level of confidence (e.g., 95%). It quantifies the uncertainty of an estimate.
Confounder: A variable that causes both the treatment and the outcome. If not controlled for, it creates a “backdoor path” that introduces omitted variable bias, making it look like the treatment caused the outcome when it was actually the confounder.
Consistency (assumption): The assumption that a unit’s observed outcome equals its potential outcome under the treatment it actually received — which requires the treatment to be well-defined in timing, dose, and content. “Send a discount” is too vague if one user gets 10% off and another gets free delivery for a month.
Contribution margin: The share of revenue remaining after subtracting variable costs. A 30% contribution margin means that for every R$1 of revenue, R$0.30 is available to cover fixed costs and generate profit. Applied when converting revenue-based causal estimates to profit.
Control group: The group of units that does not receive the treatment or intervention. They serve as the benchmark for comparison to estimate the counterfactual (what would have happened to the treated group without the treatment).
Correlation: A statistical measure of the strength and direction of the linear relationship between two variables. While correlation can indicate a pattern, “correlation does not imply causation.”
Counterfactual: A hypothetical scenario representing what would have happened to a specific unit if they had received a different treatment than they actually did. Since we can never observe the counterfactual, causal inference is often defined as a “missing data problem.”
Covariance: A statistical measure of how two variables change together. Positive covariance means they move in the same direction; negative means opposite directions.
Covariate: A variable that is observed and can be included in a statistical model to adjust for confounding or improve precision. Also called a control variable, regressor, or feature.
Covariate balance test: A diagnostic check to verify that treatment and control groups are comparable on observable characteristics. In RCTs, it confirms successful randomization; in quasi-experiments (like RDD), it supports the validity of the design by ensuring no systematic differences exist between groups other than the treatment itself.
CPDAG (Completed Partially Directed Acyclic Graph): The output of causal discovery algorithms like PC: a graph where some edges carry arrows and others remain undirected because the data cannot determine their direction. It represents a Markov equivalence class, not one settled DAG.
Credible interval: The Bayesian counterpart to a confidence interval, with the interpretation people usually want: “there is a 95% probability the true effect lies in this range” — conditional on the model and prior being reasonable.
Cross-fitting: A safeguard in Double Machine Learning: split the data into folds, train the ML models on one fold, and generate predictions on another. This prevents the models from overfitting the same data they are cleaning, which would bias the causal estimate.
CUPED (Controlled-experiment Using Pre-Experiment Data): A variance reduction technique that uses pre-experiment data to reduce noise in experiment analysis. By subtracting each user’s predicted outcome (based on their pre-experiment behavior), CUPED can dramatically shrink confidence intervals, allowing detection of smaller effects with the same sample size.
Customer Lifetime Value (CLV / LTV): The total predicted revenue or profit a customer will generate throughout their entire relationship with a business. Also abbreviated as LTV (Lifetime Value). Since it is a long-term metric, experiments often utilize short-term predictive metrics (surrogates) to estimate impact on LTV. When applied to a causal estimate, incremental CLV (or $\Delta$CLV) is the lift caused by the treatment over the projection horizon, after discounting for effect decay and the time value of money.
Cutoff: The specific threshold value of the running variable in a Regression Discontinuity Design (RDD). Crossing this line determines whether a unit is assigned to the treatment group or the control group.

D

d-separation: The rule for reading a DAG that says which variables the graph implies should be statistically independent of which others, once you condition on a given set. It is the bridge between a causal graph and testable patterns in the data.
DAG (Directed Acyclic Graph): A visual map of causal assumptions using nodes (variables) and arrows (causal links). It helps brainstorm which variables are confounders, colliders, or mediators.
Defiers: In instrumental variables settings, people who do the opposite of their assignment: they refuse treatment when assigned to it and take treatment when assigned to control. Most IV analyses assume defiers do not exist (the monotonicity assumption).
Delta method: A method for propagating uncertainty through a smooth function of one or more estimates, using a first-order Taylor expansion. In business translation, used to put a confidence interval around a derived metric like ROI when the formula is nonlinear in its inputs or when several of those inputs carry their own uncertainty.
Density test: A statistical test (e.g., the classic McCrary test) used in Regression Discontinuity Design to detect manipulation of the running variable. A sudden spike or drop in the number of observations at the cutoff suggests that units may be gaming their scores to qualify for treatment, violating key assumptions.
Difference-in-Differences (DiD): A method that estimates the causal effect of a treatment by comparing the change in outcomes over time for a treated group to the change in outcomes for a control group. It relies on the parallel trends assumption.
Diminishing returns: The phenomenon where each additional unit of investment produces a smaller incremental outcome. In scaling causal estimates, it reflects that the first users reached are typically more responsive than marginal users acquired later. The cost side often moves the same direction — the marginal cost of reaching the next user rises as the most accessible audience is exhausted.
Discount rate: The per-period rate ($r$) used to convert future cash flows into today’s reais, capturing the opportunity cost of capital. A R$1 received next month is worth $1/(1+r)$ today. In project ROI, finance teams typically supply $r$ from WACC or a risk-adjusted hurdle rate.
Do-calculus: Three rules that determine whether a causal effect can be computed from observational data alone, by converting “what happens if we intervene” into expressions involving only what we observe. If do-calculus cannot reduce the effect to observable quantities, nothing else will — from that graph and data alone.
Do-operator: Pearl’s notation, $P(Y \mid do(X = x))$, for the effect of intervening on a variable rather than passively observing it. Observing is checking the thermostat reading; intervening is overriding it to 22°C.
Double Machine Learning (DML): A method using machine learning models to “clean” the treatment and outcome of confounding from high-dimensional controls, isolating the causal effect of interest.
Doubly robust estimator: An estimator that combines an outcome model with a treatment (propensity) model so the estimate stays valid if either one is correct. The two models’ errors multiply rather than add, so one good model compensates for the other.
DR-learner: A meta-learner for heterogeneous treatment effects built on the doubly robust construction — outcome modeling combined with propensity weighting. Robust on paper, but unstable where true treatment probabilities sit near 0 or 1.
Dummy variable: A binary variable that takes on the value 0 or 1 to indicate the absence or presence of some categorical effect (e.g., “Treatment = 1” vs “Control = 0”).
Dynamic treatment effects: Treatment effects that vary with the length of exposure rather than being constant after adoption. The event-study aggregation surfaces these dynamics by reporting one effect per event time.

E

Econometrics: The application of statistical methods to economic data to test hypotheses and estimate causal relationships.
Effect-persistence factor ($\lambda$): The share of a treatment effect that persists from one period to the next under a geometric decay model: $\text{Effect}_t = \text{Effect}_0 \times \lambda^t$. A $\lambda$ of 0.90 means 90% of the effect carries over each period (a 10% per-period decay rate, $1 - \lambda$). Distinct from customer-survival rates used in CLV models, which describe whether the user is still around.
Effective sample size (ESS): The number of equally weighted observations that would carry roughly the same information as a weighted sample. Uneven or repeated weights can make ESS much smaller than the raw row count.
Encouragement design: An experimental design where participants are randomly encouraged to take a treatment (e.g., via an invitation or incentive) rather than being forced into it. The encouragement serves as an instrument for estimating the treatment effect among compliers.
Endogeneity: A condition where an explanatory variable (like treatment) is correlated with the error term in a regression model, typically due to omitted variables, reverse causality, or measurement error. It renders OLS estimates biased.
Equilibrium effects: Interference that operates through market-level mechanisms — prices, wait times, inventory — which shift for everyone once a treatment rolls out at scale. Harder to handle than local spillovers because market mechanisms may not respect cluster boundaries.
Error term: The random variable in a regression model (often $\varepsilon$) representing all unobserved factors affecting the outcome. Ideally, it should be random and uncorrelated with the predictors.
Estimand: The precise quantity we want to estimate (the target), such as the Average Treatment Effect (ATE). It is defined based on the population and the causal question, independent of the data or method used.
Estimate: The numerical value obtained from applying an estimator to a specific sample of data. It is our “best guess” of the true parameter.
Estimator: The statistical rule or formula used to calculate an estimate from data (e.g., the formula for the sample mean, or the OLS algorithm).
Event study: A dynamic version of difference-in-differences that estimates treatment effects for each time period relative to the treatment start. It allows for checking parallel trends (pre-treatment) and seeing how effects evolve over time (post-treatment).
EVPI (Expected Value of Perfect Information): The maximum amount a decision-maker should pay to eliminate uncertainty before acting. If the cost of collecting more data exceeds the EVPI, the additional certainty is not worth the delay.
Exclusion restriction: A critical assumption in instrumental variables (IV) requiring that the instrument affects the outcome only through the treatment, and not via any other channel.
Exogeneity: The condition where an explanatory variable is not correlated with the error term. Using exogenous variation (like in an RCT or IV) is key to identifying causal effects.
Expected value: The theoretical long-run average of a random variable. In data analysis, the sample mean is our best estimate of the population expected value.
Exposure mapping: A rule defining what each unit is actually “exposed” to under a given treatment assignment — for example, “directly treated,” “untreated but with treated friends,” or “no exposure.” The foundation of interference-aware analysis.
External validity: The extent to which the results of a study can be generalized to other populations, settings, or time periods.

F

Faithfulness: The causal discovery assumption that every independence in the data reflects the graph’s structure, not causal effects cancelling out exactly — like a drug that raises blood pressure through one pathway and lowers it by the same amount through another.
First stage effect: In instrumental variables, the effect of the instrument on treatment uptake. It measures how much the instrument changes the probability of receiving treatment. A strong first stage is required for reliable IV estimates.
Fixed effects: Parameters in a regression model that control for unobserved time-invariant characteristics of units (individual fixed effects) or period-specific shocks shared by all units (time fixed effects).
FP&A (Financial Planning & Analysis): The finance function responsible for budgeting, forecasting, and analyzing a company’s financial performance. FP&A teams maintain cost models and unit economics that feed into ROI calculations for causal estimates.
Front-door criterion: An identification strategy that recovers a causal effect by following it forward through a mediator, even when an unmeasured confounder makes backdoor adjustment impossible. It requires the mediator to fully carry the effect, plus clean treatment–mediator and mediator–outcome links.
Frontdoor path: A causal path flowing from treatment to outcome, potentially through mediators. It represents the mechanism by which the treatment actually works.
Fundamental problem of causal inference: Each unit has two potential outcomes — one with treatment, one without — but you only ever observe one. Causal inference is the discipline of filling in the missing half credibly.

G

G-methods: A family of estimators — the g-formula, inverse probability weighting with marginal structural models, and g-estimation — built for treatment strategies that unfold over time, where later confounders were affected by earlier treatments.
Gaussian (normal) distribution: The bell-curve probability distribution that many statistical methods assume for errors or estimates. “Non-Gaussian” means the data departs from that shape — heavy tails, skew, or multiple humps.
Guardrail metrics: Essential business metrics monitored during experiments (e.g., latency, crash rate) to ensure a new feature doesn’t accidentally harm the user experience.

H

HARKing: “Hypothesizing After the Results are Known.” The misleading practice of presenting a post-hoc hypothesis (created after seeing the data) as if it were the original plan.
Heterogeneous treatment effects: When the effect of a treatment is not the same for everyone but varies across different individuals or subgroups (e.g., a drug works better for younger patients).
HiPPO: “Highest Paid Person’s Opinion.” A decision-making culture where the intuition of senior leaders overrides data and experimental evidence.
Hypothesis testing: A statistical method used to decide whether to reject a null hypothesis (usually “no effect”) based on sample data, assessing whether an observed result is likely due to chance.

I

Identification strategy: The research design and set of assumptions allowing a researcher to claim that an estimated correlation is truly a causal effect (e.g., “we identified the effect using a randomized experiment”).
IID (Independent and Identically Distributed): The standard assumption that each observation is unrelated to the others and drawn from the same process. Time series and networked data violate it, which is why many methods mislead when applied to them naively.
Imperfect compliance: When participants do not follow their assigned treatment protocol (e.g., people assigned to a drug don’t take it), complicating the estimation of treatment effects.
Independent variable: The variable manipulating or predicting the outcome (the $X$ in $Y = \beta X$). In experiments, this is often the treatment.
Individual Treatment Effect (ITE): The treatment effect for a single, specific unit — the most granular level of causal analysis. Fundamentally unobservable because we can never see both potential outcomes for the same individual, but serves as the theoretical foundation for other estimands like CATE.
Instrumental variable (IV): A method used when the treatment is endogenous (e.g., due to unobserved confounding). An “instrument” is a variable that affects the treatment but has no direct effect on the outcome and is not correlated with confounders.
Intention-to-Treat (ITT): The average effect of being assigned to a treatment, regardless of whether the subject actually received it. It reflects the real-world impact of a policy rollout.
Interaction term: A variable added to a regression (e.g., $Treatment \times Gender$) to test if the treatment effect varies depending on another characteristic.
Interference: When one unit’s treatment affects another unit’s outcome — a rider’s wait time depends on how many other riders got the promotion. It violates SUTVA and changes what a standard A/B comparison actually estimates. See also Spillover and Network effects.
Internal validity: The extent to which a study accurately establishes a causal relationship between the treatment and the outcome within the context of the study itself (i.e., is the estimated effect unbiased?).
Interrupted Time Series (ITS): An observational method that uses data from before and after an intervention to estimate its effect, relying on the pre-intervention trend to construct a counterfactual. Useful when no control group exists.
Inverse Probability Weighting (IPW): A method that weights observations by the inverse of their probability of treatment (propensity score) to create a synthetic sample where treatment and control are balanced.
Irreversible treatment: See Absorbing treatment.

K

Kernel function: A weighting scheme used in local regression (like RDD) that assigns more importance to observations closer to the cutoff. Common types include triangular (decreasing weight with distance) and uniform (equal weight within the bandwidth).
Kitchen sink regression: A regression approach where the analyst includes every available variable, hoping to control for confounding. Often counterproductive because it may include bad controls (mediators, colliders, treatment predictors) that introduce bias or inflate standard errors.

L

Lagged effect: An impact that manifests some time after the intervention occurred (e.g., an ad campaign today increasing sales next month).
Law of Large Numbers (LLN): A theorem stating that as the sample size increases, the sample mean gets closer and closer to the true population mean.
Leads and lags: In event study designs, leads are the pre-treatment coefficients ($\delta_k$ for $k < 0$) that test whether treated and control groups were on similar trajectories before the intervention. Lags are the post-treatment coefficients ($\delta_k$ for $k \geq 0$) that capture how the treatment effect evolves over time. Leads near zero support the parallel trends assumption; significant leads signal a potential violation.
Likelihood function: The part of a statistical model that measures how well each candidate parameter value — each possible effect size — explains the data actually observed. In Bayesian inference, it is what updates the prior into the posterior.
Local Average Treatment Effect (LATE): The treatment effect specifically for “compliers” — people who are induced to take the treatment by the instrument or assignment.

M

Machine learning (ML): A field of AI focused on building algorithms that learn patterns from data to make predictions. While powerful for prediction, standard ML does not inherently solve causal problems.
Marginal structural model (MSM): A model for the causal effect of a treatment strategy over time, estimated by reweighting the data (inverse probability weighting) so that treatment at each point looks as if it had been randomly assigned. The reweighted data forms a “pseudo-population” free of measured time-varying confounding.
Markov equivalence class: The set of causal graphs that imply exactly the same independence patterns and therefore cannot be told apart by observational data. “A causes B causes C” and its reverse leave the same statistical footprint; outside knowledge must break the tie.
MAU (Monthly Active Users): The count of unique users who interact with a product at least once in a 30-day window. A common engagement metric in digital businesses used to size addressable audiences and scale per-user causal effects to population-level impact.
MCMC (Markov Chain Monte Carlo): A simulation technique used in Bayesian statistics to draw samples from complex probability distributions. In CausalImpact, it generates thousands of possible counterfactual scenarios to quantify uncertainty.
Mechanism: The process or pathway through which a cause produces an effect (the “why” or “how” it happens).
Mediation analysis: A method that decomposes a total treatment effect into an indirect effect flowing through a mediator (the channel you have in mind) and a direct effect that takes other routes. It answers “how does the treatment work,” not just “how big is the effect.”
Mediator: A variable that lies on the causal path between the treatment and the outcome (Treatment -> Mediator -> Outcome). It explains how the treatment works.
Meta-learners: Frameworks (like T-learner, S-learner) that use standard machine learning models to estimate heterogeneous treatment effects by modeling potential outcomes separately.
Minimum Detectable Effect (MDE): The smallest true effect size an experiment can reliably detect with a given sample size and statistical power.
Minimum Effect of Interest (MEI): The smallest causal effect a stakeholder considers worth shipping — a business threshold set during planning, not a statistical one. Also called the Smallest Effect Size of Interest (SESOI) (Georgiev 2019).
Monotonicity: An IV assumption (no “defiers”) meaning the instrument makes everyone more (or no less) likely to take the treatment; it doesn’t discourage anyone who would have otherwise taken it.
Multicollinearity: A situation in which two or more explanatory variables in a regression are highly correlated, making it difficult to isolate their individual effects.

N

Natural direct effect: The part of a treatment effect that does not travel through the mediator: switch the treatment on while pinning the mediator at the value it would have taken without treatment.
Natural indirect effect: The part of a treatment effect that travels through the mediator: hold the treatment on, and move only the mediator from its no-treatment value to its under-treatment value.
Negative-control exposure: A fake or irrelevant exposure that preserves plausible causes of real treatment assignment but has no causal path to the outcome. A nonzero association reveals selection or other structure the main model has not removed.
Negative-control outcome: An outcome that shares plausible causes with the main outcome but cannot be affected by the treatment. A nonzero treatment association warns that confounding, measurement, or another maintained assumption may be failing.
Network effects: A phenomenon where one user’s behavior or treatment status affects the outcomes of other users. In causal inference, network effects violate the SUTVA assumption of “no interference” (e.g., if a discounted user tells a friend about a product, that friend’s behavior is influenced by someone else’s treatment).
Never-takers: In instrumental variables settings, people who never take the treatment regardless of assignment. Even when encouraged or invited, they ignore the treatment and contribute zero to the first stage effect.
Never-treated: Units that never receive the treatment during the entire study period. A clean control group in staggered DiD because their outcomes are never contaminated by treatment.
Not-yet-treated: Units that have not yet received treatment at a given point in time — including any never-treated units plus cohorts that will be treated later. Used as a comparison group when never-treated units are scarce or unrepresentative.
Novelty effect: A temporary spike in user engagement caused by the “newness” of a feature, which often wears off as users get accustomed to it.
Nuisance parameter: The background patterns a model must account for but the analyst does not care about — for example, how covariates predict the outcome and the treatment. Causal ML hands nuisance estimation to flexible ML while keeping the causal effect itself under stricter discipline.
Null hypothesis ($H_0$): The default assumption that there is no effect or no difference. Statistical tests aim to see if there is enough evidence to reject this hypothesis.

O

Omitted variable bias (OVB): Bias in the estimated causal effect that occurs when a relevant confounder is left out of the regression model.
One-sided Non-compliance: A scenario where only one group can deviate from assignment. Typically, users in the control group cannot access treatment, but users in the treatment group can decline it.
Ordinary Least Squares (OLS): The standard method for estimating linear regression coefficients by minimizing the sum of the squared errors between observed and predicted values.
Outcome variable: The variable we are trying to change or explain (the effect). Also called the dependent variable or $Y$.
Overall Evaluation Criterion (OEC): The single primary metric that defines success for an experiment. Having a clear OEC forces stakeholders to agree on what “winning” means before the test runs and prevents post-hoc cherry-picking of favorable metrics.
Overfitting: When a model matches the training data too closely, capturing random noise rather than the signal, leading to poor performance on new data.

P

P&L (Profit and Loss): A financial statement summarizing revenue minus costs over a specific period. When a finance leader asks about “P&L impact,” they want to know where on this scorecard the causal effect shows up and by how many reais.
P-value: The probability of observing a result as extreme as the one found in the sample, assuming the null hypothesis is true. A low p-value (typically < 0.05) suggests the result is statistically significant.
Parallel trends assumption: The critical assumption in difference-in-differences designs: that in the absence of treatment, the average outcomes of the treated and control groups would have changed at the same rate over time.
Parameter: The true, unknown value in the population that we are trying to value (e.g., the true average height of all adults).
Partial interference: The assumption that units may affect others inside their own cluster — city, school, market — but not units in other clusters. It is what makes cluster-randomized experiments valid under interference.
Partial population experiment: A design where some clusters are pure controls and treated clusters expose only a fraction of eligible users. Comparing the three groups — treated, untreated-near-treated, and pure control — separates direct effects from spillovers.
Partial R-squared: The share of variation in one variable explained by another after both have been adjusted for the other variables in a model. Sensitivity analysis uses it to put observed and hypothetical confounders on a common scale.
PC algorithm: A widely used causal discovery algorithm that starts from a fully connected graph, prunes edges by testing conditional independence, then orients what remains. It returns a partially directed graph (CPDAG), not one settled DAG.
Per-protocol analysis: A naive analysis comparing outcomes based on actual treatment received rather than assignment. In experiments with imperfect compliance, this approach reintroduces selection bias and typically overestimates treatment effects.
Percentage point: The absolute difference between two percentages. If retention rises from 40% to 46%, that is a 6 percentage point lift — but a 15 percent increase. Mixing the two overstates or understates effects.
Perfect compliance: When all units follow their assignment exactly: everyone assigned to treatment receives it, and everyone assigned to control does not. This ideal scenario makes ITT equal to the true ATE.
Placebo Test: A falsification check where a “fake” treatment or outcome is tested (e.g., testing the effect on data from before the intervention). Finding an effect describes a problem with the design.
Policy learning: Methods that use estimated treatment effects to choose an assignment rule — who should get the intervention — to maximize total impact within a pre-specified class of allowed policies, where constraints like budget, fairness, and simplicity live.
Policy tree: A shallow decision tree used as a treatment assignment rule — a short set of if-then conditions a product team can inspect, question, and override. Its interpretability comes from restricting the rule to this simple form.
Population: The entire group of individuals or units that we are interested in studying.
Positivity: The requirement that every unit has a nonzero probability of receiving each treatment option given its history — the data must contain both treated and untreated units of every relevant profile. Near-violations produce the extreme weights that destabilize IPW. See also Common support (overlap).
Posterior distribution: In Bayesian inference, the updated probability curve over parameter values after combining the prior with the observed data. From it you read statements like “there is a 93% probability the effect exceeds 1 percentage point.”
Potential outcomes framework: A conceptual framework (Rubin Causal Model) where causal effects are defined as the difference between potential outcomes under different treatment scenarios (e.g., $Y(1) - Y(0)$).
Power: The probability that a statistical test will correctly reject a false null hypothesis (i.e., find an effect if it really exists).
Pre-registration: Documenting the hypothesis, design, and analysis plan before running a study. This transparent practice prevents p-hacking and HARKing.
Prior distribution: In Bayesian inference, the probability distribution encoding what you believe about a parameter before seeing the data — flat (“no idea”) or informative (based on past experiments). The prior does real work in small samples, so state where it came from and test alternatives.
Propensity Score: The probability that a unit receives the treatment given its observed characteristics. Used to match or weight control units to look like treated units.

R

R-squared ($R^2$): A measure (0 to 1) of how well the regression model explains the variance in the outcome data.
Randomization: The process of assigning units to treatment and control groups purely by chance. This ensures that the groups are comparable on average, eliminating selection bias.
Randomized saturation design: An experiment that first randomizes clusters to different treatment intensities — say 0%, 25%, 50%, or 75% of users treated — then randomizes individuals within each cluster. It measures how effects change as more of the local environment is treated.
Reduced form effect: In instrumental variables, the direct effect of the instrument on the outcome. For binary instruments, this is equivalent to the ITT. The LATE is calculated by dividing the reduced form by the first stage.
Regression (Linear): A statistical method modeling the linear relationship between a dependent variable ($Y$) and one or more independent variables ($X$).
Regression discontinuity design (RDD): A causal method that exploits a cutoff or threshold assignment rule (e.g., scholarship for grades > 80%). It compares units just above and just below the cutoff, who are assumed to be effectively randomized.
Regularization bias: The systematic tilt in flexible ML predictions caused by the very restraint (penalties, early stopping) that prevents overfitting. In naive causal estimators this bias flows straight into the treatment effect; DML’s residualization is designed to neutralize it.
Residual: The difference between the observed value and the predicted value in a regression model. It represents the “unexplained” part of the outcome for a specific unit.
Reverse causality: A bias where the direction of cause and effect is flipped or bidirectional (e.g., “do sales drive ad spend, or does ad spend drive sales?”).
Robustness check: An additional analysis run to verify that the main results remain stable under different assumptions or model specifications.
Robustness Value (RV): In partial-$R^2$ sensitivity analysis, the minimum balanced strength an unobserved confounder would need with both treatment and outcome to reduce an estimate to a chosen benchmark, often zero. It is a calibration threshold, not a universal pass/fail score.
ROI (Return on Investment): The ratio of incremental net profit (incremental profit minus the fixed investment that produced it) to the investment. An ROI of 5× means the project returned five times its investment. Useful for comparing efficiency across projects of different scales, but does not capture total value or absolute scale.
Running variable: The continuous variable (often a score, income, etc.) that determines treatment assignment in a Regression Discontinuity Design. Also known as the forcing variable.

S

Sample: A subset of the population selected for study. We use the sample to make inferences about the population.
Sample Ratio Mismatch (SRM): A randomization failure where the ratio of users in treatment vs. control differs significantly from the design (e.g., 50/50 becomes 48/52), often indicating a bug.
Selection bias: Bias introduced when the individuals in the treatment group differ systematically from those in the control group in ways that affect the outcome (e.g., sicker patients choosing a new treatment).
Sensitivity analysis: An analysis asking how strong an unmeasured confounder — or another assumption violation — would have to be to overturn your conclusion. It turns “we assume no unmeasured confounding” into “here is how much confounding it would take to break this result.”
Sequential exchangeability: Unconfoundedness on repeat: at every decision point in a treatment sequence, all confounders of that decision and the outcome have been measured. It gets harder to defend as the sequence grows longer. Also called sequential unconfoundedness.
Sequential ignorability: The key mediation-analysis assumption: no unmeasured confounding of the treatment–mediator or mediator–outcome relationships, with adjustment covariates measured before treatment. Randomizing the treatment does not buy it — the mediator was never randomized.
Serial correlation: When errors/residuals for the same unit are correlated over time (common in time-series data). Failing to account for it can lead to underestimating standard errors.
SHAP values: A method for explaining a machine learning model’s predictions by attributing each prediction to the input features. It explains the model you fit — a weaker thing than an effect you can defend.
Significance level ($\alpha$): The probability of rejecting the null hypothesis when it is true (Type I error). Typically set at 0.05.
Simpson’s Paradox: A phenomenon where a trend appears in different groups of data but disappears or reverses when these groups are combined. It often highlights the importance of controlling for confounders.
Simultaneity: A type of endogeneity where $X$ and $Y$ influence each other at the same time, making it hard to disentangle the causal effect of one on the other.
Spillover: The effect of one unit’s treatment on other, untreated units — a discount code shared between friends, a boosted seller pulling demand from unboosted ones. The reason “untreated” is not the same as “unaffected.”
Spurious correlation: A statistical association between two variables driven by coincidence or a third unobserved variable (confounder) rather than a causal link.
Stabilized weights: A variance-reducing refinement of inverse probability weights for treatment sequences: each period contributes a ratio of two probabilities instead of a single inverse, pulling extreme weights toward 1 without changing what the weighted model estimates.
Stable Unit Treatment Value Assumption (SUTVA): The assumption that (1) there is no interference between units (one person’s treatment doesn’t affect another’s outcome) and (2) there represents only one version of the treatment.
Staggered adoption: A setting where different units receive the treatment at different times rather than all at once. Common in policy rollouts and product launches. Traditional TWFE DiD may produce biased estimates under staggered adoption, motivating modern DiD estimators.
Standard deviation: A measure of the amount of variation or dispersion in a set of values. Low SD means values are close to the mean.
Standard error: A measure of the statistical accuracy of an estimate. It tells us how much the estimate would vary if we repeated the study with different samples.
Standardized mean difference (SMD): A unit-free measure of the difference in a covariate’s mean between two groups, scaled by a standard deviation. Analysts use it to check balance because gaps measured in different units can be compared on one scale.
Statistical significance: A determination that an observed result is unlikely to be due to random chance, usually indicated by a p-value below a preset threshold (e.g., 0.05).
Stratified randomization: A randomization scheme that first divides users into strata (groups with similar characteristics like platform or country), then randomizes within each stratum. Ensures balanced representation across key segments and can improve precision by reducing variance.
Structural causal model (SCM): The formal framework behind DAGs: a set of equations, one per variable the model explains, each stating how that variable is produced by its direct causes plus unmeasured background factors. The graph is read off the equations, not supplied alongside them.
Synthetic control: A method constructing a weighted combination of multiple control units to create a “synthetic” counterfactual that mimics the treated unit’s pre-intervention trend.

T

Time-varying confounding: When a variable is simultaneously a confounder of a later treatment and an outcome of an earlier one — engagement at week 4 both determines the week-5 offer and reflects the week-1 email. No regression control set fixes it; g-methods exist for exactly this trap.
Treatment assignment: The specific rule or mechanism that determines which units receive the treatment and which receive the control. For instance, in experiments, this is random; in RDD, it is based on a cutoff.
Treatment effect: The causal impact of the intervention on the outcome.
Triangulation: Combining estimates from approaches that rely on different identifying assumptions to answer the same causal question. Agreement is more informative when the approaches do not share their main source of bias.
Triggering: An experimental analysis technique that restricts analysis to users who actually encountered the treatment (e.g., those who visited the modified page). Reduces noise by excluding users who were technically in the experiment but never had a chance to be affected. Must be done carefully to avoid reintroducing selection bias.
Trimming: A technique for addressing lack of common support by excluding units with extreme propensity scores (very close to 0 or 1). By restricting the analysis to the region where treated and control groups overlap, trimming reduces extrapolation bias at the cost of narrowing the population for which causal effects can be estimated.
Two-sided non-compliance: A scenario where both treatment and control groups can deviate from assignment. Some treated users decline treatment, and some control users access it anyway (e.g., by finding the feature on their own).
Two-Stage Least Squares (2SLS): The standard regression implementation of instrumental variables. In the first stage, treatment is predicted using the instrument; in the second stage, outcomes are regressed on the predicted treatment to estimate the LATE.
Two-way fixed effects (TWFE): A standard regression specification for difference-in-differences that includes both unit fixed effects and time fixed effects.
Type I error: A “false positive”—mistakenly finding an effect when there is none (rejecting a true null hypothesis).
Type II error: A “false negative”—mistakingly finding no effect when there actually is one (failing to reject a false null hypothesis).

U

Unbalanced panel: A panel dataset in which some units are not observed in every period (e.g., they enter late or exit early). Can bias DiD estimates when missingness correlates with treatment or outcome (non-random attrition). The opposite is a balanced panel.
Unconditional parallel trends: The standard parallel trends assumption applied without conditioning on covariates: in the absence of treatment, treated and control groups would have followed parallel paths over time. Contrast with Conditional parallel trends.
Uplift modeling: Applying heterogeneous effect estimates to targeting: predict each user’s treatment effect, rank users by predicted lift, and allocate the intervention to those with the highest expected response.

V

Variance: The average of the squared differences from the mean, measuring how spread out the data is.

W

Wald estimator: An IV estimator calculated as the ratio of the reduced form effect (instrument on outcome) to the first stage effect (instrument on treatment).
Waterfall chart: A finance and product-management visualization that shows how a starting figure (e.g., a naive R$140M projection) is whittled down by a sequence of subtractive discounts (decay, adoption ramp, representativeness, cannibalization) to a final number. Used to make every intermediate haircut auditable rather than hiding uncertainty behind a single headline.
Weak instrument: An instrument with a small and weak first stage effect. Weak instruments produce unstable LATE estimates with large standard errors and can bias results toward naive OLS comparisons.
Winner’s curse: The tendency for the estimated effects of winning experiments — those that clear a significance threshold — to overstate the true effect, because selecting on significance partly selects on noise. The practical implication is that the production lift you see after rollout is typically smaller than the lift you measured in the test.
Winsorization: An outlier handling technique that caps extreme values at a specified percentile (e.g., the 99th) rather than removing them entirely. Reduces the influence of outliers on experiment results while preserving sample size.