Housekeeping

• Each of the groups to share your cleaned data (all the X and the Y variables) with us for adding the treatment and outcome variables
• How to share
  • Create a folder in your respective Group folders named Causal
  • Store your file there with the filename: diseasename-causal.csv

By now, we have covered…

• Estimating the Average Treatment Effect parameter using:
  • Difference-in-means Estimator
  • Direct Estimator
  • IPW Estimator
  • AIPW (Doubly Robust) Estimator

Diagnostics

• Assumptions: Overlap, Unconfoundedness
• How do we know they are true?
  • Basic diagnostics & best practices
  • Although failing these tests should usually cause concern, passing them does not mean that the analysis is entirely free from issues
    • Eg: Even if $\widehat{e}(X_i)$ is bounded away from 0 & 1, they may still be incorrect if we rely on incorrect modeling assumptions

Assessing balance

• Earlier, we saw, in observational settings, covariate distributions can be very different for treated and untreated individuals
• Can lead to biased estimates of the ATE
  • By up- or down-weighting observations based on inverse propensity weights, their averages should be similar
  • we should also expect that averages of basis functions of the covariates should be similar after reweighting
  • This property is called balance

How to check for balance

• Given some variable $Z_i$ (e.g., a covariate $X_{i1}$, or an interaction between covariates $X_{i1}X_{i2}$, or a polynomial in covariates $X_{i1}^2$, etc), calculate the absolute standardized mean difference (ASMD) of $Z_i$ between treated and untreated individuals in our data $$\begin{equation} \frac{|\bar{Z}_1 - \bar{Z}_0|}{\sqrt{s_1^2 + s_0^2}}, \end{equation}$$

where $\bar{Z}_1$ and $\bar{Z}_0$ are sample averages of $Z_i$, and $s_1$ and $s_0$ are standard deviations of $Z_i$ for the two samples of treated and untreated individuals

• Next, check the same quantity for their weighted counterparts $Z_{i}W_i/\hat{e}(X_i)$ and $Z_i(1-W_i)/(1-\hat{e}(X_i))$
• If our propensity scores are well-calibrated, the ASMD for the weighted version should be close to zero

Standardized absolute mean differences

Note how age and polviews – the variables we chose to introduce imbalance earlier – are far from balanced before adjustment, as are other variables that correlate with them

Other Tests

• We can check the entire distribution of covariates (and their transformations)

  • Histograms for treated and untreated individuals with and without adjustment

  • 

    Covariate histograms (unadjusted)

Other Tests

• We can check the entire distribution of covariates (and their transformations)

  • Histograms for treated and untreated individuals with and without adjustment

  • 

    Covariate histograms (unadjusted)
  • 

    Covariate histograms (adjusted)

  • Note how adjusted histograms are very similar for untreated (red) and treated (green) samples – that is what we should expect

Other balance checks

• There exist other types of balance checks
• Check out the R package cobalt. This vignette is a good place to start

Assessing overlap

• It’s also important to check the estimated propensity scores
• If they seem to cluster at zero or one, we should expect IPW and AIPW estimators to behave very poorly
• Here, the propensity score is trimodal

Other ideas

• When overlap fails, the methods described above will not produce good estimates
• Other options:
  • Changing the estimand and targeting the average treatment effect on the treated (ATT) $E[Y_i(1) - Y_i(0) | W_i=1]$
  • Trimming the sample and focusing on some subgroup $G_i$ with bounded propensity scores $E[Y_i(1) - Y_i(0) | G_i]$, as discussed in Crump, Hotz, Imbens and Mitnik (Biometrika, 2009).

What Works $\rightarrow$ What Works for Whom

• Until now, we learned how to estimate the effect of a binary treatment averaged over the entire population - the ATE $$\begin{equation} \tau := \mathrm{E}[Y_i(1) - Y_i(0)], \end{equation}$$

• The average, however, may obscure important details about how different individuals react to the treatment

What Works $\rightarrow$ What Works for Whom

• Until now, we learned how to estimate the effect of a binary treatment averaged over the entire population - the ATE $$\begin{equation} \tau := \mathrm{E}[Y_i(1) - Y_i(0)], \end{equation}$$

• The average, however, may obscure important details about how different individuals react to the treatment

• How to estimate the conditional average treatment effect (CATE):

  • Equation 5 $$\begin{equation} \tau(x) := \mathrm{E}[Y_i(1) - Y_i(0) | X_i = x], \end{equation}$$ which is a “localized” version of the average treatment effect conditional on observable characteristics

Treatment Effects by Groups

• It’s often the case that Eqn. 5 is too general to be immediately useful (especially in high-dimensional settings)
• It can be hard to estimate reliably without making strong modeling assumptions, and hard to summarize in a useful manner after estimation
• In such situations, we will instead try to estimate treatment effect averages for simpler groups: $$\begin{equation} \mathrm{E}[Y_i(1) - Y_i(0) | G_i = g], \end{equation}$$

where $G_i$ indexes subgroups of interest.

How do we discover differential impact sub-groups?

• Pre-specified hypotheses
  • Age, Gender, Race, Other comorbidities
  • Good practice to avoid “p-hacking”
  • As the #dimensions $\uparrow$, identifying sub-groups becomes hard

How do we discover differential impact sub-groups?

• Pre-specified hypotheses
  • Age, Gender, Race, Other comorbidities
  • Good practice to avoid “p-hacking”
  • As the #dimensions $\uparrow$, identifying sub-groups becomes hard
• Data-driven hypotheses
  • Useful in HD settings
  • Valid tests can be attained by sample splitting (“honesty”)

Causal Trees

• Causal trees (Athey and Imbens)(PNAS, 2016) are an intuitive algorithm providing a data-driven approach to partitioning data into subgroups that differ by the magnitude of their treatment effects

• Like decision trees, which partition the covariate space by finding subgroups with similar outcomes, causal trees find subgroups with similar treatment effects

• Even though this is an adaptive method, these subgroups do not need to be specified prior to the experiment

Causal Trees

• To ensure valid estimates of the treatment effect within each subgroup, Athey and Imbens propose a sample-splitting approach that they refer to as honesty:
  • a method is honest if it uses one subset of the data to estimate the model parameters, and a different subset to produce estimates given these estimated parameters
  • honesty implies that the asymptotic properties of treatment effect estimates within leaves are the same as if the tree partition had been exogenously given, and it is one of the assumptions required to produce unbiased and asymptotically normal estimates of the treatment effect

Causal Trees - Algorithm Steps

• High level idea:
  • Divide the sample into three subsets (not necessarily of equal size)
    • Splitting subset is used to fit a decision tree whose objective is modified to maximize heterogeneity in treatment effect estimates across leaves
    • Estimation subset is then used to produce a valid estimate of the treatment effect at each leaf of the fitted tree
    • Test subset can be used to validate the tree estimates
• Steps:
  1. Split the dataset
  2. Fit the tree
  3. Cross-validate
  4. Predict point estimates on estimation sample
  5. Compute standard errors
  6. Predict point estimates on test set

Causal Trees Example

• Values in the cell are the estimated treatment effect and an estimate of the fraction of the population that falls within each leaf
  • Both are estimated using the estimation sample

Caution while interpreting tree splits

• That a tree splits on a particular variable does not mean that this variable is relevant in the underlying data-generating process – it could simply be correlated with some other variable that is

• For the same reason, we should not try to interpret partial effects from tree output.

• But, we can:

  • try to understand how the joint distribution of covariates varies for subgroups with different estimated treatment effects
  • Try to answer what describes subgroups with strongest and weakest estimated treatment effect

Examining joint distribution of covariates across subgroups

• We see the average value of each covariate within each leaf.
• Leaves are ordered from lowest to highest treatment effect estimate
• The color is a normalized distance of each leaf-specific covariate mean from the overall covariate mean, i.e.., $\smash{\Phi^{-1}\left((\widehat{\text{E}}[X_i|L_i] - \widehat{\text{E}}[X_i])/\widehat{\text{Var}}(\widehat{\text{E}}[X_i|L_i])\right)}$
• The rows are in descending order of variation, measured by $\widehat{\text{Var}}(\widehat{\text{E}}[X_i|L_i])/\widehat{\text{Var}}(X_i)$.

Tomorrow …

• Causal forest and the R-learner
• X-learners
• Work with the data!