Scoring Estimated Causal Effects

Estimator models for estimating causal effects can not be easily evaluated due to the fact that the true effects are not directly observed. This differs from the usual machine learning tasks whose results can be easily evaluated by, for example, the value of loss functions.

Authors in [Schuler] proposed a framework, a schema suggested by [Nie], to evaluate causal effects estimated by different estimator models. Roughly speaking, this framework is a direct application of the double machine learning methods. Specifically, for a causal effect model ce_model() (trained in a training set) that is waited to be evaluated, we

  1. Train a model y_model() to estimate the outcome \(y\) and a x_model() to estimate the treatment \(x\) in a validation set, which is usually not the same as the training set;

  2. In the validation set \(D_{val}\), let \(\tilde{y}\) and \(\tilde{x}\) denote the differences

    \[\begin{split}\tilde{y} & = y - \hat{y}(v), \\ \tilde{x} & = x - \hat{x}(v)\end{split}\]

    where \(\hat{y}\) and \(\hat{x}\) are the estimated outcome and treatment on covariates \(v\) in \(D_{val}\). Furthermore, let

    \[\tau(v)\]

    denote the causal effects estimated by the ce_model() in \(D_{val}\), then the metric of the causal effect for the ce_model is calculated as

    \[E_{V}[(\tilde{y} - \tilde{x} \tau(v))^2].\]

Class Structures

class ylearn.estimator_model.effect_score.RLoss(x_model, y_model, yx_model=None, cf_fold=1, adjustment_transformer=None, covariate_transformer=None, random_state=2022, is_discrete_treatment=False, categories='auto')
Parameters:

  • x_model (estimator, optional) – Machine learning models for fitting x. Any such models should implement the fit() and predict`() (also predict_proba() if x is discrete) methods.

  • y_model (estimator, optional) – The machine learning model which is trained to modeling the outcome. Any valid y_model should implement the fit() and predict() methods.

  • yx_model (estimator, optional) – Machine learning models for fitting the residual of y on residual of x. Only support linear regression model in the current version.

  • cf_fold (int, default=1) – The number of folds for performing cross fit in the first stage.

  • adjustment_transformer (transormer, optional, default=None,) – Transformer for adjustment variables which can be used to generate new features of adjustment variables.

  • covariate_transformer (transormer, optional, default=None,) – Transformer for covariate variables which can be used to generate new features of covariate variables.

  • random_state (int, default=2022) –

  • is_discrete_treatment (bool, default=False) – If the treatment variables are discrete, set this to True.

  • categories (str, optional, default='auto') –

fit(data, outcome, treatment, adjustment=None, covariate=None, combined_treatment=True, **kwargs)

Fit the RLoss estimator model. Note that the training of a DML has two stages, where we implement them in _fit_1st_stage() and _fit_2nd_stage().

Parameters:

  • data (pandas.DataFrame) – Training dataset for training the estimator.

  • outcome (list of str, optional) – Names of the outcome.

  • treatment (list of str, optional) – Names of the treatment.

  • adjustment (list of str, optional, default=None) – Names of the adjustment set ensuring the unconfoundness,

  • covariate (list of str, optional, default=None) – Names of the covariate.

  • combined_treatment (bool, default=True) –

    When combined_treatment is set to True, then if there are multiple treatments, we can use the combined_treatment technique to covert the multiple discrete classification tasks into a single discrete classification task. For an example, if there are two different binary treatments:

    \[\begin{split}treatment_1 &: x_1 | x_1 \in \{'sleep', 'run'\}, \\ treatment_2 &: x_2 | x_2 \in \{'study', 'work'\},\end{split}\]

    then we can convert to these two binary classification tasks into a single classification with 4 different classes:

    \[treatment: x | x \in \{0, 1, 2, 3\},\]

    where, for example, 1 stands for (‘sleep’ and ‘study’).

Returns:

instance of RLoss

Return type:

The fitted RLoss model for evaluating other estimator models in the validation set.

score(test_estimator, treat=None, control=None)

Estimate the causal effect with the type of the quantity.

Parameters:

  • data (pandas.DataFrame, optional, default=None) – The test data for the estimator to evaluate the causal effect, note that the estimator directly evaluate all quantities in the training data if data is None.

  • treat (float or numpy.ndarray, optional, default=None) – In the case of single discrete treatment, treat should be an int or str of one of all possible treatment values which indicates the value of the intended treatment; in the case of multiple discrete treatment, treat should be a list or an ndarray where treat[i] indicates the value of the i-th intended treatment, for example, when there are multiple discrete treatments, array([‘run’, ‘read’]) means the treat value of the first treatment is taken as ‘run’ and that of the second treatment is taken as ‘read’; in the case of continuous treatment, treat should be a float or a ndarray.

  • control (float or numpy.ndarray, optional, default=None) – This is similar to the cases of treat.

Returns:

The score for the test_estimator

Return type:

float

effect_nji(data=None)

Calculate causal effects with different treatment values.

Parameters:

data (pandas.DataFrame, optional, default=None) – The test data for the estimator to evaluate the causal effect, note that the estimator will use the training data if data is None.

Returns:

Causal effects with different treatment values.

Return type:

ndarray

comp_transormer(x, categories='auto')

Transform the discrete treatment into one-hot vectors properly.

Parameters:

  • x (numpy.ndarray, shape (n, x_d)) – An array containing the information of the treatment variables.

  • categories (str or list, optional, default='auto') –

Returns:

The transformed one-hot vectors.

Return type:

numpy.ndarray