Covariate-Shift

I presented my paper on how the importance-weighted risk estimator’s sampling distribution is skewed for small sample sizes. The weights effectively ensure an under- or over-estimation of risk, depending on whether the source distribution has larger or smaller variance than the target distribution, respectively. I explore how this affects hyperparameter selection during importance-weighted cross-validation.

One can often not evaluate a classifier in the target domain due to the absence of target labels. Fortunately, in the covariate shift …

I presented my paper on problems with importance-weighted cross-validation under covariate shift. Under covariate shift, the standard cross-validation estimator is not consistent (i.e. it won’t return optimal hyperparameter estimates). Importance-weighting the cross-validation estimator was deemed to resolve this issue, but we show that it is still not consistent.