Importance-Weighting

Importance-weighted cross-validation produces sub-optimal hyperparameter estimates in problem settings where large weights arise with high probability. We introduce a control variate to increase its robustness to problematically large weights.

I presented my paper on how the importance-weighted risk estimator’s sampling distribution is skewed for small sample sizes. The weights effectively ensure an under- or over-estimation of risk, depending on whether the source distribution has larger or smaller variance than the target distribution, respectively. I explore how this affects hyperparameter selection during importance-weighted cross-validation.

One can often not evaluate a classifier in the target domain due to the absence of target labels. Fortunately, in the covariate shift …