cross-validation

Importance-weighted cross-validation produces sub-optimal hyperparameter estimates in problem settings where large weights arise with high probability. We introduce a control variate to increase its robustness to problematically large weights.

I presented my paper on how the importance-weighted risk estimator’s sampling distribution is skewed for small sample sizes. The weights effectively ensure an under- or over-estimation of risk, depending on whether the source distribution has larger or smaller variance than the target distribution, respectively. I explore how this affects hyperparameter selection during importance-weighted cross-validation.

I presented my paper on problems with importance-weighted cross-validation under covariate shift. Under covariate shift, the standard cross-validation estimator is not consistent (i.e. it won’t return optimal hyperparameter estimates). Importance-weighting the cross-validation estimator was deemed to resolve this issue, but we show that it is still not consistent.