Variance reduction techniques for importance-weighted cross-validation

Abstract

One can often not evaluate a classifier in the target domain due to the absence of target labels. Fortunately, in the covariate shift setting, the target risk equals the importance-weighted source risk. However, depending on the domain dissimilarity, the variance of the importance weights can drastically increase the variance of the risk estimator. Here we introduce a control variate to reduce the sampling variance of the importance-weighted risk estimator.