机器学习
尽可能精确地评估所学假设的性能非常重要。
- 原因之一仅仅是为了判断是否应该使用这个假设。
- 第二个原因是，评估假设是许多学习方法不可或缺的组成部分。
在仅有一组有限数据的情况下，学习一个假设并估计其未来准确性会遇到两个主要困难：
-
估计中的偏差：在训练样本上观察到的学习假设的准确性，通常不是其在未来样本上准确性的良好估计量。由于学习假设是从这些训练样本中推导出来的，它们通常会提供对未来样本准确性的乐观偏差估计。当学习器考虑一个非常丰富的假设空间时，这种情况尤其可能发生，因为它会导致对训练样本的过拟合。为了获得未来准确性的无偏估计，需要在独立于训练样本和假设的某个测试样本集上测试该假设。
-
估计中的方差：即使假设准确性是在独立于训练样本的无偏测试样本集上测量的，测量的准确性仍然可能与真实准确性存在差异，这取决于特定测试样本集的组成。测试样本集越小，预期的方差就越大。
It is important to evaluate the performance of learned hypotheses as precisely as possible.
One reason is simply to understand whether to use the hypothesis.
A second reason is that evaluating hypotheses is an integral component of many learning
methods.
Two key difficulties arise while learning a hypothesis and estimating its future accuracy given only a
limited set of data:
1. Bias in the estimate. The observed accuracy of the learned hypothesis over the training examples is
often a poor estimator of its accuracy over future examples. Because the learned hypothesis was
derived from these examples, they will typically provide an optimistically biased estimate of hypothesis
accuracy over future examples. This is especially likely when the learner considers a very rich hypothesis
space, enabling it to overfit the training examples. To obtain an unbiased estimate of future accuracy,test the hypothesis on some set of test examples chosen independently of the training examples and the hypothesis.
2. Variance in the estimate. Even if the hypothesis accuracy is measured over an unbiased set of test
examples independent of the training examples, the measured accuracy can still vary from the true
accuracy, depending on the makeup of the particular set of test examples. The smaller the set of test
examples, the greater the expected variance.