机器学习

The simplest way to combine multiple classifiers is by voting, which corresponds to taking a
linear combination of the learn
ers, Refer figure 1.
This is also known as ensembles and linear opinion pools. In the sim plest case, all learners are
given equal weight and we have simple voting that corresponds to taking an average. Still, taking a
(weighted) sum is only one of the possibilities and there are also other combination rules, as shown in
table 1. If the outputs are not posterior probabilities, these rules require that outputs be normalized to
the same scale
Table 1 - Classifier combination rules
An example of the use of these rules is shown in table 2, which demonstrates the effects of
different rules. Sum rule is the most intuitive and is the most widely used in practice. Median rule is
more robust to outliers; minimum and maximum rules are pessimistic and optimistic, respectively. With
the product rule, each learner has veto power; regardless of the other ones, if one learner has an
output of 0, the overall output goes to 0. Note that after the combination rules, yi do not necessarily
sum up to 1.
Table 2: Example of combination rules on three learners and three classes
In weighted sum, dji is the vote of learner j for class Ci and wj is the weight of its vote. Simple voting is a
special case where all voters have equal weight, namely, wj = 1/L. In classification, this is called plurality
voting where the class having the maximum number of votes is the winner.
When there are two classes, this is majority voting where the winning class gets more than half of the
votes. If the voters can also supply the additional information of how much they vote for each class
(e.g., by the posterior probability), then after normalization, these can be used as weights in a weighted
voting scheme. Equivalently, if dji are the class posterior probabilities, P(Ci | x,Mj ), then we can just sum
them up (wj = 1/L) and choose the class with maximum yi .
In the case of regression, simple or weighted averaging or median can be used to fuse the outputs of
base-regressors. Median is more robust to noise than the average.
Another possible way to find wj is to assess the accuracies of the learners (regressor or classifier) on a
separate validation set and use that information to compute the weights, so that we give more weights
to more accurate learners.
Voting schemes can be seen as approximations under a Bayesian framework with weights
approximating prior model probabilities, and model decisions approximating model-conditional
likelihoods.
Simple voting corresponds to a uniform prior. If we have a prior distribution preferring simpler models,
this would give larger weights to them. We cannot integrate over all models; we only choose a subset
for which we believe P(Mj ) is high, or we can have another Bayesian step and calculate P(Ci | x,Mj ), the
probability of a model given the sample, and sample high probable models from this density.
Let us assume that dj are iid with expected value E[dj] and variance Var(dj ), then when we take a simple
63
average with wj = 1/L, the expected value and variance of the output are
We see that the expected value does not change, so the bias does not change. But variance, and
therefore mean square error, decreases as the number of independent voters, L, increases. In the
general case,
which implies that if learners are positively correlated, variance (and error) increase. We can thus view
using different algorithms and input features as efforts to decrease, if not completely eliminate, the
positive correlation.