Decision trees are intelligible, but do they perform well enough that
you should use them? Have SVMs replaced neural nets, or are neural nets
still best for regression, and SVMs best for classification? Boosting
maximizes margins similar to SVMs, but can boosting compete with SVMs?
And if it does compete, is it better to boost weak models, as theory
might suggest, or to boost stronger models? Bagging is simpler than
boosting -- how well does bagging stack up against boosting? Breiman
said Random Forests are better than bagging and as good as boosting.
Was he right? And what about old friends like logistic regression, KNN,
and naive bayes? Should they be relegated to the history books, or do
they still fill important niches?
In this talk we compare the
performance of these supervised learning methods on a number of
performance criteria: Accuracy, F-score, Lift, Precision/Recall
Break-Even Point, Area under the ROC, Average Precision, Squared Error,
Cross Entropy, and Probability Calibration. The results show that no one
learning method does it all, but some methods can be "repaired" so that
they do very well across all performance metrics. In particular, we
show how to obtain the best probabilities from max margin methods such
as SVMs and boosting via Platt's Method and isotonic regression. We
then describe a new ensemble method that combines select models from
these ten learning methods to yield even better performance. Although
these ensembles perform extremely well, they are too complex for many
applications. We'll describe a model compression method we are
developing to fix that. Finally, if time permits, we'll discuss how the
performance metrics relate to each other, and which of them you probably
should (and shouldn't) use.