Abstract
Drummond and Holte introduced the theory of cost curves, a graphical technique for visualizing the performance of binary classifiers over the full range of possible class distributions and misclassification costs. In this paper, we use this concept to develop the Improvement Curve, a new performance metric for predictive models. Improvement curves are more user-friendly than cost curves and enable direct inter-classifier comparisons. We apply improvement curves to measure risk-assessment processes at Canada’s marine ports. We illustrate how implementing even a basic predictive model would lead to improved efficiency for the Canada Border Services Agency, regardless of class distributions or misclassification costs.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Boström, H. (2005). Maximizing the area under the ROC curve using incremental reduced error pruning. In Proceedings of the ICML 2005 workshop on ROC analysis in machine learning.
Boström, H. (2007). Maximizing the area under the ROC curve with decision lists and rule sets. In Proceedings of the SIAM international conference on data mining (pp. 27–34).
Bradley, A. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(6), 1145–1159.
Drummond, C., & Holte, R. C. (2000a). Explicitly representing expected cost: an alternative to ROC representation. In Proceedings of the fifth international conference on knowledge discovery and data mining (pp. 155–164).
Drummond, C., & Holte, R. C. (2000b). Exploiting the cost (in)sensitivity of decision tree splitting criteria. In Proceedings of the 17th international conference on machine learning (pp. 239–246).
Drummond, C., & Holte, R. C. (2004). What ROC curves can’t do (and cost curves can). In ECAI workshop on ROC analysis in artificial intelligence.
Drummond, C., & Holte, R. C. (2006). Cost curves: an improved method for visualizing classifier performance. Machine Learning, 65, 95–130.
Fawcett, T. (2001). Using rule sets to maximize ROC performance. In Proceedings of the 2001 IEEE international conference on data mining.
Flach, P. A. (2003). The geometry of ROC space: understanding machine learning metrics through ROC isometrics. In Proceedings of the twentieth international conference on machine learning (pp. 194–201).
Flach, P. A. (2004). The many faces of ROC analysis in machine learning. In Proceedings of the twenty-first international conference on machine learning.
Furnkranz, J., & Flach, P. A. (2003). An analysis of rule evaluation metrics. In Proceedings of the twentieth international conference on machine learning (pp. 202–209).
Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29–36.
Holte, R. C. (2006). Elaboration on two points raised in “Classifier technology and the illusion of progress”. Statistical Science, 21(1), 24–26.
Provost, F., & Fawcett, T. (1997). Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. In Proceedings of the third international conference on knowledge discovery and data mining (pp. 43–48).
Provost, F., & Fawcett, T. (2001). Robust classification for imprecise environments. Machine Learning, 42, 203–231.
Swets, J. A. (1988). Measuring the accuracy of diagnostic systems. Science, 240, 1285–1293.
Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1–26.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hoshino, R., Coughtrey, D., Sivaraja, S. et al. Applications and extensions of cost curves to marine container inspection. Ann Oper Res 187, 159–183 (2011). https://doi.org/10.1007/s10479-009-0669-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10479-009-0669-2