ROC Analysis

Description

Receiver Operating Characteristic Analysis (ROC Analysis) is related in a direct and natural way to cost/benefit analysis of diagnostic decision making. Widely used in medicine for many decades, it has been introduced relatively recently in machine learning. In this context, ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution. Furthermore, the Area Under the ROC Curve (AUC) has been shown to be a better evaluation measure than accuracy in contexts with variable misclassification costs and/or imbalanced datasets. AUC is also the standard measure when using classifiers to rank examples, and, hence, is used in applications where ranking is crucial, such as campaign design, model combination, collaboration strategies, and co-learning.

Nevertheless, there are many open questions and some limitations that hamper a broader use and applicability of ROC analysis. Its use in data mining and machine learning is still below its full potential. An important limitation of ROC analysis, despite some recent progress, is its possible but difficult extension for more than two classes.

(cited form ROCML)

Keywords

Cross Relation

Related Topic: cost sensitive learning, class imbalance problem, et al.

Alternatives: PN plots, precision-recall curves (PR curve), DET curves, cost curves, etc.

Previous Workshops

• 1st Workshop on ROC Analysis in AI (Workshop on ECAI'2004)
• 2nd Workshop on ROC Analysis in ML (Workshop on ICML'2005)  

  The main goal of this first workshop was to foster the cross-fertilisation of ideas and applications of ROC analysis with related areas in artificial intelligence and to gather points of views from broad AI fields. This second workshop will focus on the point of view of machine learning, in particular on some issues raised during the first workshop, e.g. ROC analysis software repository, multiclass extension, statistical analysis.

• 3rd Workshop on ROC Analysis in ML (Workshop on ICML'2006) [paper list]

  This third workshop is intended to investigate on the hot topics identified during the two previous workshops (e.g. multiclass extension, statistical analysis, alternative approaches), on the one hand, and to encourage cross-fertilisation with ROC practitioners in medicine.

Special Issue

Paper List

Researchers in This Field

• Hendrik Blockeel , K.U.Leuven , Belgium .
• Stephan Dreiseitl FHS Hagenberg, Austria.
• Richard M. Everson , University of Exeter, UK.
• Cèsar Ferri , Technical University of Valencia, Spain.
• Jonathan E. Fieldsend , University of Exeter, UK.
• Peter Flach , University of Bristol , UK .
• Johannes Fürnkranz , TU Darmstadt, Germany .
• José Hernández-Orallo , Technical University of Valencia, Spain .
• Rob Holte , University of Alberta, Canada.
• Nicolas Lachiche , University of Strasbourg, France.
• Sofus A. Macskassy , New York University, USA.
• Alain Rakotomamonjy , Insa de Rouen, France.
• Francesco Tortorella , University of Cassino , Italy
• ...

   

Last Modified: 2007-04-05 by Xu-Ying Liu

Machine Learning Topics

Cost-Sensitive Learning

Imbalance Problem

Rare Event Detection

Concept Drift

ROC Analysis