Our new article has been accepted in Bioinformatics:

Schwarz J and Heider D: GUESS: Projecting Machine Learning Scores to Well-Calibrated Probability Estimates for Clinical Decision Making. Bioinformatics 2018, in press.

Abstract
Motivation: Clinical decision-support-systems (CDSS) have been applied in numerous fields, ranging from
cancer survival towards drug resistance prediction. Nevertheless, CDSS typically have a caveat: many of
them are perceived as black-boxes by non-experts and, unfortunately, the obtained scores cannot usually
be interpreted as class probability estimates. In probability-focused medical applications, it is not sufficient
to perform well with regards to discrimination and, consequently, various calibration methods have been
developed to enable probabilistic interpretation. The aims of the current study were 1) to develop a tool for
fast and comparative analysis of different calibration methods, 2) to demonstrate their limitations for the
use on clinical data, and 3) to introduce our novel method GUESS.

Results: We compared the performances of two different state-of-the-art calibration methods, namely
histogram binning and Bayesian Binning in Quantiles (BBQ), as well as our novel method GUESS on
both, simulated and real-world datasets. GUESS demonstrated calibration performance comparable to
the state-of-the-art methods and always retained accurate class discrimination. GUESS showed superior
calibration performance in small datasets and therefore may be an optimal calibration method for typical
clinical datasets. Moreover, we provide a framework (CalibratR) for R, which can be used to identify
the most suitable calibration method for novel datasets in a timely and efficient manner. Using calibrated
probability estimates instead of original classifier scores will contribute to the acceptance and dissemination
of machine learning based classification models in cost-sensitive applications, such as clinical research.

Availability: GUESS as part of CalibratR can be downloaded at CRAN
Contact: dominik.heider AT uni-marburg.de
Supplementary information: Supplementary data are available at Bioinformatics online.