khalib

khalib is a classifier probability calibration package powered by the Khiops AutoML suite.

Features

  • KhalibClassifier: A scikit-learn estimator to calibrate classifiers with a similar interface as CalibratedClassifierCV.

  • calibration_error : A function to estimate the Estimated Calibration Error (ECE).

  • build_reliability_diagram : A function to build a reliability diagram.

These features are based on Khiops’s non-parametric supervised histograms, so there is no need to specify the number and width of the bins, as they are automatically estimated from data.

Installation

pip install khalib

How does it work

khalib proposes histogram-based calibration and its error estimation. Its differentiating factor is that uses Khiops to construct the histogram in which \(P(Y = 1 | S)\) is estimated. These histograms have the following properties:

  • They balance class purity, model complexity and data fitness.

  • They are non-parametric: The optimal histogram is searched without constraint in number of bins or bin width. This implies that the user doesn’t need to set a number of bins nor their widths.

See the Quickstart and API reference to learn how to use the library.