khalib

khalib is a classifier probability calibration package powered by the Khiops AutoML suite.

Features

  • KhalibClassifier: A scikit-learn estimator to calibrate classifiers with a similar interface fashion as CalibratedClassifierCV.

  • calibration_error : A function to estimate the Estimated Calibration Error (ECE).

  • build_reliability_diagram : A function that builds a reliability diagram.

These features are based on Khiops’s non-parametric supervised histograms, so there is no need to specify the number and width of the bins, as they are automatically estimated from data.

Installation

Note: We’ll improve this installation procedure soon!

pip install https://github.com/KhiopsLab/khalib/archive/refs/tags/0.1.zip

How does it work

khalib proposes histogram-based calibration and its error estimation. Its differentiating factor is that uses Khiops to construct the histogram in which \(P(Y = 1 | S)\) is estimated. These histograms have the following properties:

  • They balance class purity, model complexity and data fitness.

  • They are non-parametric: The optimal histogram is searched without constraint in number of bins or bin width. This implies that the user doesn’t need to set a number of bins nor their widths.

See the Quickstart and API reference to learn how to use the library.