pulsatrix 1.1.0
Loading...
Searching...
No Matches
calibration.hpp File Reference

Calibrating a model's probabilities (KS-13): temperature scaling fitted on held-out data, and the error in log-odds against reference probabilities, by probability band. More...

#include <cstdint>
#include <vector>
#include "pulsatrix/json.hpp"
Include dependency graph for calibration.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::TemperatureFit
 A fitted temperature and the weighted mean log loss before and after it. More...
 
struct  pulsatrix::LogOddsBand
 Log-odds errors over the examples whose reference probability is in [lower, upper). More...
 
struct  pulsatrix::LogOddsErrorReport
 

Namespaces

namespace  pulsatrix
 

Functions

TemperatureFit pulsatrix::FitBinaryTemperature (const std::vector< double > &logits, const std::vector< int64_t > &labels, const std::vector< double > &weights={})
 Fits T for binary log-odds: P(y = 1) = sigmoid(logit / T).
 
TemperatureFit pulsatrix::FitTemperature (const std::vector< std::vector< double > > &logits, const std::vector< int64_t > &labels, const std::vector< double > &weights={})
 Fits T for multi-class logits: P(y = k) = softmax(logits / T)_k.
 
LogOddsErrorReport pulsatrix::LogOddsError (const std::vector< double > &predicted, const std::vector< double > &reference, const std::vector< double > &weights={}, const std::vector< double > &band_edges={1e-3}, double clip=1e-6)
 The error in log-odds of predicted probabilities against reference ones.
 
JsonValue pulsatrix::ToJson (const LogOddsErrorReport &report)
 The report as a JSON object: clip, overall, and bands.
 

Detailed Description

Calibrating a model's probabilities (KS-13): temperature scaling fitted on held-out data, and the error in log-odds against reference probabilities, by probability band.

Temperature scaling (Guo, Pleiss, Sun and Weinberger, ICML 2017) divides a model's logits by one number T, fitted to minimize the log loss on held-out labeled data. It changes how confident the model is, never which class it ranks first. The log loss is convex in 1/T, so the fit is a safeguarded Newton iteration and finds the global minimum.

Log-odds error measures probabilities against reference probabilities rather than labels: |logit(p) - logit(q)|. It is meant for rare events. The expected calibration error (performance.hpp) averages over bins and is dominated by common cases, so a model that says 1e-2 where the truth is 1e-4 barely moves it; in log-odds that is an error of 4.6. LogOddsError() reports the error separately in each band of the reference probability, with optional importance weights for test sets that oversample rare cases.