|
pulsatrix 1.1.0
|
Calibrating a model's probabilities (KS-13): temperature scaling fitted on held-out data, and the error in log-odds against reference probabilities, by probability band. More...

Go to the source code of this file.
Classes | |
| struct | pulsatrix::TemperatureFit |
| A fitted temperature and the weighted mean log loss before and after it. More... | |
| struct | pulsatrix::LogOddsBand |
| Log-odds errors over the examples whose reference probability is in [lower, upper). More... | |
| struct | pulsatrix::LogOddsErrorReport |
Namespaces | |
| namespace | pulsatrix |
Functions | |
| TemperatureFit | pulsatrix::FitBinaryTemperature (const std::vector< double > &logits, const std::vector< int64_t > &labels, const std::vector< double > &weights={}) |
| Fits T for binary log-odds: P(y = 1) = sigmoid(logit / T). | |
| TemperatureFit | pulsatrix::FitTemperature (const std::vector< std::vector< double > > &logits, const std::vector< int64_t > &labels, const std::vector< double > &weights={}) |
| Fits T for multi-class logits: P(y = k) = softmax(logits / T)_k. | |
| LogOddsErrorReport | pulsatrix::LogOddsError (const std::vector< double > &predicted, const std::vector< double > &reference, const std::vector< double > &weights={}, const std::vector< double > &band_edges={1e-3}, double clip=1e-6) |
| The error in log-odds of predicted probabilities against reference ones. | |
| JsonValue | pulsatrix::ToJson (const LogOddsErrorReport &report) |
| The report as a JSON object: clip, overall, and bands. | |
Calibrating a model's probabilities (KS-13): temperature scaling fitted on held-out data, and the error in log-odds against reference probabilities, by probability band.
Temperature scaling (Guo, Pleiss, Sun and Weinberger, ICML 2017) divides a model's logits by one number T, fitted to minimize the log loss on held-out labeled data. It changes how confident the model is, never which class it ranks first. The log loss is convex in 1/T, so the fit is a safeguarded Newton iteration and finds the global minimum.
Log-odds error measures probabilities against reference probabilities rather than labels: |logit(p) - logit(q)|. It is meant for rare events. The expected calibration error (performance.hpp) averages over bins and is dominated by common cases, so a model that says 1e-2 where the truth is 1e-4 barely moves it; in log-odds that is an error of 4.6. LogOddsError() reports the error separately in each band of the reference probability, with optional importance weights for test sets that oversample rare cases.