|
pulsatrix 1.1.0
|
Split conformal prediction (KS-3): prediction sets for a classifier and intervals for a regressor that contain the truth with a chosen probability, whatever the model (Vovk, Gammerman and Shafer, 2005; Angelopoulos and Bates, arXiv 2107.07511). More...
#include <cstdint>#include <string>#include <vector>#include "pulsatrix/json.hpp"#include "pulsatrix/tensor.hpp"
Go to the source code of this file.
Classes | |
| struct | pulsatrix::ConformalOptions |
| CalibrateConformalClassifier()'s settings. More... | |
| struct | pulsatrix::ConformalClassifier |
| A calibrated conformal classifier. More... | |
| struct | pulsatrix::ConformalRegressor |
| A calibrated conformal regressor. More... | |
| struct | pulsatrix::PredictionInterval |
| One prediction interval. More... | |
| struct | pulsatrix::CoverageReport |
| How often the sets or intervals held the truth, and how large they were. More... | |
Namespaces | |
| namespace | pulsatrix |
Enumerations | |
| enum class | pulsatrix::ConformalScore { pulsatrix::Lac , pulsatrix::Aps } |
| How a conformal classifier scores an example. More... | |
Functions | |
| ConformalClassifier | pulsatrix::CalibrateConformalClassifier (const Tensor &probabilities, const std::vector< int64_t > &labels, const ConformalOptions &options={}) |
| Calibrates on held-out examples the model wasn't trained on. | |
| std::vector< std::vector< int64_t > > | pulsatrix::PredictionSets (const ConformalClassifier &classifier, const Tensor &probabilities) |
| The prediction set of each input: the classes, most probable first. | |
| ConformalRegressor | pulsatrix::CalibrateConformalRegressor (const std::vector< double > &predictions, const std::vector< double > &targets, double alpha=0.1) |
| Calibrates intervals ŷ ± q̂ from a point regressor's predictions on held-out examples. | |
| ConformalRegressor | pulsatrix::CalibrateConformalQuantileRegressor (const std::vector< double > &lower, const std::vector< double > &upper, const std::vector< double > &targets, double alpha=0.1) |
| Calibrates CQR intervals [lo − q̂, hi + q̂] from a quantile regressor's lower and upper predictions, such as its alpha/2 and 1 − alpha/2 quantiles, on held-out examples. | |
| std::vector< PredictionInterval > | pulsatrix::PredictionIntervals (const ConformalRegressor ®ressor, const std::vector< double > &predictions) |
| The intervals for a point regressor's predictions (absolute_residual). | |
| std::vector< PredictionInterval > | pulsatrix::PredictionIntervals (const ConformalRegressor ®ressor, const std::vector< double > &lower, const std::vector< double > &upper) |
| The intervals for a quantile regressor's predictions (cqr). | |
| CoverageReport | pulsatrix::EvaluateCoverage (const std::vector< std::vector< int64_t > > &sets, const std::vector< int64_t > &labels) |
| CoverageReport | pulsatrix::EvaluateCoverage (const std::vector< PredictionInterval > &intervals, const std::vector< double > &targets) |
| JsonValue | pulsatrix::ToJson (const CoverageReport &report) |
The report as a pulsatrix.coverage.v1 document, without the per-input misses. | |
Split conformal prediction (KS-3): prediction sets for a classifier and intervals for a regressor that contain the truth with a chosen probability, whatever the model (Vovk, Gammerman and Shafer, 2005; Angelopoulos and Bates, arXiv 2107.07511).
A model's probability of 0.9 doesn't mean it is right 90% of the time. Conformal prediction turns any model's scores into a guarantee. Score each example of a held-out calibration set by how badly the model fits it, and take the ⌈(n + 1)(1 − α)⌉-th smallest score, q̂. A new input's prediction set holds every answer that would score at most q̂, so it can be empty. If the calibration examples and the new one are exchangeable, the set contains the true answer with probability at least 1 − α, and at most 1 − α + 1/(n + 1).
The guarantee is marginal: it holds on average over inputs, not for each one or each group. Check the coverage of each slice with PerSliceError() on the misses (EvaluateCoverage()).
Scores:
LAC sets and absolute-residual intervals match MAPIE 1.1.0's SplitConformalClassifier and SplitConformalRegressor; tests/conformal_test.cpp checks them against tools/generate_conformal_reference_values.py.