|
pulsatrix 1.1.0
|
How well a model does (KS-3): a classifier's accuracy, log loss, Brier score, calibration and per-class scores; a regressor's errors; and the error on each slice of the data, with a test of whether a slice is worse than the rest. More...
#include <cstdint>#include <string>#include <vector>#include "pulsatrix/json.hpp"#include "pulsatrix/tensor.hpp"

Go to the source code of this file.
Classes | |
| struct | pulsatrix::ClassificationOptions |
| EvaluateClassifier()'s settings. More... | |
| struct | pulsatrix::ReliabilityBin |
| One confidence bin, (lower, upper]; the first bin also holds a confidence of 0. More... | |
| struct | pulsatrix::ClassificationReport |
| How well a classifier does on a labeled dataset. More... | |
| struct | pulsatrix::RegressionReport |
| How well a regressor does. More... | |
| struct | pulsatrix::SliceErrorOptions |
| PerSliceError()'s settings. More... | |
| struct | pulsatrix::SliceError |
| One slice's error. More... | |
| struct | pulsatrix::SliceErrorReport |
| PerSliceError()'s result. More... | |
Namespaces | |
| namespace | pulsatrix |
Enumerations | |
| enum class | pulsatrix::SliceTest { pulsatrix::Auto , pulsatrix::FisherExact , pulsatrix::Permutation } |
| How PerSliceError() tests a slice against the rest. More... | |
Functions | |
| ClassificationReport | pulsatrix::EvaluateClassifier (const Tensor &probabilities, const std::vector< int64_t > &labels, const ClassificationOptions &options={}) |
| Scores a classifier's probabilities against the labels. | |
| RegressionReport | pulsatrix::EvaluateRegressor (const std::vector< double > &predictions, const std::vector< double > &targets) |
| Scores a regressor's predictions against the targets. | |
| SliceErrorReport | pulsatrix::PerSliceError (const std::vector< double > &errors, const std::vector< std::string > &slices, const SliceErrorOptions &options={}) |
| The error on each slice of a dataset, each tested against the rest. | |
| SliceErrorReport | pulsatrix::PerSliceError (const Tensor &probabilities, const std::vector< int64_t > &labels, const std::vector< std::string > &slices, const SliceErrorOptions &options={}) |
| A classifier's error rate on each slice: an input is wrong when its most probable class isn't its label. | |
| JsonValue | pulsatrix::ToJson (const ClassificationReport &report) |
The report as a pulsatrix.classification_report.v1 document. | |
| JsonValue | pulsatrix::ToJson (const RegressionReport &report) |
The report as a pulsatrix.regression_report.v1 document. | |
| JsonValue | pulsatrix::ToJson (const SliceErrorReport &report) |
The report as a pulsatrix.slice_error.v1 document. | |
| std::string | pulsatrix::DescribeSliceError (const SliceErrorReport &report) |
| The slices as a plain-text table, worst first, with the test and whether any is worse. | |
How well a model does (KS-3): a classifier's accuracy, log loss, Brier score, calibration and per-class scores; a regressor's errors; and the error on each slice of the data, with a test of whether a slice is worse than the rest.
These answer the Performance question of the XAI question bank. Every number here matches scikit-learn's (or, for the tests, SciPy's) on the same data; tests/performance_test.cpp checks them against values from tools/generate_performance_reference_values.py.
Calibration is the expected calibration error of Guo, Pleiss, Sun and Weinberger (ICML 2017): group the predictions into equal-width bins of confidence, the largest probability, and average the gap between each bin's accuracy and its mean confidence, weighted by its size. A model whose 0.8-confidence predictions are right 80% of the time is calibrated.
A model can do well on average and fail one group. PerSliceError() reports the error on each slice and tests each against the rest of the data, so a failure confined to one slice isn't averaged away. It is the per-slice error check the Control reason requires.