pulsatrix 1.1.0
Loading...
Searching...
No Matches
performance.hpp File Reference

How well a model does (KS-3): a classifier's accuracy, log loss, Brier score, calibration and per-class scores; a regressor's errors; and the error on each slice of the data, with a test of whether a slice is worse than the rest. More...

#include <cstdint>
#include <string>
#include <vector>
#include "pulsatrix/json.hpp"
#include "pulsatrix/tensor.hpp"
Include dependency graph for performance.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

struct  pulsatrix::ClassificationOptions
 EvaluateClassifier()'s settings. More...
 
struct  pulsatrix::ReliabilityBin
 One confidence bin, (lower, upper]; the first bin also holds a confidence of 0. More...
 
struct  pulsatrix::ClassificationReport
 How well a classifier does on a labeled dataset. More...
 
struct  pulsatrix::RegressionReport
 How well a regressor does. More...
 
struct  pulsatrix::SliceErrorOptions
 PerSliceError()'s settings. More...
 
struct  pulsatrix::SliceError
 One slice's error. More...
 
struct  pulsatrix::SliceErrorReport
 PerSliceError()'s result. More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::SliceTest { pulsatrix::Auto , pulsatrix::FisherExact , pulsatrix::Permutation }
 How PerSliceError() tests a slice against the rest. More...
 

Functions

ClassificationReport pulsatrix::EvaluateClassifier (const Tensor &probabilities, const std::vector< int64_t > &labels, const ClassificationOptions &options={})
 Scores a classifier's probabilities against the labels.
 
RegressionReport pulsatrix::EvaluateRegressor (const std::vector< double > &predictions, const std::vector< double > &targets)
 Scores a regressor's predictions against the targets.
 
SliceErrorReport pulsatrix::PerSliceError (const std::vector< double > &errors, const std::vector< std::string > &slices, const SliceErrorOptions &options={})
 The error on each slice of a dataset, each tested against the rest.
 
SliceErrorReport pulsatrix::PerSliceError (const Tensor &probabilities, const std::vector< int64_t > &labels, const std::vector< std::string > &slices, const SliceErrorOptions &options={})
 A classifier's error rate on each slice: an input is wrong when its most probable class isn't its label.
 
JsonValue pulsatrix::ToJson (const ClassificationReport &report)
 The report as a pulsatrix.classification_report.v1 document.
 
JsonValue pulsatrix::ToJson (const RegressionReport &report)
 The report as a pulsatrix.regression_report.v1 document.
 
JsonValue pulsatrix::ToJson (const SliceErrorReport &report)
 The report as a pulsatrix.slice_error.v1 document.
 
std::string pulsatrix::DescribeSliceError (const SliceErrorReport &report)
 The slices as a plain-text table, worst first, with the test and whether any is worse.
 

Detailed Description

How well a model does (KS-3): a classifier's accuracy, log loss, Brier score, calibration and per-class scores; a regressor's errors; and the error on each slice of the data, with a test of whether a slice is worse than the rest.

These answer the Performance question of the XAI question bank. Every number here matches scikit-learn's (or, for the tests, SciPy's) on the same data; tests/performance_test.cpp checks them against values from tools/generate_performance_reference_values.py.

Calibration is the expected calibration error of Guo, Pleiss, Sun and Weinberger (ICML 2017): group the predictions into equal-width bins of confidence, the largest probability, and average the gap between each bin's accuracy and its mean confidence, weighted by its size. A model whose 0.8-confidence predictions are right 80% of the time is calibrated.

A model can do well on average and fail one group. PerSliceError() reports the error on each slice and tests each against the rest of the data, so a failure confined to one slice isn't averaged away. It is the per-slice error check the Control reason requires.