65 std::vector<ReliabilityBin>
bins;
73 std::vector<double>
f1;
110 const std::vector<double>& targets);
186 const std::vector<std::string>& slices,
One JSON value: null, a boolean, a number, a string, an array or an object.
Definition json.hpp:22
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
A small strict JSON value, parser and deterministic writer (RFC 8259).
Definition acquisition_functions.hpp:16
SliceTest
How PerSliceError() tests a slice against the rest.
Definition performance.hpp:113
@ Auto
Fisher's exact test when every error is 0 or 1, a permutation test otherwise.
@ Permutation
A permutation test, one-sided, of the slice's mean error against the rest's.
std::string DescribeSliceError(const SliceErrorReport &report)
The slices as a plain-text table, worst first, with the test and whether any is worse.
JsonValue ToJson(const AttackResult &result, const AttackOptions &options)
The result as a pulsatrix.attack.v1 document, without the adversarial inputs.
RegressionReport EvaluateRegressor(const std::vector< double > &predictions, const std::vector< double > &targets)
Scores a regressor's predictions against the targets.
SliceErrorReport PerSliceError(const std::vector< double > &errors, const std::vector< std::string > &slices, const SliceErrorOptions &options={})
The error on each slice of a dataset, each tested against the rest.
ClassificationReport EvaluateClassifier(const Tensor &probabilities, const std::vector< int64_t > &labels, const ClassificationOptions &options={})
Scores a classifier's probabilities against the labels.
EvaluateClassifier()'s settings.
Definition performance.hpp:32
int64_t bins
Equal-width confidence bins for the calibration error; Guo et al. used 15.
Definition performance.hpp:34
How well a classifier does on a labeled dataset.
Definition performance.hpp:49
int64_t n
Definition performance.hpp:50
std::vector< double > f1
Definition performance.hpp:73
double brier
Brier's multi-class score: the mean over inputs of sum_k (p_k - y_k)^2, from 0 to 2 (scikit-learn's b...
Definition performance.hpp:59
double accuracy
The fraction of inputs whose most probable class is the label.
Definition performance.hpp:53
std::vector< double > recall
Definition performance.hpp:72
int64_t classes
Definition performance.hpp:51
std::vector< ReliabilityBin > bins
The reliability diagram: one entry per bin, from confidence 0 to 1.
Definition performance.hpp:65
std::vector< std::vector< int64_t > > confusion
confusion[true class][predicted class]: how many inputs.
Definition performance.hpp:67
double macro_f1
The unweighted mean of f1 over the classes.
Definition performance.hpp:77
double ece
The expected calibration error: sum over bins of count/n |accuracy - confidence|.
Definition performance.hpp:61
double log_loss
The mean negative log probability of the label (scikit-learn's log_loss). A probability of 0 counts a...
Definition performance.hpp:56
std::vector< double > precision
Per class: the fraction of its predictions that are right, of its inputs found, and their harmonic me...
Definition performance.hpp:71
std::vector< int64_t > support
Per class: how many inputs have it as their label.
Definition performance.hpp:75
double mce
The maximum calibration error: the largest |accuracy - confidence| of a non-empty bin.
Definition performance.hpp:63
How well a regressor does.
Definition performance.hpp:94
double rmse
Definition performance.hpp:97
double r2
The coefficient of determination, 1 - SSE / SST (scikit-learn's r2_score). When the targets are const...
Definition performance.hpp:101
int64_t n
Definition performance.hpp:95
double mae
Definition performance.hpp:98
double mse
Definition performance.hpp:96
One confidence bin, (lower, upper]; the first bin also holds a confidence of 0.
Definition performance.hpp:38
double confidence
The mean confidence of the predictions in the bin; 0 when it is empty.
Definition performance.hpp:43
double upper
Definition performance.hpp:40
double accuracy
The fraction of them that are right; 0 when it is empty.
Definition performance.hpp:45
int64_t count
Definition performance.hpp:41
double lower
Definition performance.hpp:39
PerSliceError()'s settings.
Definition performance.hpp:124
int64_t permutations
Permutation test only: how many relabelings, and their seed.
Definition performance.hpp:131
double alpha
The family-wise error rate: a slice is worse when its p-value is at most alpha / (number of slices),...
Definition performance.hpp:128
uint64_t seed
Definition performance.hpp:132
SliceTest test
Definition performance.hpp:129
PerSliceError()'s result.
Definition performance.hpp:155
double alpha
Definition performance.hpp:161
double error
The mean error over the whole dataset.
Definition performance.hpp:158
std::string test
The test that ran: "fisher_exact" or "permutation".
Definition performance.hpp:160
int64_t n
Definition performance.hpp:156
std::vector< SliceError > slices
Every slice, the highest error first; ties keep the order slices first appear in.
Definition performance.hpp:163
bool any_worse() const
Whether any slice is worse than the rest.
One slice's error.
Definition performance.hpp:136
int64_t count
Definition performance.hpp:138
double error
The slice's mean error: its error rate when the errors are 0 or 1.
Definition performance.hpp:140
bool worse
p_value <= alpha / (number of slices).
Definition performance.hpp:151
double p_value
One-sided: how likely an error this much above the rest's is if slice membership doesn't matter....
Definition performance.hpp:149
double rest_error
The mean error of every input outside the slice.
Definition performance.hpp:146
double upper
Definition performance.hpp:144
double lower
An interval for it at level 1 - alpha: Wilson's score interval for 0/1 errors, the normal approximati...
Definition performance.hpp:143
std::string name
Definition performance.hpp:137
N-dimensional tensor – owns a buffer via DeviceBackend*, RAII (Rule of Five).