Skip to content

Performance

How well does the model do, and where does it fail? This is the Performance question of the XAI question bank, and an explanation can't answer it: an attribution says why the model made a prediction, not whether the prediction is right. Pulsatrix answers it with the metrics in pulsatrix/performance.hpp (KS-3):

  • EvaluateClassifier: accuracy, log loss, the Brier score, calibration with a reliability diagram, a confusion matrix, and each class's precision, recall and F1;
  • EvaluateRegressor: mean squared and absolute error, and R²;
  • PerSliceError: the error on each slice of the data, each tested against the rest;
  • McDropoutSamples and EnsembleSamples, in pulsatrix/uncertainty.hpp: how sure the model is of each prediction, and whether more data would make it surer;
  • split conformal prediction, in pulsatrix/conformal.hpp: prediction sets and intervals that hold the truth with a chosen probability, for any model;
  • FitTemperature and LogOddsError, in pulsatrix/calibration.hpp (KS-13): temperature scaling fitted on held-out data, and the error of rare-event probabilities in log-odds.

Every number matches scikit-learn's on the same data, and the slice test matches SciPy's fisher_exact. tests/performance_test.cpp checks them against values from tools/generate_performance_reference_values.py; tests/uncertainty_test.cpp checks the uncertainty split against numpy.

The recipe How sure is the model? runs all of it on a real model: a head on ESM-2 that predicts which mutations of TEM-1 β-lactamase still work, with its calibration, its uncertainty at positions it never saw, and where its conformal guarantee breaks.

Scoring a classifier

EvaluateClassifier takes the model's probabilities, (N, K), and the true classes:

#include "pulsatrix/performance.hpp"

Tensor probabilities = softmax.forward(model.forward(inputs));   // (N, K)
ClassificationReport r = EvaluateClassifier(probabilities, labels);
std::printf("accuracy %.3f, log loss %.3f, Brier %.3f, ECE %.3f\n", r.accuracy, r.log_loss, r.brier, r.ece);

The predicted class is the most probable one, the lower index on a tie. Rows aren't renormalized. The definitions are scikit-learn's:

Field Definition scikit-learn
accuracy the fraction of inputs whose predicted class is the label accuracy_score
log_loss the mean of −log p(label); a probability of 0 counts as 10⁻¹⁵ log_loss
brier the mean over inputs of Σₖ (pₖ − yₖ)², from 0 to 2 (Brier, 1950) brier_score_loss(scale_by_half=False)
precision, recall, f1 per class; 0 where there is nothing to divide by precision_recall_fscore_support(zero_division=0)
confusion[true][predicted] counts confusion_matrix

Calibration

A model is calibrated when its confidence means what it says: of the predictions it makes with probability 0.8, 80% are right. Modern networks are often overconfident (Guo, Pleiss, Sun and Weinberger, ICML 2017).

EvaluateClassifier groups the predictions into equal-width bins of confidence, the largest probability: 15 by default, as Guo et al. used (ClassificationOptions::bins). Each bin is (lower, upper]. The report gives:

  • bins: each bin's count, mean confidence and accuracy, the reliability diagram;
  • ece, the expected calibration error: the gap between accuracy and confidence, averaged over the bins and weighted by their size;
  • mce, the maximum calibration error: the largest gap in a non-empty bin.

On the test data, 200 predictions from 4 classes:

accuracy 0.575  log loss 1.411  Brier 0.662  ECE 0.152  MCE 0.455
(0.533, 0.600]  n=28  confidence 0.567  accuracy 0.393
(0.733, 0.800]  n=17  confidence 0.769  accuracy 0.471
(0.800, 0.867]  n=28  confidence 0.834  accuracy 0.536
(0.867, 0.933]  n=23  confidence 0.904  accuracy 0.696

Its most confident predictions are right far less often than it claims.

RenderReliabilityDiagramSvg draws the bins, one line per classifier, so two models can be compared on one figure. A calibrated classifier lies on the diagonal. The marks grow with the bin's share of the inputs, and the strip below shows that share: a bin of three inputs can sit far from the diagonal by chance.

#include "pulsatrix/viz/performance_views.hpp"

std::ofstream("reliability.svg") << RenderReliabilityDiagramSvg({{"one network", single}, {"ensemble", ensemble}});

pulsatrix_svg one.json ensemble.json -o reliability.svg draws the same from the reports' documents (WriteJson(ToJson(report))), each named after its file.

A reliability diagram of two classifiers that make the same guesses, one more confidently: its points sit further below the diagonal, and its ECE is higher

Note

The ECE depends on the number of bins, and with few predictions per bin it is biased upward. Compare models with the same bins, and read the reliability diagram, not just the number.

Temperature scaling

Temperature scaling divides a model's logits by one number T, fitted to minimize the log loss on held-out labeled data (Guo et al.). T > 1 makes an overconfident model less sure. It never changes which class ranks first, so accuracy is unchanged.

#include "pulsatrix/calibration.hpp"

// Held-out data the model wasn't trained on: logits, labels, and optional per-example weights.
const TemperatureFit fit = FitTemperature(held_out_logits, held_out_labels);       // (N, K) rows
const TemperatureFit binary = FitBinaryTemperature(held_out_log_odds, held_out_labels);
// Then use logits / fit.temperature everywhere else.

The log loss is convex in 1/T, so the fit is exact up to rounding (about ten Newton steps). Two cases have no finite answer, and fit.converged is false for them: - The labels are separated: every label already has the largest logit. A smaller T is always better, and the fit stops at T = 10⁻⁶. - The logits are no better than chance, or point the wrong way. The fit stops at T = 10⁶.

Calibration of rare events

The ECE is useless for rare events. It averages over bins, so the common cases dominate it. A model that says 10⁻² where the truth is 10⁻⁴ is wrong by a factor of 100, but that barely moves the ECE. When reference probabilities are available (an exact model, a simulator, a larger model), LogOddsError measures |logit(p) − logit(q)| instead, separately in bands of the reference probability:

const LogOddsErrorReport r = LogOddsError(predicted, reference, /*weights=*/{}, /*band_edges=*/{1e-3});
// r.bands[0] holds references below 1e-3, r.bands[1] the rest; r.overall all of them.
// Each band: count, weight, mean_abs, bias (> 0: predictions too high), rms, median_abs, p90_abs, max_abs.
  • Clipping: both probabilities are clipped to [10⁻⁶, 1 − 10⁻⁶] first (clip), so a reference of exactly 0 gives a finite error.
  • Weights: importance weights, for test sets that oversample rare cases. A weight counts as that many copies of the example, in the means and in the quantiles.
  • Report: ToJson(report) gives the report as JSON.

Error by slice

A model can do well on average and fail one group. PerSliceError reports the error on each slice, such as a demographic group, a data source or a class of input. It tests each slice against the rest of the data, so a failure confined to one group isn't averaged away:

SliceErrorReport s = PerSliceError(probabilities, labels, slices);   // slices: one name per input
std::cout << DescribeSliceError(s);
error 0.425 over 200 inputs; each slice against the rest (Fisher's exact test, alpha 0.05 over 3 slices)
  slice  n       error    95% interval      rest     p
  c      66      0.591    0.470 to 0.701    0.343    0.000751  WORSE
  a      67      0.358    0.254 to 0.478    0.459    0.935
  b      67      0.328    0.228 to 0.447    0.474    0.983
at least one slice is worse than the rest

How each slice is tested:

  • Errors of 0 or 1 (right or wrong) use Fisher's exact test, one-sided. Is the slice wrong more often than the rest? The interval is Wilson's score interval.
  • Losses use a permutation test of the slice's mean against the rest's: 999 relabelings that keep the slice sizes. The interval is the normal approximation. Pass the losses as a vector: PerSliceError(errors, slices).
  • A slice is worse when its p-value is at most α divided by the number of slices: Bonferroni's correction, so testing many slices doesn't find one by chance. α is 0.05 by default (SliceErrorOptions::alpha); the intervals are at level 1 − α.

A slice that isn't flagged hasn't been shown to be fine. A small slice can have a high error and still not be significantly worse; its interval shows how little is known.

RenderSliceErrorSvg(s) draws the report, worst first, with each interval and p-value; slices worse than the rest are red. SliceErrorViewOptions names the metric and adds a reference line:

Five slices' error rates with their intervals; the night slice is red and far above the overall error of 0.18

Uncertainty

A probability of 0.6 from one network doesn't say whether the network has seen inputs like this one. Several predictions of the same input that disagree do. pulsatrix/uncertainty.hpp gets them two ways:

  • Monte Carlo dropout (Gal and Ghahramani, ICML 2016): McDropoutSamples keeps the dropout layers on at inference and runs the network several times. Everything else stays in eval mode, so BatchNorm uses its running statistics, and each module's mode is restored afterwards. LoRA adapters with dropout count as dropout layers.
  • Deep ensembles (Lakshminarayanan, Pritzel and Blundell, NeurIPS 2017): train several networks from different seeds, and EnsembleSamples runs each one.
#include "pulsatrix/uncertainty.hpp"

auto predict = [&](const Tensor& x) { return softmax.forward(model.forward(x)); };
std::vector<Tensor> samples = McDropoutSamples(model, predict, inputs, 30);
UncertaintyDecomposition u = DecomposeUncertainty(samples);
// u.mean: the prediction to use; u.total, u.aleatoric, u.epistemic: per input, in nats

DecomposeUncertainty splits the entropy of the mean prediction into two parts (Depeweg et al., ICML 2018):

Part Formula What it means
total H[E p], the entropy of the mean how unsure the prediction is
aleatoric E H[p], the samples' mean entropy noise in the data: each sample is unsure too, and more data wouldn't help
epistemic total − aleatoric, the mutual information (BALD) the samples disagree: the model hasn't learned this region, and more data would help

Samples that agree on 50/50 give all aleatoric uncertainty; confident samples that disagree give the same total, all epistemic. agreement is the fraction of samples that pick the mean's class.

For regression, each member predicts a mean and a variance, trained with GaussianNLLLoss: the network outputs (μ, s) and its variance is softplus(s) + 10⁻⁶, as Lakshminarayanan et al. did. CombineGaussians mixes the members: the aleatoric part is their mean variance, and the epistemic part is the variance of their means.

Note

The decomposition is only as good as the samples. Dropout samples come from one network and tend to agree more than an ensemble's do. Neither is the true posterior, and the epistemic part says how much these samples disagree, nothing more.

Conformal prediction

A probability of 0.9 doesn't mean the model is right 90% of the time; calibration says how far off it is, but not what to do about it. Split conformal prediction (Vovk, Gammerman and Shafer; Angelopoulos and Bates, arXiv 2107.07511) turns any model's scores into a guarantee:

  1. Score each example of a held-out calibration set, which the model wasn't trained on, by how badly the model fits it.
  2. Take q̂, the ⌈(n + 1)(1 − α)⌉-th smallest score.
  3. A new input's prediction set holds every answer that would score at most q̂.

If the calibration examples and the new one are exchangeable, the set holds the true answer with probability at least 1 − α:

#include "pulsatrix/conformal.hpp"

ConformalOptions options;
options.alpha = 0.1;                       // 90% coverage
options.score = ConformalScore::Aps;
ConformalClassifier c = CalibrateConformalClassifier(calibration_probabilities, calibration_labels, options);
std::vector<std::vector<int64_t>> sets = PredictionSets(c, test_probabilities);
CoverageReport r = EvaluateCoverage(sets, test_labels);
Score Definition Sets
Lac 1 − p(true class) (Sadinle, Lei and Wasserman, 2019) the smallest on average, but they can be empty, and hard inputs are undercovered
Aps the mass of the classes ranked at or above the true one (Romano, Sesia and Candès, 2020) larger for harder inputs; randomized brings coverage close to exactly 1 − α

For regression, CalibrateConformalRegressor scores the absolute residual and gives ŷ ± q̂. CalibrateConformalQuantileRegressor takes a quantile regressor's lower and upper predictions and gives CQR's [lo − q̂, hi + q̂] (Romano, Patterson and Candès, 2019), narrower where the noise is.

EvaluateCoverage reports the coverage, the mean set size or interval width, how many sets were empty, and the coverage of each set size. Large sets that are undercovered are a sign the score doesn't adapt to hard inputs.

The guarantee is marginal: it holds on average over inputs, not for each input or each group. A model can cover 90% overall and 70% of one group. CoverageReport::misses holds a 0 or 1 per input, so PerSliceError(r.misses, slices) tests each group's coverage like any other error. Draw it with the target miss rate as the reference:

SliceErrorViewOptions view;
view.metric = "miss rate";
view.reference = 0.1;          // alpha
view.reference_label = "target";
std::ofstream("coverage.svg") << RenderSliceErrorSvg(PerSliceError(r.misses, slices), view);

Note

With too few calibration examples, the rank ⌈(n + 1)(1 − α)⌉ is above n, q̂ is infinite and every set holds every class: at α = 0.1 that takes fewer than 9 examples.

LAC sets and absolute-residual intervals match MAPIE 1.1.0's, and APS sets Angelopoulos and Bates' reference code (tools/generate_conformal_reference_values.py).

Adversarial robustness

How small a change to the input makes the model change its answer? pulsatrix/adversarial.hpp searches for one by following the gradient of the loss with respect to the input:

  • FGSM (Goodfellow, Shlens and Szegedy, ICLR 2015) takes one step of size ε: the sign of the gradient under the L∞ norm, its direction under L2.
  • PGD (Madry et al., ICLR 2018) starts at a random point within ε of the input, takes steps smaller steps, and after each one projects back within ε and into the input's valid range.
  • Targeted attacks push the input toward a chosen class instead of away from its own.
#include "pulsatrix/adversarial.hpp"

AttackOptions options;
options.epsilon = 8.0 / 255;          // L∞ budget, for pixels in [0, 1]
options.clip_min = 0;
options.clip_max = 1;
AttackResult r = Pgd(ClassifierFrom(model), images, labels, &backend, options);
// r.adversarial, r.success, r.robust_accuracy, r.perturbation

RobustnessCurve curve = RobustAccuracyCurve(ClassifierFrom(model), images, labels, {1.0 / 255, 2.0 / 255, 4.0 / 255, 8.0 / 255},
                                            &backend, options);

ClassifierFrom takes a Module or an ExplainerContext. For any other model, fill a DifferentiableClassifier with its forward pass to logits and its backward pass to the input.

RobustAccuracyCurve gives the accuracy that survives untargeted PGD at each ε. It also gives the fraction of correctly classified inputs each attack flipped, and the mean size of the successful perturbations. The results are pulsatrix.attack.v1 and pulsatrix.robustness_curve.v1 documents.

The steps, projection, clipping and random start follow torchattacks' FGSM, PGD and PGDL2. tools/generate_adversarial_reference_values.py reimplements them in numpy, and the tests match it. On a linear model, where the best attack is known in closed form, PGD finds it.

Note

PGD finds adversarial examples; it doesn't prove there are none. A model that resists it may fall to a stronger attack, so the robust accuracy it reports is an upper bound. Gradient masking, where a model's gradients are uninformative rather than its decisions robust, makes gradient attacks look weaker than they are (Athalye, Carlini and Wagner, ICML 2018).

Explanation robustness

An explanation that changes a lot when the input barely does says more about the explainer than the model. Ghorbani, Abid and Zou (AAAI 2019) changed saliency maps beyond recognition with imperceptible perturbations. ExplanationSensitivity (in pulsatrix/explanation_robustness.hpp) measures it the way Quantus does:

  • max- and average-sensitivity (Yeh et al., NeurIPS 2019): ‖E(x′) − E(x)‖₂ / ‖E(x)‖₂ over samples x′ drawn uniformly within radius times the input's spread, the largest and the mean;
  • the local Lipschitz estimate (Alvarez-Melis and Jaakkola, 2018): the largest ‖E(x′) − E(x)‖₂ / ‖x′ − x‖₂;
  • adversarial sensitivity: the same ratio at FGSM's point, the perturbation an attacker would try first. The step is halved until the prediction stays.

Only samples that keep the predicted class count: an explanation should change when the prediction does.

#include "pulsatrix/explanation_robustness.hpp"

DifferentiableClassifier classifier = ClassifierFrom(model);
SensitivityReport s = ExplanationSensitivity(x, explain, predict, &classifier, &backend);
// s.max_sensitivity, s.avg_sensitivity, s.local_lipschitz, s.adversarial_sensitivity, s.kept

tools/generate_explanation_robustness_reference_values.py computes the same metrics in numpy, on the same samples, and the tests match it. Under the Control reason, the planner requires it as the robustness check, and BuildExplanation runs it (Reporting an explanation).

In a report

The Control reason requires per-slice error. The planner marks it available when the caller has the dataset's labels and slices (ModelTraits::has_labels, or pulsatrix_plan --labels). BuildExplanation then runs it on request.dataset, request.labels and request.slices. The check fails, and blocks the report, when a slice is worse than the rest:

request.reason = XaiReason::Control;
request.traits.has_dataset = true;
request.traits.has_labels = true;
request.dataset = inputs;      // each (1, ...)
request.labels = labels;
request.slices = slices;
Explanation e = BuildExplanation(request, x, explain, predict, &model);

For a regressor, or any model whose error isn't the wrong class, run PerSliceError on its losses and pass SliceErrorCheck(report) to ReportExplanation.

The Performance question itself plans ClassifierMetrics, RegressorMetrics, PerSliceError, MonteCarloDropout, DeepEnsemble, SplitConformal and AdversarialAttack. Those that score against true labels (the metrics, conformal calibration and the attacks) need ModelTraits::has_labels. They describe how the model behaves, not what it learned, so the planner doesn't ask for a null-model baseline for them.

Each report has a JSON document: pulsatrix.classification_report.v1, pulsatrix.regression_report.v1, pulsatrix.slice_error.v1, pulsatrix.uncertainty.v1 and pulsatrix.coverage.v1.