Gradient-Based Explainers¶
These explainers look inside a pulsatrix network to show which input features drove one
prediction. They read its gradients and cached activations. Use them when the model is built
from pulsatrix Modules. For a black-box model, use the
model-agnostic explainers instead.
All of them run through ExplainerContext (explainer_context.hpp). It wraps your network's
modules, given as a std::vector<Module*> in forward order. It runs traced forward, backward
and relevance passes and exposes per-node activations and gradients, so each explainer doesn't
need its own graph-walking code.
What's inside¶
Saliency(saliency.hpp): the raw gradient of a target output with respect to the input.IntegratedGradients(integrated_gradients.hpp): gradients averaged along a straight path from a baseline input to your input. This avoids saliency's blind spot where the output is flat (saturated).GradCAM(grad_cam.hpp): gradient-weighted class activation mapping. It produces a coarse heatmap over the lastConv2DModule's feature maps.FindCounterfactual(counterfactual.hpp): the nearest input that reaches a different class or an output range (Wachter et al. 2017), with immutable features, limits, and integer and one-hot features. It answers "what would have to change?"FindDiverseCounterfactualsfinds several that take different routes (DiCE), andPlausibilityscores how typical a counterfactual is of real data.FindPertinentNegative,FindPertinentPositive(contrastive.hpp): the contrastive explanation method (CEM; Dhurandhar et al. 2018). A pertinent negative is what is absent from the input and would change its class if present. A pertinent positive is the least of the input that keeps the class on its own.ComputeAttributionStability(runs)(explainer_stability.hpp): measures how much repeated runs of any explainer vary on the same input.
LRP (Layer-wise Relevance Propagation) also runs through ExplainerContext, but
propagates relevance with per-layer rules instead of gradients. It has its own page.
Every explainer returns an Attribution and requires a rank-2 (N, num_classes) network output.
Full API reference: Doxygen: Gradient-Based Explainers
When to use which¶
| Method | Cost | Good for | Watch out for |
|---|---|---|---|
Saliency |
1 forward + 1 backward | A quick first look | Noisy; reads zero where the output saturates |
IntegratedGradients |
steps forward + backward passes |
Attributions that add up to f(input) - f(baseline) |
You must choose a baseline |
GradCAM |
1 forward + 1 backward | "Where in the image?" for CNNs | Coarse: feature-map resolution, not pixels |
FindDiverseCounterfactuals |
count × max_steps forward + backward passes |
Several different ways to change the outcome | The diversity and proximity weights trade off against each other |
FindCounterfactual |
Up to rounds × steps forward + backward passes |
"What is the smallest change that flips this?" | On images it finds adversarial noise, not a meaningful change |
FindPertinentNegative, FindPertinentPositive |
search_steps × steps × (2 forward + 1 backward) |
"What is missing that would change it?" and "what is enough to keep it?" | No autoencoder term: on images the result can look like noise |
LRP |
1 forward + 1 relevance pass | Per-feature relevance that (approximately) sums to the output score | The rule choice changes the result |
How to implement¶
The snippets below share this setup: a small two-layer classifier.
#include "pulsatrix/cpu_backend.hpp"
#include "pulsatrix/explainer_context.hpp"
#include "pulsatrix/linear_module.hpp"
#include "pulsatrix/relu_module.hpp"
using namespace pulsatrix;
CPUBackend backend;
LinearModule linear1(2, 4, &backend);
ReluModule relu(&backend);
LinearModule linear2(4, 2, &backend);
// ... train or load weights ...
ExplainerContext ctx({&linear1, &relu, &linear2}); // the layers, in forward order
Tensor input(Shape({1, 2}), &backend, {1.0f, 0.0f}); // a batch of one example
Saliency¶
#include "pulsatrix/saliency.hpp"
Saliency saliency;
Attribution result = saliency.explain(ctx, input, /*target_index=*/0, &backend);
// result.values: d(output[target_index]) / d(input), same shape as input
What's happening: explain() runs ctx.forward_pass(input). It then seeds a one-hot
vector at target_index for every row in the batch and calls ctx.backward_pass(seed). The
gradient that comes back is the saliency map.
Recipe: Saliency and Integrated Gradients.
Integrated Gradients¶
#include "pulsatrix/integrated_gradients.hpp"
Tensor baseline(Shape({1, 2}), &backend, {0.0f, 0.0f}); // an "uninformative" reference input
IntegratedGradients ig;
Attribution result = ig.explain(ctx, input, baseline, /*target_index=*/0, /*steps=*/200, &backend);
// result.values: the path-integrated attribution, same shape as input
What's happening: explain() runs Saliency at steps evenly spaced points between
baseline and input. It averages those gradients and multiplies by (input - baseline).
The attributions then sum to f(input) - f(baseline), up to an approximation error that
shrinks as steps grows (50–300 is typical). Because it collects gradients along the whole
path, it doesn't go blind where the output is flat at the input itself.
Recipe: Saliency and Integrated Gradients.
Grad-CAM¶
Grad-CAM needs a network with at least one Conv2DModule. This example uses the built-in
MNIST classifier, MnistConvNet (Conv2D → ReLU → Flatten → Linear).
#include "pulsatrix/cpu_backend.hpp"
#include "pulsatrix/explainer_context.hpp"
#include "pulsatrix/grad_cam.hpp"
#include "pulsatrix/mnist_classifier_example.hpp"
using namespace pulsatrix;
CPUBackend backend;
MnistConvNet net(&backend);
// ... train the network ...
ExplainerContext ctx(net.modules());
std::vector<float> pixels(28 * 28, 0.0f); // one 28x28 image, values in [0, 1]
Tensor image(Shape({1, 1, 28, 28}), &backend, pixels);
GradCAM grad_cam;
Attribution result = grad_cam.explain(ctx, image, /*target_index=*/3, &backend);
// result.values: (N, H, W) class activation map, at the conv layer's feature-map size
What's happening: explain() runs forward and backward like Saliency. It then finds the
last OpType::Conv node in the graph and reads that node's activation A and gradient. It
averages the gradient over each channel's spatial positions to get a weight αₖ per channel.
The map is ReLU(Σₖ αₖ · Aᵏ).
Recipe: Grad-CAM walkthrough.
Counterfactuals¶
#include "pulsatrix/counterfactual.hpp"
#include "pulsatrix/viz/svg.hpp"
CounterfactualConstraints c;
c.scale = MedianAbsoluteDeviation(background); // a change of one MAD costs 1
c.immutable = {0}; // feature 0 (say, age) may not change
CounterfactualResult cf = FindCounterfactual(ctx, input, CounterfactualTarget::ToClass(1), c);
// cf.valid, cf.counterfactual, cf.distance, cf.num_changed, cf.output_before, cf.output_after
std::string svg = RenderCounterfactualSvg(
ToCounterfactualDocument(input, cf, {"age", "income"}, c.scale, "class approved"));
What's happening: the search minimizes lambda × loss + Σⱼ |x'ⱼ − xⱼ| / scaleⱼ. The loss is
a hinge that is zero once the target class leads every other output by margin (or the output
is inside the target range). Each step is a gradient step on the loss followed by
soft-thresholding toward the input, so features the prediction doesn't need stay exactly
unchanged and the result is sparse. If a round of steps_per_round steps ends without a valid
point, lambda grows by lambda_growth and the search continues. Integer features are rounded
and one-hot groups snapped at the end, and valid is judged after that. On a linear model the
result is the L1-optimal change, which the tests check.
Several routes. One counterfactual hides the others: raising income and paying down debt may both work.
DiverseCounterfactualOptions opts;
opts.count = 4;
DiverseCounterfactualResult set = FindDiverseCounterfactuals(ctx, input, CounterfactualTarget::ToClass(1), c, opts);
// set.validity, set.diversity, set.count_diversity; set.counterfactuals[i] as above
float typical = Plausibility(set.counterfactuals[0].counterfactual, background, c.scale, /*k=*/5);
std::vector<CounterfactualDocument> docs;
for (const auto& cf : set.counterfactuals) docs.push_back(ToCounterfactualDocument(input, cf, names, c.scale));
std::string svg = RenderCounterfactualSetSvg(docs);
The counterfactuals are optimized together: each one's target hinge and distance, minus a
diversity term, the log-determinant of the kernel 1 / (1 + distance) between them, which pushes
them apart. Afterwards each one's unneeded changes are undone, smallest first, as DiCE does.
diversity is their mean pairwise distance and count_diversity the mean fraction of features
that differ between pairs. Plausibility is the mean scaled distance to the k nearest
background instances: a valid counterfactual far from all real data may describe nobody.
Test reference: tests/counterfactual_test.cpp.
Pertinent negatives and positives (CEM)¶
#include "pulsatrix/contrastive.hpp"
ContrastiveResult pn = FindPertinentNegative(ctx, input); // what, if added, changes the class
ContrastiveResult pp = FindPertinentPositive(ctx, input); // the least that keeps the class
// pn.delta, pn.features, pn.explanation (input + delta), pn.explanation_class
What's happening: both search for a sparse change δ by FISTA on an elastic-net objective,
c · hinge + β|δ|₁ + |δ|², with a binary search over the weight c, as in the paper.
- Absent features. A feature is absent when it equals its baseline, 0 by default; set
options.baseline otherwise.
- Pertinent negative. δ may only raise absent features, up to options.upper, and the
hinge pushes the input's class below the strongest other class.
- Pertinent positive. δ keeps part of each feature between the baseline and the input, and
the hinge keeps the input's class on top.
Take the model z₀ = x₀ + x₁, z₁ = x₂ + 0.5x₃ and the input (1, 0, 0, 0), which is class 0.
- Adding x₁ would only confirm class 0, so the pertinent negative leaves it alone.
- It raises x₂ by 0.81 and x₃ by 0.38, the elastic net's exact optimum. A single feature would
also flip the class, but the |δ|² term spreads the change.
- Raise beta for sparser explanations.
Two differences from the paper: - No autoencoder term. The paper adds an autoencoder's reconstruction error to keep δ on the data manifold. Without it, a pertinent negative on images can be noise a person wouldn't recognize. - Absence is a fixed baseline. On standardized tabular data no value means "absent", so pick the baseline deliberately.
Before you explain¶
- Call
model.set_training(false)first if the model has BatchNorm or Dropout. Otherwise each sample's gradients, and so its explanation, depend on the rest of its batch. - To save or plot a result, see Visualization. A
Grad-CAM map goes through
ToSaliencyHeatmap(), thenToHeatmapDocument()for JSON orRenderHeatmapSvg()for an SVG figure.