Layer-wise Relevance Propagation (LRP)¶
LRP explains one prediction by starting at the output and passing a "relevance" score backward, layer by layer, until every input feature has a share. Unlike gradient-based explainers, each layer uses its own propagation rule, chosen to fit what that layer computes.
In pulsatrix, LRP is built into the layer contract: every Module must implement
propagate_relevance(). The LRP class runs those rules over a whole network, so any network
built from pulsatrix layers can be explained without extra setup.
Quick example¶
#include "pulsatrix/cpu_backend.hpp"
#include "pulsatrix/explainer_context.hpp"
#include "pulsatrix/linear_module.hpp"
#include "pulsatrix/lrp.hpp"
#include "pulsatrix/relu_module.hpp"
using namespace pulsatrix;
CPUBackend backend;
LinearModule fc1(2, 4, &backend);
ReluModule relu(&backend);
LinearModule fc2(4, 2, &backend);
// ... train or load weights ...
ExplainerContext ctx({&fc1, &relu, &fc2}); // the layers, in forward order
Tensor input(Shape({1, 2}), &backend, {1.0f, 0.5f});
// Which input features pushed class 1 up? Epsilon rule on every layer:
Attribution eps = LRP().explain(ctx, input, /*target_index=*/1, &backend);
// Or a Zennit-style composite that picks a rule per layer type:
Attribution plus = LRP::epsilon_plus().explain(ctx, input, 1, &backend);
// eps.values has the input's shape: one relevance score per input feature.
The network output must be rank 2, (batch, num_classes).
From Python:
import pulsatrix_py as px
ctx = px.ExplainerContext([fc1, relu, fc2])
x = px.Tensor.from_values([1, 2], [1.0, 0.5])
attr = px.LRP().explain(ctx, x, [1]) # epsilon everywhere
attr = px.LRP.epsilon_plus().explain(ctx, x, [1]) # composite
attr = px.LRP(rule=px.LRPRule.Gamma, gamma=0.25).explain(ctx, x, [1])
Rules¶
Pick a rule with LRPRuleConfig and pass it to LRP(config) to use it on every layer.
LRPRule |
What it does | Parameters |
|---|---|---|
Epsilon (default) |
Divides each contribution by the layer output plus a small stabilizer. | epsilon (default 1e-6) |
Gamma |
Boosts positive contributions before dividing, which reduces noise. | gamma (default 0.25) |
AlphaBeta |
Weights positive and negative contributions separately. alpha - beta must be 1. alpha = 1, beta = 0 is ZPlus. |
alpha, beta |
ZBox |
For the input layer when inputs are bounded, e.g. pixels in [0, 1]. |
low, high |
The semantics follow Zennit 1.0.0. Epsilon works on every layer. Gamma, AlphaBeta
and ZBox are defined only for the affine layers, LinearModule and Conv2DModule.
Pass-through layers (ReluModule, FlattenModule, DropoutModule) accept any rule and pass
relevance through unchanged. MaxPool2DModule sends each window's relevance to the input that
won it (winner-take-all, as Zennit does), and AvgPool2DModule and AdaptiveAvgPool2DModule
share it among each window's inputs in proportion to their values (the z-rule, Zennit's Norm
rule).
If you ask a layer for a rule it doesn't implement, explain() throws
std::invalid_argument before propagating anything. It never silently falls back to epsilon.
LRPRuleConfig config;
config.rule = LRPRule::AlphaBeta;
config.alpha = 2.0f;
config.beta = 1.0f;
Attribution a = LRP(config).explain(ctx, input, 1, &backend);
Bias handling¶
By default the epsilon rule leaves the bias out of the denominator. Under that default, linear
and convolution layers pass on all the relevance they receive. Zennit and LXT include the bias.
Set epsilon_bias_in_denominator = true to match them; the bias then absorbs part of the
relevance.
Composites: a different rule per layer¶
A composite chooses a rule for each layer. Three of Zennit's presets are built in:
| Preset | LinearModule |
Conv2DModule |
|---|---|---|
LRP::epsilon_plus() |
Epsilon | ZPlus |
LRP::epsilon_alpha2_beta1() |
Epsilon | AlphaBeta(2, 1) |
LRP::epsilon_gamma_box(low, high) |
Epsilon | ZBox on the first Conv2D, Gamma on the rest |
Every other layer gets the epsilon rule. The presets set epsilon_bias_in_denominator = true
so they match Zennit. Like Zennit, they choose by layer type throughout the network: a
convolution inside a residual block or a SequentialModule gets the convolution rule too
(through LRPRuleConfig::conv_rule). ZBox is the exception: it goes to the first top-level
Conv2D, which in ResNet and VGG is the first convolution. As in Zennit, epsilon_gamma_box never applies ZBox to a LinearModule,
so on a network without convolutions it is epsilon everywhere.
To write your own, pass any function (size_t layer_index, const Module&) -> LRPRuleConfig:
LRP custom([](size_t i, const Module&) {
LRPRuleConfig c;
if (i == 0) { c.rule = LRPRule::ZBox; c.low = 0.0f; c.high = 1.0f; }
return c;
}, "zbox_then_epsilon");
Targets, contrasts and seeds¶
explain(ctx, input, target_index, backend) explains one class. For more control, pass an
LRPTarget:
// Why class 3 rather than class 8?
LRPTarget target{{3}, {8}};
Attribution a = LRP().explain(ctx, input, target, &backend);
targets: one class index for the whole batch, or one per row.contrasts(optional): the explanation becomesexplain(target) - explain(contrast), which answers "why this class rather than that one?"seed:LRPSeed::OutputValue(default) starts from the logit itself, so the relevance sums to the output.LRPSeed::OneHotstarts from 1, which is the convention Zennit and LXT use. UseOneHotwhen comparing against them.
The returned Attribution has method == "lrp", the relevance in values (same shape as
the input), and metadata with the rule(s) used, the seed, and relevance_out_sum /
relevance_in_sum. Compare those two sums to see how much relevance was conserved.
Which layers have which rule¶
Every layer implements propagate_relevance(). The rules beyond plain epsilon come from
the literature:
| Layer | Rule |
|---|---|
LinearModule, Conv2DModule |
Epsilon, Gamma, AlphaBeta, ZBox |
RNNModule, LSTMModule, GRUModule |
Arras et al. 2019 (relevance follows the signal path; gates pass none) |
SoftmaxModule, MultiHeadAttentionModule, TransformerBlock |
AttnLRP |
MambaModule |
MambaLRP |
RetNetModule |
AttnLRP-style rule over the retention scores |
RWKVModule |
MambaLRP-style rule adapted to the WKV quotient |
Normalization layers pass relevance through unchanged by default. With
LRPRuleConfig::layer_norm_detach_std (set by LxtAttnLrpConfig()), LayerNormModule uses
AttnLRP's rule as LXT applies it instead: the standard deviation is held constant, and the rest,
γ (x - mean) / std + β, takes the epsilon rule. For RMSNorm, which has no mean or bias, the two
are the same. For a BatchNormModule after a
Conv2DModule, fold the BatchNorm into the convolution while explaining, as Zennit's
canonizer does. The convolution's rule then covers the combined affine map:
#include "pulsatrix/batch_norm_fold.hpp"
bn.set_training(false); // BatchNorm must use its running statistics
{
BatchNormFold fold(conv, bn); // conv's weights absorb bn; bn becomes the identity
Attribution r = LRP().explain(ctx, input, /*target_index=*/1, &backend);
} // both modules are restored exactly here
Rules that conserve relevance by construction have conservation tests. The AttnLRP rules for softmax and attention don't conserve exactly; their tests measure and report the gap instead.
The VAE, GAN and diffusion building blocks (Reparameterize, KLDivergenceLoss,
BCEWithLogitsLoss, NoiseSchedule, SinusoidalTimestepEmbedding) are losses and sampling
steps, not Modules, so they have no LRP rule. Explaining a diffusion model means following
relevance through the whole sampler instead. LRP for diffusion models sets
out the rules; the ones for continuous samplers (ExplainTrajectory) and masked diffusion
language models (ExplainCommitments) are built, and the one for adaLN is still a design.
Validation against Zennit and LXT¶
tests/lrp_reference_test.cpp checks LRP::explain() (one-hot seed) against values computed
by the reference libraries. The values were generated offline by
tools/generate_lrp_reference_values.py in the environment pinned by
tools/lrp_reference.Dockerfile (torch 2.14.1+cpu, zennit 1.0.0, lxt 2.1).
Zennit, using its Gradient attributor on an MLP and a two-conv CNN with nonzero biases and
mixed-sign inputs:
- uniform Epsilon (ε = 1e-6 and 0.25), ZPlus, AlphaBeta(2, 1) and Gamma(0.25)
- the
EpsilonPlus,EpsilonAlpha2Beta1andEpsilonGammaBoxpresets
LXT, using its explicit AttnLRP rules on MultiHeadAttentionModule and TransformerBlock
(RoPE and QK-Norm off), each followed by Flatten → Linear.
Zennit on ResNet and VGG: tests/vision_models_test.cpp (and its HIP and CUDA
counterparts) compares a small torchvision ResNet (BatchNorm with non-trivial statistics, a
downsampling shortcut) and VGG with Zennit's three presets. The ResNet uses Zennit's
ResNetCanonizer; pulsatrix folds the BatchNorms and splits relevance at each residual sum in
proportion to the two branches, which is the same thing. The values are written by
tools/golden/make_vision_golden.py. With PULSATRIX_TORCHVISION_DIR set, the same tests run on
the published ResNet18 and VGG16; their heatmaps agree to 2e-3 of the largest value.
The tolerance is 1e-4 · max(1, |ref|) (float32 vs float32); the largest observed error is
under 1e-5. Matching needs epsilon_bias_in_denominator = true (see Bias handling).
LXT on whole language models: CompareToAttnLrp (attnlrp_parity.hpp) and
pulsatrix_attnlrp compare a CausalLM loaded from Hugging Face with LXT's own AttnLRP
(lxt.efficient, the transformers patch) on the same model and tokens. The references are written
by tools/golden/make_attnlrp_reference.py in the same Docker image. Each text is explained from
its argmax logit at the last position, with relevance = logit at the output (LxtAttnLrpConfig():
ε = 1e-9, biases in the denominator). It compares:
- the relevance of every input token
- each layer's relevance at the K and V projection outputs, per KV head, after the grouped-query
sum (
MultiHeadAttentionModule::key_relevance()/value_relevance())
| Model | Largest relative difference (CPU) | Correlation |
|---|---|---|
| SmolLM2-135M | 4.1e-6 | 1.000000 |
| Qwen2.5-0.5B | 8.0e-6 | 1.000000 |
| Qwen3-0.6B | 1.2e-5 | 1.000000 |
| Llama-3.2-1B | 2.3e-5 | 1.000000 |
CI runs the tiny Llama, Qwen2 and Qwen3 models, plus a negative control: plain gradient × input
(LXT's patches off) must fail, and it does, with correlations below 0. Real models run locally
through PULSATRIX_GOLDEN_DIR.
LXT 2.1 needs two shims under transformers 5, both in the generator:
- Its BERT patch imports a removed helper, so that name is stubbed.
- Its attention patch replaces ALL_ATTENTION_FUNCTIONS with a plain dict, so only
eager_attention_forward is wrapped, with LXT's own wrapper.
Known differences and gaps:
- LXT stabilizes with
z + εfor every sign ofz; pulsatrix and Zennit usez + ε·sign(z). They differ only where|z|is on the order of ε. - LayerNorm is covered through ESM-2 (Protein language models): 4.8e-5 against LXT's rules on the 650M model.
- Not covered by the reference tests: the recurrent and state-space rules.
Concepts: Concept Relevance Propagation¶
A heatmap says where the evidence for a class is, not what it is. Concept Relevance
Propagation (CRP; Achtibat et al., Nature Machine Intelligence 2023) works at a hidden layer,
where each channel (a convolution's filter, a dense layer's unit) stands for a learned concept.
It answers three questions, with the same LRP object and rules as explain():
#include "pulsatrix/concept_relevance.hpp"
const LRP lrp = LRP::epsilon_plus();
// Node i + 1 is module i's output: 2 is the first ReLU's.
const Tensor at_layer = LayerRelevance(lrp, ctx, image, LRPTarget{{cat}}, /*layer=*/2, &backend);
const std::vector<double> per_channel = ChannelRelevance(at_layer); // (N, C)
// Where in the image are concepts 7 and 12?
const std::vector<Attribution> maps =
ConditionalHeatmaps(lrp, ctx, image, LRPTarget{{cat}}, {{{2, {7}}}, {{2, {12}}}}, &backend);
// What is concept 7? The images where it does most for "cat".
const std::vector<int64_t> examples =
ConceptReferenceSamples(lrp, ctx, dataset, cat, 2, 7, /*k=*/8, ReferenceMode::Relevance, &backend);
- How much each concept matters.
ChannelRelevancesums the relevance at the layer per channel. With a conserving rule, the channels add up to the explained output. - Where each concept is.
ConditionalHeatmapsruns LRP with the relevance at the layer zeroed outside a concept's channels. The rules are linear in the relevance they pass on, so the heatmaps of all a layer's channels add up to the plain heatmap (tested under three presets, to 1e-5). - A condition can mask several layers, for a concept and a sub-concept below it.
- The forward pass and the seed are computed once for all conditions.
- What each concept is.
ConceptReferenceSamplesranks a set of inputs by how relevant a channel is in each for the class (relevance maximization), or by its largest activation (activation maximization). - Activation shows everything that excites the channel.
- Relevance shows when the channel actually does something for the class. In the test, the input with the highest activation contributes nothing, because a later unit blocks it, so relevance maximization passes it over.
The pieces underneath are public too:
LRP::seed_relevanceandLRP::rule_configsare the two halves ofexplain().ExplainerContext::relevance_pass_through(relevance, configs, begin, end)propagates through a range of modules. A pass can stop at any layer, change the relevance there and continue.
CRP works on ExplainerContext chains. A SequentialModule counts as one layer, so to condition
inside it, list its modules in the context directly.
See also¶
- Recipe: LRP on a trained MNIST classifier
- Recipe: tracing relevance through a Datalog derivation
- Customization: writing
propagate_relevance()for your own layer - Concepts given by examples (TCAV): the other way to explain with concepts
- Checking an explanation: deletion and insertion curves, the randomization test and the null-model baseline, for any LRP result
- Visualization: saving relevance as JSON and drawing it as SVG
- API reference: Layer-wise Relevance Propagation