Skip to content

Layer-wise Relevance Propagation (LRP)

LRP explains one prediction by starting at the output and passing a "relevance" score backward, layer by layer, until every input feature has a share. Unlike gradient-based explainers, each layer uses its own propagation rule, chosen to fit what that layer computes.

In pulsatrix, LRP is built into the layer contract: every Module must implement propagate_relevance(). The LRP class runs those rules over a whole network, so any network built from pulsatrix layers can be explained without extra setup.

Quick example

#include "pulsatrix/cpu_backend.hpp"
#include "pulsatrix/explainer_context.hpp"
#include "pulsatrix/linear_module.hpp"
#include "pulsatrix/lrp.hpp"
#include "pulsatrix/relu_module.hpp"

using namespace pulsatrix;

CPUBackend backend;
LinearModule fc1(2, 4, &backend);
ReluModule relu(&backend);
LinearModule fc2(4, 2, &backend);
// ... train or load weights ...

ExplainerContext ctx({&fc1, &relu, &fc2});  // the layers, in forward order
Tensor input(Shape({1, 2}), &backend, {1.0f, 0.5f});

// Which input features pushed class 1 up? Epsilon rule on every layer:
Attribution eps = LRP().explain(ctx, input, /*target_index=*/1, &backend);

// Or a Zennit-style composite that picks a rule per layer type:
Attribution plus = LRP::epsilon_plus().explain(ctx, input, 1, &backend);

// eps.values has the input's shape: one relevance score per input feature.

The network output must be rank 2, (batch, num_classes).

From Python:

import pulsatrix_py as px

ctx = px.ExplainerContext([fc1, relu, fc2])
x = px.Tensor.from_values([1, 2], [1.0, 0.5])
attr = px.LRP().explain(ctx, x, [1])                 # epsilon everywhere
attr = px.LRP.epsilon_plus().explain(ctx, x, [1])    # composite
attr = px.LRP(rule=px.LRPRule.Gamma, gamma=0.25).explain(ctx, x, [1])

Rules

Pick a rule with LRPRuleConfig and pass it to LRP(config) to use it on every layer.

LRPRule What it does Parameters
Epsilon (default) Divides each contribution by the layer output plus a small stabilizer. epsilon (default 1e-6)
Gamma Boosts positive contributions before dividing, which reduces noise. gamma (default 0.25)
AlphaBeta Weights positive and negative contributions separately. alpha - beta must be 1. alpha = 1, beta = 0 is ZPlus. alpha, beta
ZBox For the input layer when inputs are bounded, e.g. pixels in [0, 1]. low, high

The semantics follow Zennit 1.0.0. Epsilon works on every layer. Gamma, AlphaBeta and ZBox are defined only for the affine layers, LinearModule and Conv2DModule. Pass-through layers (ReluModule, FlattenModule, DropoutModule) accept any rule and pass relevance through unchanged. MaxPool2DModule sends each window's relevance to the input that won it (winner-take-all, as Zennit does), and AvgPool2DModule and AdaptiveAvgPool2DModule share it among each window's inputs in proportion to their values (the z-rule, Zennit's Norm rule).

If you ask a layer for a rule it doesn't implement, explain() throws std::invalid_argument before propagating anything. It never silently falls back to epsilon.

LRPRuleConfig config;
config.rule = LRPRule::AlphaBeta;
config.alpha = 2.0f;
config.beta = 1.0f;
Attribution a = LRP(config).explain(ctx, input, 1, &backend);

Bias handling

By default the epsilon rule leaves the bias out of the denominator. Under that default, linear and convolution layers pass on all the relevance they receive. Zennit and LXT include the bias. Set epsilon_bias_in_denominator = true to match them; the bias then absorbs part of the relevance.

Composites: a different rule per layer

A composite chooses a rule for each layer. Three of Zennit's presets are built in:

Preset LinearModule Conv2DModule
LRP::epsilon_plus() Epsilon ZPlus
LRP::epsilon_alpha2_beta1() Epsilon AlphaBeta(2, 1)
LRP::epsilon_gamma_box(low, high) Epsilon ZBox on the first Conv2D, Gamma on the rest

Every other layer gets the epsilon rule. The presets set epsilon_bias_in_denominator = true so they match Zennit. Like Zennit, they choose by layer type throughout the network: a convolution inside a residual block or a SequentialModule gets the convolution rule too (through LRPRuleConfig::conv_rule). ZBox is the exception: it goes to the first top-level Conv2D, which in ResNet and VGG is the first convolution. As in Zennit, epsilon_gamma_box never applies ZBox to a LinearModule, so on a network without convolutions it is epsilon everywhere.

To write your own, pass any function (size_t layer_index, const Module&) -> LRPRuleConfig:

LRP custom([](size_t i, const Module&) {
    LRPRuleConfig c;
    if (i == 0) { c.rule = LRPRule::ZBox; c.low = 0.0f; c.high = 1.0f; }
    return c;
}, "zbox_then_epsilon");

Targets, contrasts and seeds

explain(ctx, input, target_index, backend) explains one class. For more control, pass an LRPTarget:

// Why class 3 rather than class 8?
LRPTarget target{{3}, {8}};
Attribution a = LRP().explain(ctx, input, target, &backend);
  • targets: one class index for the whole batch, or one per row.
  • contrasts (optional): the explanation becomes explain(target) - explain(contrast), which answers "why this class rather than that one?"
  • seed: LRPSeed::OutputValue (default) starts from the logit itself, so the relevance sums to the output. LRPSeed::OneHot starts from 1, which is the convention Zennit and LXT use. Use OneHot when comparing against them.

The returned Attribution has method == "lrp", the relevance in values (same shape as the input), and metadata with the rule(s) used, the seed, and relevance_out_sum / relevance_in_sum. Compare those two sums to see how much relevance was conserved.

Which layers have which rule

Every layer implements propagate_relevance(). The rules beyond plain epsilon come from the literature:

Layer Rule
LinearModule, Conv2DModule Epsilon, Gamma, AlphaBeta, ZBox
RNNModule, LSTMModule, GRUModule Arras et al. 2019 (relevance follows the signal path; gates pass none)
SoftmaxModule, MultiHeadAttentionModule, TransformerBlock AttnLRP
MambaModule MambaLRP
RetNetModule AttnLRP-style rule over the retention scores
RWKVModule MambaLRP-style rule adapted to the WKV quotient

Normalization layers pass relevance through unchanged by default. With LRPRuleConfig::layer_norm_detach_std (set by LxtAttnLrpConfig()), LayerNormModule uses AttnLRP's rule as LXT applies it instead: the standard deviation is held constant, and the rest, γ (x - mean) / std + β, takes the epsilon rule. For RMSNorm, which has no mean or bias, the two are the same. For a BatchNormModule after a Conv2DModule, fold the BatchNorm into the convolution while explaining, as Zennit's canonizer does. The convolution's rule then covers the combined affine map:

#include "pulsatrix/batch_norm_fold.hpp"

bn.set_training(false);  // BatchNorm must use its running statistics
{
    BatchNormFold fold(conv, bn);  // conv's weights absorb bn; bn becomes the identity
    Attribution r = LRP().explain(ctx, input, /*target_index=*/1, &backend);
}  // both modules are restored exactly here

Rules that conserve relevance by construction have conservation tests. The AttnLRP rules for softmax and attention don't conserve exactly; their tests measure and report the gap instead.

The VAE, GAN and diffusion building blocks (Reparameterize, KLDivergenceLoss, BCEWithLogitsLoss, NoiseSchedule, SinusoidalTimestepEmbedding) are losses and sampling steps, not Modules, so they have no LRP rule. Explaining a diffusion model means following relevance through the whole sampler instead. LRP for diffusion models sets out the rules; the ones for continuous samplers (ExplainTrajectory) and masked diffusion language models (ExplainCommitments) are built, and the one for adaLN is still a design.

Validation against Zennit and LXT

tests/lrp_reference_test.cpp checks LRP::explain() (one-hot seed) against values computed by the reference libraries. The values were generated offline by tools/generate_lrp_reference_values.py in the environment pinned by tools/lrp_reference.Dockerfile (torch 2.14.1+cpu, zennit 1.0.0, lxt 2.1).

Zennit, using its Gradient attributor on an MLP and a two-conv CNN with nonzero biases and mixed-sign inputs:

  • uniform Epsilon (ε = 1e-6 and 0.25), ZPlus, AlphaBeta(2, 1) and Gamma(0.25)
  • the EpsilonPlus, EpsilonAlpha2Beta1 and EpsilonGammaBox presets

LXT, using its explicit AttnLRP rules on MultiHeadAttentionModule and TransformerBlock (RoPE and QK-Norm off), each followed by Flatten → Linear.

Zennit on ResNet and VGG: tests/vision_models_test.cpp (and its HIP and CUDA counterparts) compares a small torchvision ResNet (BatchNorm with non-trivial statistics, a downsampling shortcut) and VGG with Zennit's three presets. The ResNet uses Zennit's ResNetCanonizer; pulsatrix folds the BatchNorms and splits relevance at each residual sum in proportion to the two branches, which is the same thing. The values are written by tools/golden/make_vision_golden.py. With PULSATRIX_TORCHVISION_DIR set, the same tests run on the published ResNet18 and VGG16; their heatmaps agree to 2e-3 of the largest value.

The tolerance is 1e-4 · max(1, |ref|) (float32 vs float32); the largest observed error is under 1e-5. Matching needs epsilon_bias_in_denominator = true (see Bias handling).

LXT on whole language models: CompareToAttnLrp (attnlrp_parity.hpp) and pulsatrix_attnlrp compare a CausalLM loaded from Hugging Face with LXT's own AttnLRP (lxt.efficient, the transformers patch) on the same model and tokens. The references are written by tools/golden/make_attnlrp_reference.py in the same Docker image. Each text is explained from its argmax logit at the last position, with relevance = logit at the output (LxtAttnLrpConfig(): ε = 1e-9, biases in the denominator). It compares:

  • the relevance of every input token
  • each layer's relevance at the K and V projection outputs, per KV head, after the grouped-query sum (MultiHeadAttentionModule::key_relevance() / value_relevance())
Model Largest relative difference (CPU) Correlation
SmolLM2-135M 4.1e-6 1.000000
Qwen2.5-0.5B 8.0e-6 1.000000
Qwen3-0.6B 1.2e-5 1.000000
Llama-3.2-1B 2.3e-5 1.000000

CI runs the tiny Llama, Qwen2 and Qwen3 models, plus a negative control: plain gradient × input (LXT's patches off) must fail, and it does, with correlations below 0. Real models run locally through PULSATRIX_GOLDEN_DIR.

LXT 2.1 needs two shims under transformers 5, both in the generator: - Its BERT patch imports a removed helper, so that name is stubbed. - Its attention patch replaces ALL_ATTENTION_FUNCTIONS with a plain dict, so only eager_attention_forward is wrapped, with LXT's own wrapper.

Known differences and gaps:

  • LXT stabilizes with z + ε for every sign of z; pulsatrix and Zennit use z + ε·sign(z). They differ only where |z| is on the order of ε.
  • LayerNorm is covered through ESM-2 (Protein language models): 4.8e-5 against LXT's rules on the 650M model.
  • Not covered by the reference tests: the recurrent and state-space rules.

Concepts: Concept Relevance Propagation

A heatmap says where the evidence for a class is, not what it is. Concept Relevance Propagation (CRP; Achtibat et al., Nature Machine Intelligence 2023) works at a hidden layer, where each channel (a convolution's filter, a dense layer's unit) stands for a learned concept. It answers three questions, with the same LRP object and rules as explain():

#include "pulsatrix/concept_relevance.hpp"

const LRP lrp = LRP::epsilon_plus();
// Node i + 1 is module i's output: 2 is the first ReLU's.
const Tensor at_layer = LayerRelevance(lrp, ctx, image, LRPTarget{{cat}}, /*layer=*/2, &backend);
const std::vector<double> per_channel = ChannelRelevance(at_layer);  // (N, C)

// Where in the image are concepts 7 and 12?
const std::vector<Attribution> maps =
    ConditionalHeatmaps(lrp, ctx, image, LRPTarget{{cat}}, {{{2, {7}}}, {{2, {12}}}}, &backend);

// What is concept 7? The images where it does most for "cat".
const std::vector<int64_t> examples =
    ConceptReferenceSamples(lrp, ctx, dataset, cat, 2, 7, /*k=*/8, ReferenceMode::Relevance, &backend);
  • How much each concept matters. ChannelRelevance sums the relevance at the layer per channel. With a conserving rule, the channels add up to the explained output.
  • Where each concept is. ConditionalHeatmaps runs LRP with the relevance at the layer zeroed outside a concept's channels. The rules are linear in the relevance they pass on, so the heatmaps of all a layer's channels add up to the plain heatmap (tested under three presets, to 1e-5).
  • A condition can mask several layers, for a concept and a sub-concept below it.
  • The forward pass and the seed are computed once for all conditions.
  • What each concept is. ConceptReferenceSamples ranks a set of inputs by how relevant a channel is in each for the class (relevance maximization), or by its largest activation (activation maximization).
  • Activation shows everything that excites the channel.
  • Relevance shows when the channel actually does something for the class. In the test, the input with the highest activation contributes nothing, because a later unit blocks it, so relevance maximization passes it over.

The pieces underneath are public too:

  • LRP::seed_relevance and LRP::rule_configs are the two halves of explain().
  • ExplainerContext::relevance_pass_through(relevance, configs, begin, end) propagates through a range of modules. A pass can stop at any layer, change the relevance there and continue.

CRP works on ExplainerContext chains. A SequentialModule counts as one layer, so to condition inside it, list its modules in the context directly.

See also