pulsatrix 1.1.0
Loading...
Searching...
No Matches
concept_relevance.hpp File Reference

Concept Relevance Propagation (INT-6): which of a layer's channels an LRP explanation passes through, where each one looks in the input, and which inputs show it at work. More...

#include <cstdint>
#include <vector>
#include "pulsatrix/attribution.hpp"
#include "pulsatrix/explainer_context.hpp"
#include "pulsatrix/lrp.hpp"
#include "pulsatrix/tensor.hpp"
Include dependency graph for concept_relevance.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::ConceptMask
 Keep the relevance of these channels at this layer, and zero the rest. More...
 

Namespaces

namespace  pulsatrix
 

Typedefs

using pulsatrix::ConceptCondition = std::vector< ConceptMask >
 One conditional heatmap's masks, at most one per layer.
 

Enumerations

enum class  pulsatrix::ReferenceMode { pulsatrix::Relevance , pulsatrix::Activation }
 How ConceptReferenceSamples() ranks inputs. More...
 

Functions

Tensor pulsatrix::LayerRelevance (const LRP &lrp, ExplainerContext &ctx, const Tensor &input, const LRPTarget &target, NodeId layer, DeviceBackend *backend)
 The relevance at layer for explaining target: the layer's shape, (N, C, ...).
 
std::vector< double > pulsatrix::ChannelRelevance (const Tensor &layer_relevance)
 Each row's relevance per channel, summed over the remaining dimensions: (N, C) values, row-major. With a conserving rule they add up to the explained output.
 
std::vector< Attribution > pulsatrix::ConditionalHeatmaps (const LRP &lrp, ExplainerContext &ctx, const Tensor &input, const LRPTarget &target, const std::vector< ConceptCondition > &conditions, DeviceBackend *backend)
 One input heatmap per condition: LRP from target with the relevance zeroed, at each masked layer, outside the mask's channels.
 
std::vector< int64_t > pulsatrix::ConceptReferenceSamples (const LRP &lrp, ExplainerContext &ctx, const Tensor &candidates, int64_t target, NodeId layer, int64_t channel, int64_t k, ReferenceMode mode, DeviceBackend *backend)
 The k inputs of candidates that show a channel best, best first: the reference samples by which CRP shows what a concept is.
 

Detailed Description

Concept Relevance Propagation (INT-6): which of a layer's channels an LRP explanation passes through, where each one looks in the input, and which inputs show it at work.

An LRP heatmap says where in the input the evidence for a class is, but not what it is. CRP (Achtibat et al., Nature Machine Intelligence 2023) takes the relevance at a hidden layer, where each channel (a convolution's filter, a dense layer's unit) stands for a learned concept, and asks three questions:

  • How much does each concept matter for this prediction? LayerRelevance() and ChannelRelevance(): the relevance at the layer, summed per channel.
  • Where in the input is each concept? ConditionalHeatmaps(): LRP with the relevance at the layer zeroed outside the chosen channels, so only that concept's share reaches the input. Conditions can mask several layers at once, a concept and its sub-concepts below it.
  • What is the concept? ConceptReferenceSamples(): the inputs in which the channel is most relevant for a class (relevance maximization), or most active (activation maximization). Relevance shows the concept as the model uses it for the class; activation shows everything that excites the channel, used or not.

LRP's rules are linear in the relevance they pass down, so the conditional heatmaps of every channel of a layer add up to the plain LRP heatmap. Everything uses the same LRP object, so the rules are the ones LRP::explain() would apply.