pulsatrix 1.1.0
Loading...
Searching...
No Matches
tcav.hpp File Reference

Testing with Concept Activation Vectors (INT-6): how much a concept, given only by examples, pushes a model towards a class, and whether that's more than chance. More...

#include <cstdint>
#include <vector>
#include "pulsatrix/explainer_context.hpp"
#include "pulsatrix/tensor.hpp"
Include dependency graph for tcav.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::CavOptions
 
struct  pulsatrix::Cav
 A concept activation vector. More...
 
struct  pulsatrix::TTestResult
 A two-sample t-test's result. More...
 
struct  pulsatrix::TcavOptions
 
struct  pulsatrix::TcavResult
 RunTcav()'s result. More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::CavMethod { pulsatrix::LogisticRegression , pulsatrix::DifferenceOfMeans }
 How TrainCav() finds the concept direction. More...
 

Functions

Cav pulsatrix::TrainCav (const Tensor &concept_activations, const Tensor &random_activations, const CavOptions &options={})
 Trains a CAV from a layer's activations on concept examples and on random examples.
 
std::vector< double > pulsatrix::ConceptualSensitivities (const Tensor &gradients, const Cav &cav)
 Each input's directional derivative along the CAV: the row of gradients dotted with cav.direction.
 
double pulsatrix::TcavScore (const std::vector< double > &sensitivities)
 The fraction of sensitivities above zero.
 
TTestResult pulsatrix::StudentTTest (const std::vector< double > &a, const std::vector< double > &b)
 Student's two-sample t-test with pooled variance, two-sided, as scipy.stats.ttest_ind computes by default and the TCAV library uses.
 
Tensor pulsatrix::LayerActivations (ExplainerContext &ctx, const Tensor &inputs, NodeId layer)
 The layer's activations for inputs, from one forward pass: shape (n, width).
 
Tensor pulsatrix::LayerGradients (ExplainerContext &ctx, const Tensor &inputs, NodeId layer, int64_t target, DeviceBackend *backend)
 The gradient of logit target with respect to the layer, for each of inputs: shape (n, width).
 
TcavResult pulsatrix::RunTcav (ExplainerContext &ctx, NodeId layer, int64_t target, const Tensor &class_inputs, const Tensor &concept_inputs, const std::vector< Tensor > &random_inputs, DeviceBackend *backend, const TcavOptions &options={})
 TCAV with Kim et al.'s significance test, for one concept, layer and class.
 

Detailed Description

Testing with Concept Activation Vectors (INT-6): how much a concept, given only by examples, pushes a model towards a class, and whether that's more than chance.

TCAV (Kim et al., ICML 2018) works in three steps:

  1. A concept activation vector (CAV). A linear classifier separates a layer's activations on examples of a concept (striped images, say) from its activations on random examples. Its normal, pointing towards the concept, is the CAV.
  2. Conceptual sensitivity. For an input, the directional derivative of a class logit along the CAV, at that layer: grad_layer(logit) . cav. Positive means moving the activation towards the concept raises the class.
  3. The TCAV score. The fraction of a class's inputs whose sensitivity is positive.

One score means little. A CAV trained against one random set can point anywhere that separates the two sets, and a meaningless direction scores anywhere between 0 and 1. So RunTcav() trains CAVs against several random sets, and does the same with random sets in place of the concept. A two-sided t-test between the two lists of scores says whether the concept differs from chance.

The models are ExplainerContext chains, whose gradients are available at every module's output. Language models aren't covered: they don't expose gradients at an intermediate layer.