|
pulsatrix 1.1.0
|
Testing with Concept Activation Vectors (INT-6): how much a concept, given only by examples, pushes a model towards a class, and whether that's more than chance. More...
#include <cstdint>#include <vector>#include "pulsatrix/explainer_context.hpp"#include "pulsatrix/tensor.hpp"
Go to the source code of this file.
Classes | |
| struct | pulsatrix::CavOptions |
| struct | pulsatrix::Cav |
| A concept activation vector. More... | |
| struct | pulsatrix::TTestResult |
| A two-sample t-test's result. More... | |
| struct | pulsatrix::TcavOptions |
| struct | pulsatrix::TcavResult |
| RunTcav()'s result. More... | |
Namespaces | |
| namespace | pulsatrix |
Enumerations | |
| enum class | pulsatrix::CavMethod { pulsatrix::LogisticRegression , pulsatrix::DifferenceOfMeans } |
| How TrainCav() finds the concept direction. More... | |
Functions | |
| Cav | pulsatrix::TrainCav (const Tensor &concept_activations, const Tensor &random_activations, const CavOptions &options={}) |
| Trains a CAV from a layer's activations on concept examples and on random examples. | |
| std::vector< double > | pulsatrix::ConceptualSensitivities (const Tensor &gradients, const Cav &cav) |
Each input's directional derivative along the CAV: the row of gradients dotted with cav.direction. | |
| double | pulsatrix::TcavScore (const std::vector< double > &sensitivities) |
| The fraction of sensitivities above zero. | |
| TTestResult | pulsatrix::StudentTTest (const std::vector< double > &a, const std::vector< double > &b) |
| Student's two-sample t-test with pooled variance, two-sided, as scipy.stats.ttest_ind computes by default and the TCAV library uses. | |
| Tensor | pulsatrix::LayerActivations (ExplainerContext &ctx, const Tensor &inputs, NodeId layer) |
The layer's activations for inputs, from one forward pass: shape (n, width). | |
| Tensor | pulsatrix::LayerGradients (ExplainerContext &ctx, const Tensor &inputs, NodeId layer, int64_t target, DeviceBackend *backend) |
The gradient of logit target with respect to the layer, for each of inputs: shape (n, width). | |
| TcavResult | pulsatrix::RunTcav (ExplainerContext &ctx, NodeId layer, int64_t target, const Tensor &class_inputs, const Tensor &concept_inputs, const std::vector< Tensor > &random_inputs, DeviceBackend *backend, const TcavOptions &options={}) |
| TCAV with Kim et al.'s significance test, for one concept, layer and class. | |
Testing with Concept Activation Vectors (INT-6): how much a concept, given only by examples, pushes a model towards a class, and whether that's more than chance.
TCAV (Kim et al., ICML 2018) works in three steps:
grad_layer(logit) . cav. Positive means moving the activation towards the concept raises the class.One score means little. A CAV trained against one random set can point anywhere that separates the two sets, and a meaningless direction scores anywhere between 0 and 1. So RunTcav() trains CAVs against several random sets, and does the same with random sets in place of the concept. A two-sided t-test between the two lists of scores says whether the concept differs from chance.
The models are ExplainerContext chains, whose gradients are available at every module's output. Language models aren't covered: they don't expose gradients at an intermediate layer.