pulsatrix 1.1.0
Loading...
Searching...
No Matches
Mechanistic Interpretability

Tools for inspecting and probing a model's internal computation directly – activation caching, linear probes and their controls, representation similarity (linear CKA, RSA, SVCCA), nearest neighbors and mutual-kNN alignment, representation geometry (anisotropy, effective rank, intrinsic dimension), PCA projections, export for UMAP and t-SNE, TCAV and CRP, sparse autoencoders, circuit graphs – plus the GFlowNet-based non-goal-directed training objectives motivated by the Scientist AI research direction. More...

Files

file  activation_snapshot.hpp
 Self-contained, enumerable copy of one forward pass's cached activations.
 
file  block_sparse_featurizer.hpp
 Block-sparse featurizers (FEAT-6; Fel et al., "Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds", arXiv 2606.25234): features are small subspaces (blocks of directions) instead of single directions, so a concept that lives on a low-dimensional manifold, such as an angle or a lighting direction, is one feature.
 
file  circuit_graph.hpp
 Self-contained circuit-graph artifact – scored nodes and weighted edges.
 
file  circuit_tracing.hpp
 Attribution graphs with cross-layer transcoders (FEAT-10), after Ameisen, Lindsey et al., "Circuit Tracing: Revealing Computational Graphs in Language Models" (Transformer Circuits, 2025) and its open implementation, circuit-tracer (github.com/decoderesearch/circuit-tracer). Each MLP is replaced by a transcoder's features plus an error term, attention patterns and normalization scales are frozen, and what is left is linear: the graph's edges are exact attributions between features, errors, token embeddings and output logits.
 
file  concept_relevance.hpp
 Concept Relevance Propagation (INT-6): which of a layer's channels an LRP explanation passes through, where each one looks in the input, and which inputs show it at work.
 
file  crosscoder.hpp
 Crosscoders (FEAT-8): one sparse dictionary read from and written to several sources at once, such as the residual stream at several layers, or the same layer in a base model and its fine-tune (Lindsey et al., "Sparse Crosscoders for Cross-Layer Features and Model Diffing", Transformer Circuits, 2024). Model diffing reads which latents' decoders live in one model only, with Minder et al.'s BatchTopK crosscoder and Latent Scaling ("Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning", arXiv 2504.02922), and Kassem et al.'s Delta-Crosscoder for narrow fine-tunes ("Delta- Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimes", arXiv 2603.04426).
 
file  detailed_balance_loss.hpp
 GFlowNet Detailed Balance loss (Bengio et al., "GFlowNet Foundations", arXiv:2111.09266).
 
file  embedding_export.hpp
 Hand embeddings or activations to UMAP and t-SNE tools, and bring their coordinates back as a projection document.
 
file  featurizer.hpp
 The featurizer family's common interface (FEAT-1): models that rewrite activations as sparse codes over learned directions and back, such as sparse autoencoders, with a shared training step and metrics (L0, dead and dense latents).
 
file  featurizer_metrics.hpp
 How good a featurizer is (FEAT-3), after SAEBench (Karvonen et al., arXiv 2503.09532): reconstruction (explained variance, cosine, norm ratio, L0, dead and dense latents), loss recovered when the reconstruction is spliced back into the model, and feature absorption (Chanin et al., arXiv 2409.14507) with a linear probe as its baseline.
 
file  gflownet_forward_policy.hpp
 Masked stochastic categorical policy over a logit-producing Module – P_F for GFlowNet training objectives.
 
file  gflownet_trajectory.hpp
 Rolls out one full HyperGrid episode under a GFlowNetForwardPolicy.
 
file  hypergrid_env.hpp
 HyperGrid – the standard minimal GFlowNet correctness-check environment.
 
file  jumprelu_sparse_autoencoder.hpp
 The JumpReLU sparse autoencoder (FEAT-4; Rajamanoharan et al., "Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders", arXiv 2407.14435): a learned threshold per latent, trained against an L0 penalty through straight-through estimators.
 
file  learnable_scalar.hpp
 A single trainable float, updated by plain SGD – not a Tensor/Module parameter.
 
file  linear_probe.hpp
 Linear probe – trains one linear classifier to decode a binary concept from a layer's activations, answering "is this concept linearly represented here?".
 
file  mlp_hook.hpp
 Reads or replaces a transformer block's MLP output during forward() (FEAT-5), for collecting a transcoder's training pairs and splicing it in.
 
file  nearest_neighbors.hpp
 Nearest neighbors over embeddings or activations, mutual nearest neighbors, and mutual-kNN alignment between two representations.
 
file  parameter_decomposition.hpp
 Parameter decomposition (FEAT-9): explaining a network in weight space. Each weight matrix is split into rank-one subcomponents, and training asks that, for every input, few of them be needed to compute the output. SPD (Bushnaq, Braun and Sharkey, "Stochastic Parameter Decomposition", arXiv 2506.20790) and VPD (Bushnaq et al., "Interpreting Language Model Parameters", Goodfire, 2026: adversarial parameter decomposition) are both here, after their reference code (github.com/goodfire-ai/ param-decomp: tag v1 for SPD, nano_param_decomp for VPD).
 
file  pca.hpp
 Principal component analysis of a layer's representations (INT-5): 2D and 3D views, directions to steer along, and fewer dimensions for what comes after.
 
file  probe_controls.hpp
 Linear probes with controls (INT-3): a multi-class probe trainer, Hewitt and Liang's control task and selectivity, and a comparison against a baseline representation.
 
file  relevance_graph.hpp
 A language model's attribution graph from AttnLRP (VIZ-4): how relevance for one predicted token flows from the input tokens through every layer's residual stream, ready for Neuronpedia's and circuit-tracer's graph viewers.
 
file  representation_geometry.hpp
 One number per layer for the shape of its representations: anisotropy, effective rank and TwoNN intrinsic dimension.
 
file  representation_similarity.hpp
 How similar two layers' representations of the same inputs are: linear CKA, RSA and SVCCA.
 
file  sparse_autoencoder.hpp
 Sparse autoencoder – reconstructs an activation through an overcomplete, L1-penalized hidden layer, decomposing it into a larger, sparser basis.
 
file  steering.hpp
 Steering a model by adding a direction to its residual stream (FEAT-7), with difference-of-means as the default direction, a featurizer's decoder direction as an option, and a report on how reliably steering works.
 
file  subtb_loss.hpp
 GFlowNet SubTB(lambda) loss (Madan et al., "Learning GFlowNets from partial episodes for improved convergence and stability", arXiv:2209.12782).
 
file  tcav.hpp
 Testing with Concept Activation Vectors (INT-6): how much a concept, given only by examples, pushes a model towards a class, and whether that's more than chance.
 
file  topk_sparse_autoencoder.hpp
 The TopK sparse autoencoder (FEAT-2; Gao et al., "Scaling and evaluating sparse autoencoders", arXiv 2406.04093): sparsity set directly by keeping each input's k largest latents, with an auxiliary loss that revives dead latents. Two variants (FEAT-4): BatchTopK (Bussmann et al., arXiv 2412.06410) and Matryoshka (Bussmann et al., arXiv 2503.17547).
 
file  trajectory_balance_loss.hpp
 GFlowNet Trajectory Balance loss (Malkin et al. 2022, arXiv:2201.13259).
 
file  transcoder.hpp
 Transcoders and skip transcoders (FEAT-5): sparse, wide replacements for an MLP that predict its output from its input (Dunefsky et al., "Transcoders Find Interpretable LLM Feature Circuits", arXiv 2406.11944), optionally beside a linear skip connection (Paulo et al., "Transcoders Beat Sparse Autoencoders for Interpretability", arXiv 2501.18823).
 

Detailed Description

Tools for inspecting and probing a model's internal computation directly – activation caching, linear probes and their controls, representation similarity (linear CKA, RSA, SVCCA), nearest neighbors and mutual-kNN alignment, representation geometry (anisotropy, effective rank, intrinsic dimension), PCA projections, export for UMAP and t-SNE, TCAV and CRP, sparse autoencoders, circuit graphs – plus the GFlowNet-based non-goal-directed training objectives motivated by the Scientist AI research direction.