pulsatrix 1.1.0
Loading...
Searching...
No Matches
parameter_decomposition.hpp File Reference

Parameter decomposition (FEAT-9): explaining a network in weight space. Each weight matrix is split into rank-one subcomponents, and training asks that, for every input, few of them be needed to compute the output. SPD (Bushnaq, Braun and Sharkey, "Stochastic Parameter Decomposition", arXiv 2506.20790) and VPD (Bushnaq et al., "Interpreting Language Model Parameters", Goodfire, 2026: adversarial parameter decomposition) are both here, after their reference code (github.com/goodfire-ai/ param-decomp: tag v1 for SPD, nano_param_decomp for VPD). More...

#include <cstdint>
#include <functional>
#include <memory>
#include <optional>
#include <vector>
#include "pulsatrix/linear_module.hpp"
#include "pulsatrix/module.hpp"
Include dependency graph for parameter_decomposition.hpp:

Go to the source code of this file.

Classes

class  pulsatrix::ComponentLinear
 A LinearModule split into C rank-one subcomponents: W ≈ V U, V (in, C) reading the input and U (C, out) writing the output, so subcomponent c is V[:, c] U[c, :] and its inner activation is h_c = (x V)_c. The bias is the target's, frozen. More...
 
struct  pulsatrix::DecompositionOptions
 
struct  pulsatrix::DecompositionLoss
 
class  pulsatrix::ParameterDecomposition
 Trains a decomposition of the ComponentLinear layers inside model against the model itself: the target is the model with every layer in target mode. More...
 
struct  pulsatrix::ComponentAlignment
 How well a layer's subcomponents line up with its target weight's rows, the input features (SPD's toy-model measure): for each input feature j, the best cosine over c between subcomponent c's row j, V[j, c] U[c, :], and the target's row j (MMCS), and that subcomponent's norm over the target row's (ML2R). More...
 
struct  pulsatrix::ImportanceStats
 Causal importance statistics of one layer over inputs. More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::DecompositionMethod { pulsatrix::SPD , pulsatrix::VPD }
 
enum class  pulsatrix::OutputDivergence { pulsatrix::MeanSquaredError , pulsatrix::KlOnLogits }
 How the masked model's output is compared with the target's. More...
 

Functions

template<typename Optimizer >
DecompositionLoss pulsatrix::TrainDecomposition (ParameterDecomposition &d, const Tensor &x, Optimizer &optimizer)
 One training step: zero the gradients, loss_and_backward(), one optimizer step.
 
ComponentAlignment pulsatrix::AlignComponentsToRows (ComponentLinear &layer)
 
std::vector< ImportanceStats > pulsatrix::MeasureImportance (ParameterDecomposition &d, const Tensor &x)
 

Detailed Description

Parameter decomposition (FEAT-9): explaining a network in weight space. Each weight matrix is split into rank-one subcomponents, and training asks that, for every input, few of them be needed to compute the output. SPD (Bushnaq, Braun and Sharkey, "Stochastic Parameter Decomposition", arXiv 2506.20790) and VPD (Bushnaq et al., "Interpreting Language Model Parameters", Goodfire, 2026: adversarial parameter decomposition) are both here, after their reference code (github.com/goodfire-ai/ param-decomp: tag v1 for SPD, nano_param_decomp for VPD).