|
pulsatrix 1.1.0
|
Transcoders and skip transcoders (FEAT-5): sparse, wide replacements for an MLP that predict its output from its input (Dunefsky et al., "Transcoders Find Interpretable LLM Feature Circuits", arXiv 2406.11944), optionally beside a linear skip connection (Paulo et al., "Transcoders Beat Sparse Autoencoders for Interpretability", arXiv 2501.18823). More...
#include <cstdint>#include <memory>#include <optional>#include <vector>#include "pulsatrix/featurizer.hpp"#include "pulsatrix/linear_module.hpp"
Go to the source code of this file.
Classes | |
| struct | pulsatrix::TranscoderOptions |
| class | pulsatrix::Transcoder |
A transcoder: z = ReLU(TopK_k(x W_enc + b_enc)), ŷ = z W_dec + b_dec [+ x W_skip], trained to predict an MLP's output y from its input x by mean((ŷ - y)²) plus AuxK. More... | |
Namespaces | |
| namespace | pulsatrix |
Transcoders and skip transcoders (FEAT-5): sparse, wide replacements for an MLP that predict its output from its input (Dunefsky et al., "Transcoders Find Interpretable LLM Feature Circuits", arXiv 2406.11944), optionally beside a linear skip connection (Paulo et al., "Transcoders Beat Sparse Autoencoders for Interpretability", arXiv 2501.18823).