pulsatrix 1.1.0
Loading...
Searching...
No Matches
transcoder.hpp File Reference

Transcoders and skip transcoders (FEAT-5): sparse, wide replacements for an MLP that predict its output from its input (Dunefsky et al., "Transcoders Find Interpretable LLM Feature Circuits", arXiv 2406.11944), optionally beside a linear skip connection (Paulo et al., "Transcoders Beat Sparse Autoencoders for Interpretability", arXiv 2501.18823). More...

#include <cstdint>
#include <memory>
#include <optional>
#include <vector>
#include "pulsatrix/featurizer.hpp"
#include "pulsatrix/linear_module.hpp"
Include dependency graph for transcoder.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::TranscoderOptions
 
class  pulsatrix::Transcoder
 A transcoder: z = ReLU(TopK_k(x W_enc + b_enc)), ŷ = z W_dec + b_dec [+ x W_skip], trained to predict an MLP's output y from its input x by mean((ŷ - y)²) plus AuxK. More...
 

Namespaces

namespace  pulsatrix
 

Detailed Description

Transcoders and skip transcoders (FEAT-5): sparse, wide replacements for an MLP that predict its output from its input (Dunefsky et al., "Transcoders Find Interpretable LLM Feature Circuits", arXiv 2406.11944), optionally beside a linear skip connection (Paulo et al., "Transcoders Beat Sparse Autoencoders for Interpretability", arXiv 2501.18823).