|
pulsatrix 1.1.0
|
The TopK sparse autoencoder (FEAT-2; Gao et al., "Scaling and evaluating sparse autoencoders", arXiv 2406.04093): sparsity set directly by keeping each input's k largest latents, with an auxiliary loss that revives dead latents. Two variants (FEAT-4): BatchTopK (Bussmann et al., arXiv 2412.06410) and Matryoshka (Bussmann et al., arXiv 2503.17547). More...
#include <cstdint>#include <optional>#include <vector>#include "pulsatrix/featurizer.hpp"#include "pulsatrix/linear_module.hpp"
Go to the source code of this file.
Classes | |
| struct | pulsatrix::TopKSaeOptions |
| class | pulsatrix::TopKSparseAutoencoder |
A TopK sparse autoencoder: z = ReLU(TopK_k(W_enc (x - b_dec) + b_enc)), x̂ = z W_dec + b_dec. More... | |
Namespaces | |
| namespace | pulsatrix |
The TopK sparse autoencoder (FEAT-2; Gao et al., "Scaling and evaluating sparse autoencoders", arXiv 2406.04093): sparsity set directly by keeping each input's k largest latents, with an auxiliary loss that revives dead latents. Two variants (FEAT-4): BatchTopK (Bussmann et al., arXiv 2412.06410) and Matryoshka (Bussmann et al., arXiv 2503.17547).