pulsatrix 1.1.0
Loading...
Searching...
No Matches
topk_sparse_autoencoder.hpp File Reference

The TopK sparse autoencoder (FEAT-2; Gao et al., "Scaling and evaluating sparse autoencoders", arXiv 2406.04093): sparsity set directly by keeping each input's k largest latents, with an auxiliary loss that revives dead latents. Two variants (FEAT-4): BatchTopK (Bussmann et al., arXiv 2412.06410) and Matryoshka (Bussmann et al., arXiv 2503.17547). More...

#include <cstdint>
#include <optional>
#include <vector>
#include "pulsatrix/featurizer.hpp"
#include "pulsatrix/linear_module.hpp"
Include dependency graph for topk_sparse_autoencoder.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::TopKSaeOptions
 
class  pulsatrix::TopKSparseAutoencoder
 A TopK sparse autoencoder: z = ReLU(TopK_k(W_enc (x - b_dec) + b_enc)), x̂ = z W_dec + b_dec. More...
 

Namespaces

namespace  pulsatrix
 

Detailed Description

The TopK sparse autoencoder (FEAT-2; Gao et al., "Scaling and evaluating sparse autoencoders", arXiv 2406.04093): sparsity set directly by keeping each input's k largest latents, with an auxiliary loss that revives dead latents. Two variants (FEAT-4): BatchTopK (Bussmann et al., arXiv 2412.06410) and Matryoshka (Bussmann et al., arXiv 2503.17547).