pulsatrix 1.1.0
Loading...
Searching...
No Matches
jumprelu_sparse_autoencoder.hpp File Reference

The JumpReLU sparse autoencoder (FEAT-4; Rajamanoharan et al., "Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders", arXiv 2407.14435): a learned threshold per latent, trained against an L0 penalty through straight-through estimators. More...

#include <cstdint>
#include <optional>
#include <vector>
#include "pulsatrix/featurizer.hpp"
#include "pulsatrix/linear_module.hpp"
Include dependency graph for jumprelu_sparse_autoencoder.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::JumpReLUSaeOptions
 
class  pulsatrix::JumpReLUSparseAutoencoder
 A JumpReLU sparse autoencoder: z = pre · H(pre - θ) with pre = x W_enc + b_enc, and x̂ = z W_dec + b_dec. Each latent has its own threshold θ > 0, kept as log θ. More...
 

Namespaces

namespace  pulsatrix
 

Detailed Description

The JumpReLU sparse autoencoder (FEAT-4; Rajamanoharan et al., "Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders", arXiv 2407.14435): a learned threshold per latent, trained against an L0 penalty through straight-through estimators.