#include <topk_sparse_autoencoder.hpp>
|
| int64_t | k = 32 |
| | Latents kept per input: L0 is exactly this (or less, where fewer are positive).
|
| |
| int64_t | k_aux = 0 |
| | The auxiliary loss's latents: the k_aux largest among the dead ones. 0 means half the input dimension, Gao et al.'s choice.
|
| |
| float | aux_coefficient = 1.0f / 32.0f |
| | The auxiliary loss's weight, 1/32 in Gao et al.; 0 turns it off.
|
| |
| int64_t | dead_after = 10'000'000 |
| | A latent is dead once it hasn't fired in this many training inputs (Gao et al. use 10 million tokens).
|
| |
| uint64_t | seed = 0 |
| |
| bool | batch_topk = false |
| | BatchTopK: in training, keep the batch's N * k largest activations wherever they fall, so L0 is k on average and inputs that need more latents get them. Inference uses a threshold instead, an exponential moving average of the smallest kept value per training batch (threshold()), so an input's codes don't depend on its batch.
|
| |
| float | threshold_decay = 0.99f |
| | The threshold's moving-average decay, per training batch.
|
| |
| std::vector< int64_t > | matryoshka_prefixes |
| | Matryoshka: nested dictionary sizes, increasing (for example {m/16, m/4, m}); the full size is added if missing. Training minimizes the sum of every prefix's reconstruction error, the prefix using only latents [0, size), so the first latents learn general features on their own and the later ones refine them. Empty: off.
|
| |
◆ aux_coefficient
| float pulsatrix::TopKSaeOptions::aux_coefficient = 1.0f / 32.0f |
The auxiliary loss's weight, 1/32 in Gao et al.; 0 turns it off.
◆ batch_topk
| bool pulsatrix::TopKSaeOptions::batch_topk = false |
BatchTopK: in training, keep the batch's N * k largest activations wherever they fall, so L0 is k on average and inputs that need more latents get them. Inference uses a threshold instead, an exponential moving average of the smallest kept value per training batch (threshold()), so an input's codes don't depend on its batch.
◆ dead_after
| int64_t pulsatrix::TopKSaeOptions::dead_after = 10'000'000 |
A latent is dead once it hasn't fired in this many training inputs (Gao et al. use 10 million tokens).
| int64_t pulsatrix::TopKSaeOptions::k = 32 |
Latents kept per input: L0 is exactly this (or less, where fewer are positive).
◆ k_aux
| int64_t pulsatrix::TopKSaeOptions::k_aux = 0 |
The auxiliary loss's latents: the k_aux largest among the dead ones. 0 means half the input dimension, Gao et al.'s choice.
◆ matryoshka_prefixes
| std::vector<int64_t> pulsatrix::TopKSaeOptions::matryoshka_prefixes |
Matryoshka: nested dictionary sizes, increasing (for example {m/16, m/4, m}); the full size is added if missing. Training minimizes the sum of every prefix's reconstruction error, the prefix using only latents [0, size), so the first latents learn general features on their own and the later ones refine them. Empty: off.
◆ seed
| uint64_t pulsatrix::TopKSaeOptions::seed = 0 |
◆ threshold_decay
| float pulsatrix::TopKSaeOptions::threshold_decay = 0.99f |
The threshold's moving-average decay, per training batch.
The documentation for this struct was generated from the following file: