pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TopKSaeOptions Struct Reference

#include <topk_sparse_autoencoder.hpp>

Public Attributes

int64_t k = 32
 Latents kept per input: L0 is exactly this (or less, where fewer are positive).
 
int64_t k_aux = 0
 The auxiliary loss's latents: the k_aux largest among the dead ones. 0 means half the input dimension, Gao et al.'s choice.
 
float aux_coefficient = 1.0f / 32.0f
 The auxiliary loss's weight, 1/32 in Gao et al.; 0 turns it off.
 
int64_t dead_after = 10'000'000
 A latent is dead once it hasn't fired in this many training inputs (Gao et al. use 10 million tokens).
 
uint64_t seed = 0
 
bool batch_topk = false
 BatchTopK: in training, keep the batch's N * k largest activations wherever they fall, so L0 is k on average and inputs that need more latents get them. Inference uses a threshold instead, an exponential moving average of the smallest kept value per training batch (threshold()), so an input's codes don't depend on its batch.
 
float threshold_decay = 0.99f
 The threshold's moving-average decay, per training batch.
 
std::vector< int64_t > matryoshka_prefixes
 Matryoshka: nested dictionary sizes, increasing (for example {m/16, m/4, m}); the full size is added if missing. Training minimizes the sum of every prefix's reconstruction error, the prefix using only latents [0, size), so the first latents learn general features on their own and the later ones refine them. Empty: off.
 

Member Data Documentation

◆ aux_coefficient

float pulsatrix::TopKSaeOptions::aux_coefficient = 1.0f / 32.0f

The auxiliary loss's weight, 1/32 in Gao et al.; 0 turns it off.

◆ batch_topk

bool pulsatrix::TopKSaeOptions::batch_topk = false

BatchTopK: in training, keep the batch's N * k largest activations wherever they fall, so L0 is k on average and inputs that need more latents get them. Inference uses a threshold instead, an exponential moving average of the smallest kept value per training batch (threshold()), so an input's codes don't depend on its batch.

◆ dead_after

int64_t pulsatrix::TopKSaeOptions::dead_after = 10'000'000

A latent is dead once it hasn't fired in this many training inputs (Gao et al. use 10 million tokens).

◆ k

int64_t pulsatrix::TopKSaeOptions::k = 32

Latents kept per input: L0 is exactly this (or less, where fewer are positive).

◆ k_aux

int64_t pulsatrix::TopKSaeOptions::k_aux = 0

The auxiliary loss's latents: the k_aux largest among the dead ones. 0 means half the input dimension, Gao et al.'s choice.

◆ matryoshka_prefixes

std::vector<int64_t> pulsatrix::TopKSaeOptions::matryoshka_prefixes

Matryoshka: nested dictionary sizes, increasing (for example {m/16, m/4, m}); the full size is added if missing. Training minimizes the sum of every prefix's reconstruction error, the prefix using only latents [0, size), so the first latents learn general features on their own and the later ones refine them. Empty: off.

◆ seed

uint64_t pulsatrix::TopKSaeOptions::seed = 0

◆ threshold_decay

float pulsatrix::TopKSaeOptions::threshold_decay = 0.99f

The threshold's moving-average decay, per training batch.


The documentation for this struct was generated from the following file: