pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::CrosscoderOptions Struct Reference

#include <crosscoder.hpp>

Public Attributes

CrosscoderSparsity sparsity = CrosscoderSparsity::BatchTopK
 
float l1_coefficient = 1e-3f
 L1: λ.
 
int64_t k = 32
 BatchTopK: mean latents kept per input (the delta partition's, with delta).
 
int64_t k_aux = 0
 AuxK, as TopKSparseAutoencoder: the k_aux largest dead latents reconstruct the error (0: half the input size), weighted by aux_coefficient (0 turns it off).
 
float aux_coefficient = 1.0f / 32.0f
 
int64_t dead_after = 10'000'000
 
float threshold_decay = 0.99f
 BatchTopK's inference threshold: a moving average of the smallest kept scaled value per training batch, with this decay.
 
float decoder_init_norm = 0.0f
 Every latent's decoder starts at this norm in every source (0: 0.05 for L1, 1 for BatchTopK, Minder et al.'s values; they report the small L1 value is crucial).
 
bool delta = false
 The Delta-Crosscoder (Kassem et al.), for two sources: source 0 the base model and source 1 the fine-tune, BatchTopK only.
 
double shared_fraction = 0.2
 
int64_t k_shared = 0
 The shared partition's latents per input (0: twice k, the paper's multiplier).
 
float delta_coefficient = 0.005f
 
uint64_t seed = 0
 

Member Data Documentation

◆ aux_coefficient

float pulsatrix::CrosscoderOptions::aux_coefficient = 1.0f / 32.0f

◆ dead_after

int64_t pulsatrix::CrosscoderOptions::dead_after = 10'000'000

◆ decoder_init_norm

float pulsatrix::CrosscoderOptions::decoder_init_norm = 0.0f

Every latent's decoder starts at this norm in every source (0: 0.05 for L1, 1 for BatchTopK, Minder et al.'s values; they report the small L1 value is crucial).

◆ delta

bool pulsatrix::CrosscoderOptions::delta = false

The Delta-Crosscoder (Kassem et al.), for two sources: source 0 the base model and source 1 the fine-tune, BatchTopK only.

  • The encoder averages the sources: u = ½ Σ_s x^s W_enc^s + b_enc.
  • The first shared_fraction of the latents are shared, kept by their own BatchTopK at k_shared per input; the rest are delta latents, at k.
  • A delta loss has the delta latents alone predict the fine-tune's change: delta_coefficient · mean((z_Δ (W_dec^1 - W_dec^0) - (x^1 - x^0))²).

◆ delta_coefficient

float pulsatrix::CrosscoderOptions::delta_coefficient = 0.005f

◆ k

int64_t pulsatrix::CrosscoderOptions::k = 32

BatchTopK: mean latents kept per input (the delta partition's, with delta).

◆ k_aux

int64_t pulsatrix::CrosscoderOptions::k_aux = 0

AuxK, as TopKSparseAutoencoder: the k_aux largest dead latents reconstruct the error (0: half the input size), weighted by aux_coefficient (0 turns it off).

◆ k_shared

int64_t pulsatrix::CrosscoderOptions::k_shared = 0

The shared partition's latents per input (0: twice k, the paper's multiplier).

◆ l1_coefficient

float pulsatrix::CrosscoderOptions::l1_coefficient = 1e-3f

L1: λ.

◆ seed

uint64_t pulsatrix::CrosscoderOptions::seed = 0

◆ shared_fraction

double pulsatrix::CrosscoderOptions::shared_fraction = 0.2

◆ sparsity

CrosscoderSparsity pulsatrix::CrosscoderOptions::sparsity = CrosscoderSparsity::BatchTopK

◆ threshold_decay

float pulsatrix::CrosscoderOptions::threshold_decay = 0.99f

BatchTopK's inference threshold: a moving average of the smallest kept scaled value per training batch, with this decay.


The documentation for this struct was generated from the following file: