#include <crosscoder.hpp>
|
| CrosscoderSparsity | sparsity = CrosscoderSparsity::BatchTopK |
| |
| float | l1_coefficient = 1e-3f |
| | L1: λ.
|
| |
| int64_t | k = 32 |
| | BatchTopK: mean latents kept per input (the delta partition's, with delta).
|
| |
| int64_t | k_aux = 0 |
| | AuxK, as TopKSparseAutoencoder: the k_aux largest dead latents reconstruct the error (0: half the input size), weighted by aux_coefficient (0 turns it off).
|
| |
| float | aux_coefficient = 1.0f / 32.0f |
| |
| int64_t | dead_after = 10'000'000 |
| |
| float | threshold_decay = 0.99f |
| | BatchTopK's inference threshold: a moving average of the smallest kept scaled value per training batch, with this decay.
|
| |
| float | decoder_init_norm = 0.0f |
| | Every latent's decoder starts at this norm in every source (0: 0.05 for L1, 1 for BatchTopK, Minder et al.'s values; they report the small L1 value is crucial).
|
| |
| bool | delta = false |
| | The Delta-Crosscoder (Kassem et al.), for two sources: source 0 the base model and source 1 the fine-tune, BatchTopK only.
|
| |
| double | shared_fraction = 0.2 |
| |
| int64_t | k_shared = 0 |
| | The shared partition's latents per input (0: twice k, the paper's multiplier).
|
| |
| float | delta_coefficient = 0.005f |
| |
| uint64_t | seed = 0 |
| |
◆ aux_coefficient
| float pulsatrix::CrosscoderOptions::aux_coefficient = 1.0f / 32.0f |
◆ dead_after
| int64_t pulsatrix::CrosscoderOptions::dead_after = 10'000'000 |
◆ decoder_init_norm
| float pulsatrix::CrosscoderOptions::decoder_init_norm = 0.0f |
Every latent's decoder starts at this norm in every source (0: 0.05 for L1, 1 for BatchTopK, Minder et al.'s values; they report the small L1 value is crucial).
◆ delta
| bool pulsatrix::CrosscoderOptions::delta = false |
The Delta-Crosscoder (Kassem et al.), for two sources: source 0 the base model and source 1 the fine-tune, BatchTopK only.
- The encoder averages the sources:
u = ½ Σ_s x^s W_enc^s + b_enc.
- The first shared_fraction of the latents are shared, kept by their own BatchTopK at k_shared per input; the rest are delta latents, at k.
- A delta loss has the delta latents alone predict the fine-tune's change:
delta_coefficient · mean((z_Δ (W_dec^1 - W_dec^0) - (x^1 - x^0))²).
◆ delta_coefficient
| float pulsatrix::CrosscoderOptions::delta_coefficient = 0.005f |
| int64_t pulsatrix::CrosscoderOptions::k = 32 |
BatchTopK: mean latents kept per input (the delta partition's, with delta).
◆ k_aux
| int64_t pulsatrix::CrosscoderOptions::k_aux = 0 |
AuxK, as TopKSparseAutoencoder: the k_aux largest dead latents reconstruct the error (0: half the input size), weighted by aux_coefficient (0 turns it off).
◆ k_shared
| int64_t pulsatrix::CrosscoderOptions::k_shared = 0 |
The shared partition's latents per input (0: twice k, the paper's multiplier).
◆ l1_coefficient
| float pulsatrix::CrosscoderOptions::l1_coefficient = 1e-3f |
◆ seed
| uint64_t pulsatrix::CrosscoderOptions::seed = 0 |
◆ shared_fraction
| double pulsatrix::CrosscoderOptions::shared_fraction = 0.2 |
◆ sparsity
◆ threshold_decay
| float pulsatrix::CrosscoderOptions::threshold_decay = 0.99f |
BatchTopK's inference threshold: a moving average of the smallest kept scaled value per training batch, with this decay.
The documentation for this struct was generated from the following file: