Crosscoders (FEAT-8): one sparse dictionary read from and written to several sources at once, such as the residual stream at several layers, or the same layer in a base model and its fine-tune (Lindsey et al., "Sparse Crosscoders for Cross-Layer Features and
Model Diffing", Transformer Circuits, 2024). Model diffing reads which latents' decoders live in one model only, with Minder et al.'s BatchTopK crosscoder and Latent Scaling ("Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning", arXiv 2504.02922), and Kassem et al.'s Delta-Crosscoder for narrow fine-tunes ("Delta-
Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimes", arXiv 2603.04426).
More...
|
| std::vector< CrosscoderLatentStats > | pulsatrix::CrosscoderLatents (Crosscoder &crosscoder, int64_t a=0, int64_t b=1) |
| | Every latent's stats for sources a and b.
|
| |
| LatentClass | pulsatrix::ClassifyLatent (const CrosscoderLatentStats &stats) |
| | Minder et al.'s bins on Δnorm.
|
| |
| std::vector< LatentScaling > | pulsatrix::MeasureLatentScaling (Crosscoder &crosscoder, const Tensor &x, const std::vector< int64_t > &latents, int64_t a=0, int64_t b=1, int64_t batch=4096) |
| | Latent Scaling of each of latents on x, (N, S·d), in batches of batch. β is 0 (and ν 0) for a latent that never fires.
|
| |
| std::vector< double > | pulsatrix::ExplainedVarianceBySource (Crosscoder &crosscoder, const Tensor &x, int64_t batch=4096) |
| | Each source's explained variance on x: 1 - Σ|x^s - x̂^s|² / Σ|x^s - mean(x^s)|².
|
| |
Crosscoders (FEAT-8): one sparse dictionary read from and written to several sources at once, such as the residual stream at several layers, or the same layer in a base model and its fine-tune (Lindsey et al., "Sparse Crosscoders for Cross-Layer Features and
Model Diffing", Transformer Circuits, 2024). Model diffing reads which latents' decoders live in one model only, with Minder et al.'s BatchTopK crosscoder and Latent Scaling ("Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning", arXiv 2504.02922), and Kassem et al.'s Delta-Crosscoder for narrow fine-tunes ("Delta-
Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimes", arXiv 2603.04426).