|
pulsatrix 1.1.0
|
A cross-layer transcoder (CLT) over a model's L MLPs, or a set of per-layer transcoders (PLTs, a CLT whose features only write to their own layer). More...
#include <circuit_tracing.hpp>
Public Member Functions | |
| CrossLayerTranscoder (int64_t num_layers, int64_t d_model, DeviceBackend *backend) | |
| void | set_layer (int64_t layer, Tensor w_enc, std::vector< float > b_enc, std::vector< float > threshold, std::vector< Tensor > w_dec) |
Sets layer layer's features. | |
| void | set_output_bias (int64_t layer, std::vector< float > b_dec) |
The decoder bias of layer layer's MLP output, d values (zero by default). | |
| void | set_skip (int64_t layer, Tensor w_skip) |
A linear skip from layer layer's MLP input to its output, (d, d). | |
| int64_t | num_layers () const |
| int64_t | d_model () const |
| int64_t | num_features (int64_t layer) const |
| int64_t | span (int64_t layer) const |
How many layers layer layer's features write to (1 for a PLT). | |
| bool | has_skip (int64_t layer) const |
| std::vector< float > | preactivations (int64_t layer, const Tensor &x) const |
Pre-activations x W_encᵀ + b_enc, (rows, F) on the host, for x (rows, d). | |
| std::vector< float > | encode (int64_t layer, const Tensor &x) const |
| The activations: JumpReLU (or ReLU) of the pre-activations. | |
| std::vector< float > | reconstruct (int64_t layer, const std::vector< std::vector< float > > &activations, const Tensor *mlp_input) const |
Layer layer's MLP output as the transcoder predicts it, (rows, d) on the host: every earlier layer's decoders into it, its bias and its skip. | |
| std::vector< float > | encoder_row (int64_t layer, int64_t f) const |
Feature f of layer layer: its encoder row and its decoder into out_layer, d each. | |
| std::vector< float > | decoder_row (int64_t layer, int64_t f, int64_t out_layer) const |
| const Tensor & | encoder (int64_t layer) const |
| const Tensor & | decoder (int64_t layer, int64_t out_layer) const |
| const Tensor & | skip (int64_t layer) const |
| DeviceBackend * | backend () const |
A cross-layer transcoder (CLT) over a model's L MLPs, or a set of per-layer transcoders (PLTs, a CLT whose features only write to their own layer).
The features of layer ℓ read the MLP's input there (after the block's pre-MLP norm), and write to the MLP outputs of layers ℓ to ℓ + span − 1:
pre^ℓ = x^ℓ W_enc^ℓᵀ + b_enc^ℓ, a^ℓ = pre^ℓ [pre^ℓ > θ^ℓ] (JumpReLU; ReLU without θ);Weights are stored as circuit-tracer does: W_enc^ℓ (F, d), each W_dec (F, d), W_skip (d, d), on the backend's device. Inference only: pretrained transcoders are loaded with LoadTranscoders().
| pulsatrix::CrossLayerTranscoder::CrossLayerTranscoder | ( | int64_t | num_layers, |
| int64_t | d_model, | ||
| DeviceBackend * | backend | ||
| ) |
| std::invalid_argument | for a size < 1. |
|
inline |
|
inline |
| const Tensor & pulsatrix::CrossLayerTranscoder::decoder | ( | int64_t | layer, |
| int64_t | out_layer | ||
| ) | const |
| std::vector< float > pulsatrix::CrossLayerTranscoder::decoder_row | ( | int64_t | layer, |
| int64_t | f, | ||
| int64_t | out_layer | ||
| ) | const |
| std::vector< float > pulsatrix::CrossLayerTranscoder::encode | ( | int64_t | layer, |
| const Tensor & | x | ||
| ) | const |
The activations: JumpReLU (or ReLU) of the pre-activations.
| const Tensor & pulsatrix::CrossLayerTranscoder::encoder | ( | int64_t | layer | ) | const |
| std::vector< float > pulsatrix::CrossLayerTranscoder::encoder_row | ( | int64_t | layer, |
| int64_t | f | ||
| ) | const |
Feature f of layer layer: its encoder row and its decoder into out_layer, d each.
| bool pulsatrix::CrossLayerTranscoder::has_skip | ( | int64_t | layer | ) | const |
| int64_t pulsatrix::CrossLayerTranscoder::num_features | ( | int64_t | layer | ) | const |
|
inline |
| std::vector< float > pulsatrix::CrossLayerTranscoder::preactivations | ( | int64_t | layer, |
| const Tensor & | x | ||
| ) | const |
Pre-activations x W_encᵀ + b_enc, (rows, F) on the host, for x (rows, d).
| std::vector< float > pulsatrix::CrossLayerTranscoder::reconstruct | ( | int64_t | layer, |
| const std::vector< std::vector< float > > & | activations, | ||
| const Tensor * | mlp_input | ||
| ) | const |
Layer layer's MLP output as the transcoder predicts it, (rows, d) on the host: every earlier layer's decoders into it, its bias and its skip.
| activations | activations[ℓ] (rows, F_ℓ) for each ℓ ≤ layer (later ones ignored). |
| mlp_input | Layer layer's MLP input (rows, d), for the skip; ignored without one. |
| void pulsatrix::CrossLayerTranscoder::set_layer | ( | int64_t | layer, |
| Tensor | w_enc, | ||
| std::vector< float > | b_enc, | ||
| std::vector< float > | threshold, | ||
| std::vector< Tensor > | w_dec | ||
| ) |
Sets layer layer's features.
| w_enc | (F, d). |
| b_enc | F values. |
| threshold | F values (JumpReLU), or empty for ReLU. |
| w_dec | One (F, d) matrix per output layer, from layer on; at most L − layer. |
| std::invalid_argument | for a layer out of range or inconsistent sizes. |
| void pulsatrix::CrossLayerTranscoder::set_output_bias | ( | int64_t | layer, |
| std::vector< float > | b_dec | ||
| ) |
The decoder bias of layer layer's MLP output, d values (zero by default).
| void pulsatrix::CrossLayerTranscoder::set_skip | ( | int64_t | layer, |
| Tensor | w_skip | ||
| ) |
A linear skip from layer layer's MLP input to its output, (d, d).
| const Tensor & pulsatrix::CrossLayerTranscoder::skip | ( | int64_t | layer | ) | const |
| int64_t pulsatrix::CrossLayerTranscoder::span | ( | int64_t | layer | ) | const |
How many layers layer layer's features write to (1 for a PLT).