pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::CrossLayerTranscoder Class Reference

A cross-layer transcoder (CLT) over a model's L MLPs, or a set of per-layer transcoders (PLTs, a CLT whose features only write to their own layer). More...

#include <circuit_tracing.hpp>

Public Member Functions

 CrossLayerTranscoder (int64_t num_layers, int64_t d_model, DeviceBackend *backend)
 
void set_layer (int64_t layer, Tensor w_enc, std::vector< float > b_enc, std::vector< float > threshold, std::vector< Tensor > w_dec)
 Sets layer layer's features.
 
void set_output_bias (int64_t layer, std::vector< float > b_dec)
 The decoder bias of layer layer's MLP output, d values (zero by default).
 
void set_skip (int64_t layer, Tensor w_skip)
 A linear skip from layer layer's MLP input to its output, (d, d).
 
int64_t num_layers () const
 
int64_t d_model () const
 
int64_t num_features (int64_t layer) const
 
int64_t span (int64_t layer) const
 How many layers layer layer's features write to (1 for a PLT).
 
bool has_skip (int64_t layer) const
 
std::vector< float > preactivations (int64_t layer, const Tensor &x) const
 Pre-activations x W_encᵀ + b_enc, (rows, F) on the host, for x (rows, d).
 
std::vector< float > encode (int64_t layer, const Tensor &x) const
 The activations: JumpReLU (or ReLU) of the pre-activations.
 
std::vector< float > reconstruct (int64_t layer, const std::vector< std::vector< float > > &activations, const Tensor *mlp_input) const
 Layer layer's MLP output as the transcoder predicts it, (rows, d) on the host: every earlier layer's decoders into it, its bias and its skip.
 
std::vector< float > encoder_row (int64_t layer, int64_t f) const
 Feature f of layer layer: its encoder row and its decoder into out_layer, d each.
 
std::vector< float > decoder_row (int64_t layer, int64_t f, int64_t out_layer) const
 
const Tensor & encoder (int64_t layer) const
 
const Tensor & decoder (int64_t layer, int64_t out_layer) const
 
const Tensor & skip (int64_t layer) const
 
DeviceBackend * backend () const
 

Detailed Description

A cross-layer transcoder (CLT) over a model's L MLPs, or a set of per-layer transcoders (PLTs, a CLT whose features only write to their own layer).

The features of layer ℓ read the MLP's input there (after the block's pre-MLP norm), and write to the MLP outputs of layers ℓ to ℓ + span − 1:

  • pre^ℓ = x^ℓ W_enc^ℓᵀ + b_enc^ℓ, a^ℓ = pre^ℓ [pre^ℓ > θ^ℓ] (JumpReLU; ReLU without θ);
  • ‘ŷ^ℓ’ = Σ_{ℓ ≤ ℓ'} a^ℓ W_dec^{ℓ→ℓ'} + b_dec^ℓ' [+ x^ℓ' W_skip^ℓ'ᵀ]`.

Weights are stored as circuit-tracer does: W_enc^ℓ (F, d), each W_dec (F, d), W_skip (d, d), on the backend's device. Inference only: pretrained transcoders are loaded with LoadTranscoders().

Constructor & Destructor Documentation

◆ CrossLayerTranscoder()

pulsatrix::CrossLayerTranscoder::CrossLayerTranscoder ( int64_t  num_layers,
int64_t  d_model,
DeviceBackend *  backend 
)
Exceptions
std::invalid_argumentfor a size < 1.

Member Function Documentation

◆ backend()

DeviceBackend * pulsatrix::CrossLayerTranscoder::backend ( ) const
inline

◆ d_model()

int64_t pulsatrix::CrossLayerTranscoder::d_model ( ) const
inline

◆ decoder()

const Tensor & pulsatrix::CrossLayerTranscoder::decoder ( int64_t  layer,
int64_t  out_layer 
) const

◆ decoder_row()

std::vector< float > pulsatrix::CrossLayerTranscoder::decoder_row ( int64_t  layer,
int64_t  f,
int64_t  out_layer 
) const

◆ encode()

std::vector< float > pulsatrix::CrossLayerTranscoder::encode ( int64_t  layer,
const Tensor &  x 
) const

The activations: JumpReLU (or ReLU) of the pre-activations.

◆ encoder()

const Tensor & pulsatrix::CrossLayerTranscoder::encoder ( int64_t  layer) const

◆ encoder_row()

std::vector< float > pulsatrix::CrossLayerTranscoder::encoder_row ( int64_t  layer,
int64_t  f 
) const

Feature f of layer layer: its encoder row and its decoder into out_layer, d each.

◆ has_skip()

bool pulsatrix::CrossLayerTranscoder::has_skip ( int64_t  layer) const

◆ num_features()

int64_t pulsatrix::CrossLayerTranscoder::num_features ( int64_t  layer) const

◆ num_layers()

int64_t pulsatrix::CrossLayerTranscoder::num_layers ( ) const
inline

◆ preactivations()

std::vector< float > pulsatrix::CrossLayerTranscoder::preactivations ( int64_t  layer,
const Tensor &  x 
) const

Pre-activations x W_encᵀ + b_enc, (rows, F) on the host, for x (rows, d).

◆ reconstruct()

std::vector< float > pulsatrix::CrossLayerTranscoder::reconstruct ( int64_t  layer,
const std::vector< std::vector< float > > &  activations,
const Tensor *  mlp_input 
) const

Layer layer's MLP output as the transcoder predicts it, (rows, d) on the host: every earlier layer's decoders into it, its bias and its skip.

Parameters
activationsactivations[ℓ] (rows, F_ℓ) for each ℓ ≤ layer (later ones ignored).
mlp_inputLayer layer's MLP input (rows, d), for the skip; ignored without one.

◆ set_layer()

void pulsatrix::CrossLayerTranscoder::set_layer ( int64_t  layer,
Tensor  w_enc,
std::vector< float >  b_enc,
std::vector< float >  threshold,
std::vector< Tensor >  w_dec 
)

Sets layer layer's features.

Parameters
w_enc(F, d).
b_encF values.
thresholdF values (JumpReLU), or empty for ReLU.
w_decOne (F, d) matrix per output layer, from layer on; at most L − layer.
Exceptions
std::invalid_argumentfor a layer out of range or inconsistent sizes.

◆ set_output_bias()

void pulsatrix::CrossLayerTranscoder::set_output_bias ( int64_t  layer,
std::vector< float >  b_dec 
)

The decoder bias of layer layer's MLP output, d values (zero by default).

◆ set_skip()

void pulsatrix::CrossLayerTranscoder::set_skip ( int64_t  layer,
Tensor  w_skip 
)

A linear skip from layer layer's MLP input to its output, (d, d).

◆ skip()

const Tensor & pulsatrix::CrossLayerTranscoder::skip ( int64_t  layer) const

◆ span()

int64_t pulsatrix::CrossLayerTranscoder::span ( int64_t  layer) const

How many layers layer layer's features write to (1 for a PLT).


The documentation for this class was generated from the following file: