pulsatrix 1.1.0
Loading...
Searching...
No Matches
mlp_hook.hpp File Reference

Reads or replaces a transformer block's MLP output during forward() (FEAT-5), for collecting a transcoder's training pairs and splicing it in. More...

#include <functional>
#include <stdexcept>
#include "pulsatrix/tensor.hpp"
Include dependency graph for mlp_hook.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Namespaces

namespace  pulsatrix
 
namespace  pulsatrix::detail
 

Typedefs

using pulsatrix::MlpHook = std::function< Tensor(const Tensor &mlp_input, const Tensor &mlp_output)>
 Called with an MLP's input (after the block's norm, where it has one) and its output (the branch added to the residual stream), both shaped like the block's input. What it returns, the same shape, is added to the residual stream instead of the output: return the output unchanged to only read it.
 

Functions

Tensor pulsatrix::detail::ApplyMlpHook (const MlpHook &hook, const Tensor &input, const Tensor &output)
 Runs hook and checks its result keeps the shape.
 

Detailed Description

Reads or replaces a transformer block's MLP output during forward() (FEAT-5), for collecting a transcoder's training pairs and splicing it in.