pulsatrix 1.1.0
Loading...
Searching...
No Matches
swiglu_module.hpp
Go to the documentation of this file.
1
6#pragma once
7
8#include <optional>
10#include "pulsatrix/module.hpp"
11
12namespace pulsatrix {
13
17
53class SwiGLUModule : public Module {
54public:
68 SwiGLUModule(int64_t d_model, int64_t d_ff, DeviceBackend* backend, bool use_bias = true,
70 [[nodiscard]] GatedActivation activation() const { return activation_; }
71
83 [[nodiscard]] Tensor backward(const Tensor& grad_output) override;
84
86 [[nodiscard]] OpType op_type() const override { return OpType::Elementwise; }
87
100 [[nodiscard]] Tensor propagate_relevance(const Tensor& relevance_out, const LRPRuleConfig& config) override;
101
103 [[nodiscard]] std::vector<NamedParamRef> named_parameters() override;
104 [[nodiscard]] std::vector<NamedModuleRef> named_children() override;
105
107 [[nodiscard]] std::vector<NamedBufferRef> named_buffers() override;
108
109 [[nodiscard]] int64_t d_model() const { return d_model_; }
110 [[nodiscard]] int64_t d_ff() const { return d_ff_; }
111
114 [[nodiscard]] LinearModule& gate_proj() { return gate_proj_; }
115 [[nodiscard]] LinearModule& up_proj() { return up_proj_; }
116 [[nodiscard]] LinearModule& down_proj() { return down_proj_; }
118
119
121 [[nodiscard]] std::optional<DeviceType> compute_device() const override { return backend_->device(); }
122
123 void release_activations() override;
124
125protected:
132 [[nodiscard]] Tensor forward_impl(const Tensor& input) override;
133
134private:
135 int64_t d_model_;
136 int64_t d_ff_;
137 DeviceBackend* backend_;
139
140 LinearModule gate_proj_;
141 LinearModule up_proj_;
142 LinearModule down_proj_;
143
144 // Forward caches -- the operands the gate multiply's forward/backward/LRP rule needs.
145 // gate_proj_/up_proj_/down_proj_ also cache their own inputs internally; this module
146 // additionally caches these three for its own composed math, same choice
147 // MultiHeadAttentionModule made for its Q/K/V/scores caches.
148 Shape last_input_shape_ = Shape({0});
149 int64_t last_n_flat_ = 0;
150 Tensor last_gate_pre_;
151 Tensor last_gate_post_;
152 Tensor last_up_;
153 bool has_forwarded_ = false;
154};
155
156} // namespace pulsatrix
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
virtual DeviceType device() const noexcept=0
Which device this backend's buffers reside on.
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original u...
Definition linear_module.hpp:43
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
An N-dimensional shape. A plain aggregate of dimensions with no invariant beyond "non-negative dimens...
Definition shape.hpp:24
down_proj(silu(gate_proj(x)) * up_proj(x)), the gated feedforward block used in place of a plain two-...
Definition swiglu_module.hpp:53
std::vector< NamedModuleRef > named_children() override
This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(),...
OpType op_type() const override
Elementwise per this module's own op_type() note above.
Definition swiglu_module.hpp:86
LinearModule & up_proj()
Definition swiglu_module.hpp:115
std::optional< DeviceType > compute_device() const override
Where this layer computes, so forward() rejects an input on another device (FND-8).
Definition swiglu_module.hpp:121
int64_t d_ff() const
Definition swiglu_module.hpp:110
Tensor forward_impl(const Tensor &input) override
Runs: project (gate, up) -> silu(gate) -> gate*up -> project (down).
SwiGLUModule(int64_t d_model, int64_t d_ff, DeviceBackend *backend, bool use_bias=true, GatedActivation activation=GatedActivation::Silu)
Constructs a SwiGLU block with zero-initialized projections.
GatedActivation activation() const
Definition swiglu_module.hpp:70
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
Tensor backward(const Tensor &grad_output) override
Gradient w.r.t. this module's input; sub-module parameter gradients accumulate inside gate_proj_/up_p...
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
LRP relevance propagation: down_proj_'s epsilon rule, then the diagonal Eq. 15 split into gate/up sha...
int64_t d_model() const
Definition swiglu_module.hpp:109
std::vector< NamedBufferRef > named_buffers() override
Every layer's named_buffers(), prefixed the same way as named_parameters().
std::vector< NamedParamRef > named_parameters() override
gate_proj_'s, up_proj_'s, and down_proj_'s parameters, flattened.
LinearModule & gate_proj()
Definition swiglu_module.hpp:114
LinearModule & down_proj()
Definition swiglu_module.hpp:116
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
Dense/fully-connected layer – the reference Module implementation.
Abstract base every layer subclasses – NVI forward(), pure-virtual LRP contract.
Definition acquisition_functions.hpp:16
OpType
The op-type tag a Node carries. Charter Part 2 §3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
GatedActivation
The gate activation of a gated MLP: SiLU (SwiGLU: Llama, Qwen), GELU's tanh approximation (GeGLU: Gem...
Definition swiglu_module.hpp:16
@ Silu
x * sigmoid(x) – a.k.a. swish; the gate half of SwiGLU
@ GeluTanh
GELU, tanh approximation (gelu_pytorch_tanh) – the gate of Gemma's GeGLU (LLM-9)
@ Gelu
exact GELU, x * Phi(x) with erf – the MLP activation of BERT and ESM (PLM-1)
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72