pulsatrix 1.1.0
Loading...
Searching...
No Matches
feed_forward_module.hpp
Go to the documentation of this file.
1
6#pragma once
7
8#include <optional>
9
12#include "pulsatrix/module.hpp"
13
14namespace pulsatrix {
15
21class FeedForwardModule : public Module {
22public:
29 bool bias = true);
30
31 [[nodiscard]] Tensor backward(const Tensor& grad_output) override;
32 [[nodiscard]] Tensor propagate_relevance(const Tensor& relevance_out, const LRPRuleConfig& config) override;
34 [[nodiscard]] OpType op_type() const override { return OpType::Composite; }
35 [[nodiscard]] std::vector<NamedParamRef> named_parameters() override;
36 [[nodiscard]] std::vector<NamedModuleRef> named_children() override;
37 [[nodiscard]] std::vector<NamedBufferRef> named_buffers() override;
38 void set_training(bool training) override;
39 [[nodiscard]] std::optional<DeviceType> compute_device() const override { return backend_->device(); }
40
41 [[nodiscard]] LinearModule& fc1() { return fc1_; }
42 [[nodiscard]] ActivationModule& activation() { return act_; }
43 [[nodiscard]] LinearModule& fc2() { return fc2_; }
44
45 void release_activations() override;
46
47protected:
49 [[nodiscard]] Tensor forward_impl(const Tensor& input) override;
50
51private:
52 int64_t d_model_;
53 DeviceBackend* backend_;
54 LinearModule fc1_;
56 LinearModule fc2_;
57 Shape last_input_shape_ = Shape({0});
58 bool has_forwarded_ = false;
59};
60
61} // namespace pulsatrix
A pointwise activation chosen at construction: ReLU, tanh, sigmoid, SiLU, exact GELU or GELU's tanh a...
y = f(x), elementwise, for one of the backend's activations. No parameters.
Definition activation_module.hpp:19
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
virtual DeviceType device() const noexcept=0
Which device this backend's buffers reside on.
fc2(f(fc1(x))) over the last dimension. Shape (..., d_model) -> (..., d_model).
Definition feed_forward_module.hpp:21
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
void set_training(bool training) override
Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Mod...
LinearModule & fc2()
Definition feed_forward_module.hpp:43
LinearModule & fc1()
Definition feed_forward_module.hpp:41
ActivationModule & activation()
Definition feed_forward_module.hpp:42
std::vector< NamedModuleRef > named_children() override
This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(),...
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
Computes this module's contribution to LRP relevance propagation.
OpType op_type() const override
A chain of other modules.
Definition feed_forward_module.hpp:34
Tensor backward(const Tensor &grad_output) override
Computes the gradient w.r.t. this module's input, given the gradient w.r.t. its output....
std::vector< NamedParamRef > named_parameters() override
This module's trainable parameters, each with its hierarchical name – the one place a module declares...
Tensor forward_impl(const Tensor &input) override
FeedForwardModule(int64_t d_model, int64_t d_ff, DeviceBackend *backend, ElementwiseOp activation=ElementwiseOp::Gelu, bool bias=true)
std::optional< DeviceType > compute_device() const override
The device this module computes on, so forward() can reject an input on another device before any ker...
Definition feed_forward_module.hpp:39
std::vector< NamedBufferRef > named_buffers() override
This module's buffers: state a checkpoint must save that no optimizer updates.
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original u...
Definition linear_module.hpp:43
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
An N-dimensional shape. A plain aggregate of dimensions with no invariant beyond "non-negative dimens...
Definition shape.hpp:24
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
Dense/fully-connected layer – the reference Module implementation.
Abstract base every layer subclasses – NVI forward(), pure-virtual LRP contract.
Definition acquisition_functions.hpp:16
OpType
The op-type tag a Node carries. Charter Part 2 §3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
ElementwiseOp
Unary elementwise operations supported by DeviceBackend::elementwise().
Definition device_backend.hpp:43
@ Gelu
exact GELU, x * Phi(x) with erf – the MLP activation of BERT and ESM (PLM-1)
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72