pulsatrix 1.1.0
Loading...
Searching...
No Matches
tied_lm_head_module.hpp
Go to the documentation of this file.
1
5#pragma once
6
7#include <optional>
8#include <vector>
9
11#include "pulsatrix/module.hpp"
12
13namespace pulsatrix {
14
32class TiedLMHeadModule : public Module {
33public:
39
46 [[nodiscard]] Tensor backward(const Tensor& grad_output) override;
47
54 [[nodiscard]] Tensor propagate_relevance(const Tensor& relevance_out, const LRPRuleConfig& config) override;
55
56 [[nodiscard]] bool supports_lrp_rule(LRPRule) const override { return true; }
57 [[nodiscard]] OpType op_type() const override { return OpType::Linear; }
58
60 [[nodiscard]] std::vector<NamedParamRef> named_parameters() override { return {}; }
61
62 [[nodiscard]] int64_t vocab_size() const { return weight_->shape().dim(0); }
63 [[nodiscard]] int64_t d_model() const { return weight_->shape().dim(1); }
64
65 [[nodiscard]] std::optional<DeviceType> compute_device() const override { return backend_->device(); }
66
67 void release_activations() override;
68
69protected:
75 [[nodiscard]] Tensor forward_impl(const Tensor& input) override;
76
77private:
78 DeviceBackend* backend_;
79 // The embedding's own weight and gradient members. Their addresses never change (set_weight
80 // assigns into them), so the head always sees the current table.
81 Tensor* weight_;
82 Tensor* weight_grad_;
83 Tensor zero_bias_;
84 Tensor last_input_;
85 Tensor last_logits_;
86 Shape last_input_shape_ = Shape({0});
87 Shape last_output_shape_ = Shape({0});
88 bool has_forwarded_ = false;
89};
90
91} // namespace pulsatrix
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
virtual DeviceType device() const noexcept=0
Which device this backend's buffers reside on.
Embedding lookup table, rank-2 input (N, L) of float-encoded indices -> rank-3 output (N,...
Definition embedding_module.hpp:39
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
An N-dimensional shape. A plain aggregate of dimensions with no invariant beyond "non-negative dimens...
Definition shape.hpp:24
int64_t dim(size_t index) const
Size of a single dimension.
Definition shape.hpp:93
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) โ€“ th...
Definition tensor.hpp:29
const Shape & shape() const
This tensor's shape.
Definition tensor.hpp:113
logits = x @ E^T, where E is a token embedding table (vocab, d_model) that the head shares rather tha...
Definition tied_lm_head_module.hpp:32
std::vector< NamedParamRef > named_parameters() override
Empty: the shared table is the embedding's parameter, not the head's.
Definition tied_lm_head_module.hpp:60
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
std::optional< DeviceType > compute_device() const override
The device this module computes on, so forward() can reject an input on another device before any ker...
Definition tied_lm_head_module.hpp:65
TiedLMHeadModule(EmbeddingModule &embedding, DeviceBackend *backend)
bool supports_lrp_rule(LRPRule) const override
Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::...
Definition tied_lm_head_module.hpp:56
int64_t vocab_size() const
Definition tied_lm_head_module.hpp:62
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
LRP through x @ E^T, by config.rule (Epsilon, Gamma, AlphaBeta or ZBox).
OpType op_type() const override
This module's operation-category tag, for ComputationGraph node tagging.
Definition tied_lm_head_module.hpp:57
Tensor backward(const Tensor &grad_output) override
Input gradient grad @ E; adds grad^T @ x into the embedding's weight gradient unless the embedding is...
Tensor forward_impl(const Tensor &input) override
int64_t d_model() const
Definition tied_lm_head_module.hpp:63
Lookup-table (row-select) layer โ€“ y = W[index], batched (N, L) -> (N, L, embedding_dim).
Abstract base every layer subclasses โ€“ NVI forward(), pure-virtual LRP contract.
Definition acquisition_functions.hpp:16
OpType
The op-type tag a Node carries. Charter Part 2 ยง3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
LRPRule
The LRP rule family a module applies. Semantics follow Zennit 1.0.0 exactly (Anders et al....
Definition lrp_rule_config.hpp:21
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72