pulsatrix 1.1.0
Loading...
Searching...
No Matches
tagger_finetune_example.hpp
Go to the documentation of this file.
1
12#pragma once
13
14#include <cstdint>
15#include <string>
16#include <vector>
17
20#include "pulsatrix/module.hpp"
22
23namespace pulsatrix {
24
26class TinyTagger : public Module {
27public:
28 static constexpr int64_t kVocab = 8;
29 static constexpr int64_t kSeqLen = 8;
30 static constexpr int64_t kClasses = 4;
31 static constexpr int64_t kDim = 16;
32 static constexpr int64_t kHeads = 2;
33 static constexpr int64_t kFeedForward = 32;
34
35 explicit TinyTagger(DeviceBackend* backend);
36
37 [[nodiscard]] Tensor backward(const Tensor& grad_output) override;
38 [[nodiscard]] Tensor propagate_relevance(const Tensor& relevance_out, const LRPRuleConfig& config) override;
39 [[nodiscard]] OpType op_type() const override { return OpType::Composite; }
40 [[nodiscard]] std::vector<NamedParamRef> named_parameters() override;
41 [[nodiscard]] std::vector<NamedModuleRef> named_children() override;
42 [[nodiscard]] std::vector<NamedBufferRef> named_buffers() override;
43 void set_training(bool training) override;
44
45protected:
46 [[nodiscard]] Tensor forward_impl(const Tensor& input) override;
47
48private:
49 EmbeddingModule embed_;
50 TransformerBlock block_;
51 LinearModule head_;
52 int64_t last_n_ = 0;
53};
54
57void InitTagger(TinyTagger& tagger, uint64_t seed);
58
62
68
70[[nodiscard]] TaggingBatch MakeTaggingBatch(TaggingRule rule, int64_t batch_size, uint64_t seed,
71 DeviceBackend* backend = nullptr);
72
75 int64_t steps = 300;
76 int64_t micro_batches = 2;
77 int64_t micro_batch_size = 8;
78 float learning_rate = 1e-2f;
79 int64_t warmup = 10;
80 float weight_decay = 0.01f;
81 float max_grad_norm = 1.0f;
82 uint64_t data_seed = 1000;
83 int64_t stop_after = -1;
84 std::string checkpoint_path;
85 std::string resume_from;
86};
87
96[[nodiscard]] std::vector<float> TrainTagger(TinyTagger& tagger, TaggingRule rule, const FineTuneConfig& config,
97 DeviceBackend* backend);
98
100[[nodiscard]] float TaggingAccuracy(TinyTagger& tagger, TaggingRule rule, int64_t num_sequences);
101
102} // namespace pulsatrix
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
Embedding lookup table, rank-2 input (N, L) of float-encoded indices -> rank-3 output (N,...
Definition embedding_module.hpp:39
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original u...
Definition linear_module.hpp:43
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
Embedding -> TransformerBlock -> per-token Linear head: (N, L) tokens to (N, L, C) logits.
Definition tagger_finetune_example.hpp:26
static constexpr int64_t kVocab
token 0 is padding
Definition tagger_finetune_example.hpp:28
static constexpr int64_t kSeqLen
Definition tagger_finetune_example.hpp:29
Tensor forward_impl(const Tensor &input) override
The actual forward computation. Called by forward() after precondition checks.
std::vector< NamedParamRef > named_parameters() override
This module's trainable parameters, each with its hierarchical name – the one place a module declares...
OpType op_type() const override
This module's operation-category tag, for ComputationGraph node tagging.
Definition tagger_finetune_example.hpp:39
std::vector< NamedBufferRef > named_buffers() override
This module's buffers: state a checkpoint must save that no optimizer updates.
static constexpr int64_t kFeedForward
Definition tagger_finetune_example.hpp:33
std::vector< NamedModuleRef > named_children() override
This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(),...
TinyTagger(DeviceBackend *backend)
void set_training(bool training) override
Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Mod...
static constexpr int64_t kHeads
Definition tagger_finetune_example.hpp:32
Tensor backward(const Tensor &grad_output) override
Computes the gradient w.r.t. this module's input, given the gradient w.r.t. its output....
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
Computes this module's contribution to LRP relevance propagation.
static constexpr int64_t kClasses
Definition tagger_finetune_example.hpp:30
static constexpr int64_t kDim
Definition tagger_finetune_example.hpp:31
y1 = x + MHA(RMSNorm(x)), y2 = y1 + SwiGLU(RMSNorm(y1)). Shape (N, L, d_model) -> (N,...
Definition transformer_block.hpp:67
Lookup-table (row-select) layer – y = W[index], batched (N, L) -> (N, L, embedding_dim).
Dense/fully-connected layer – the reference Module implementation.
Abstract base every layer subclasses – NVI forward(), pure-virtual LRP contract.
Definition acquisition_functions.hpp:16
float TaggingAccuracy(TinyTagger &tagger, TaggingRule rule, int64_t num_sequences)
Fraction of real tokens tagged correctly on num_sequences fresh sequences.
void InitTagger(TinyTagger &tagger, uint64_t seed)
Fills the tagger's parameters reproducibly on every platform: matrices uniform in +-1/sqrt(fan_in),...
std::vector< float > TrainTagger(TinyTagger &tagger, TaggingRule rule, const FineTuneConfig &config, DeviceBackend *backend)
Trains every parameter of tagger on rule with the whole v1.1 training stack: AdamW with parameter gro...
OpType
The op-type tag a Node carries. Charter Part 2 §3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
TaggingRule
How a token's tag follows from it and the token before it (the first token's predecessor is 0)....
Definition tagger_finetune_example.hpp:61
TaggingBatch MakeTaggingBatch(TaggingRule rule, int64_t batch_size, uint64_t seed, DeviceBackend *backend=nullptr)
A deterministic, platform-independent batch; lengths are 5 to kSeqLen.
Training settings. The defaults are the recipe's.
Definition tagger_finetune_example.hpp:74
int64_t micro_batch_size
Definition tagger_finetune_example.hpp:77
float max_grad_norm
(TRN-3)
Definition tagger_finetune_example.hpp:81
int64_t stop_after
stop after this many steps and checkpoint to checkpoint_path
Definition tagger_finetune_example.hpp:83
std::string resume_from
resume model, optimizer and schedule from this checkpoint (IO-2)
Definition tagger_finetune_example.hpp:85
float weight_decay
AdamW, none on biases and norms (TRN-1, TRN-2)
Definition tagger_finetune_example.hpp:80
float learning_rate
Definition tagger_finetune_example.hpp:78
int64_t steps
Definition tagger_finetune_example.hpp:75
uint64_t data_seed
Definition tagger_finetune_example.hpp:82
int64_t micro_batches
accumulated per optimizer step (TRN-5)
Definition tagger_finetune_example.hpp:76
int64_t warmup
then cosine decay to step steps (TRN-4)
Definition tagger_finetune_example.hpp:79
std::string checkpoint_path
Definition tagger_finetune_example.hpp:84
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72
Ragged sequences: real tokens 1..kVocab-1 followed by padding (input 0, target -100).
Definition tagger_finetune_example.hpp:64
Tensor inputs
(N, kSeqLen) token ids as whole-number floats
Definition tagger_finetune_example.hpp:65
Tensor targets
(N, kSeqLen) tags, -100 at padding
Definition tagger_finetune_example.hpp:66
Pre-LN transformer block – Phase 3's literal exit-gate deliverable, third and last composition missio...