pulsatrix 1.1.0
Loading...
Searching...
No Matches
tagger_finetune_example.hpp File Reference

A full fine-tuning recipe (roadmap TRN-6): pretrain a tiny transformer tagger on one rule, save it, reload it and fine-tune every parameter on a related rule. More...

#include <cstdint>
#include <string>
#include <vector>
#include "pulsatrix/embedding_module.hpp"
#include "pulsatrix/linear_module.hpp"
#include "pulsatrix/module.hpp"
#include "pulsatrix/transformer_block.hpp"
Include dependency graph for tagger_finetune_example.hpp:

Go to the source code of this file.

Classes

class  pulsatrix::TinyTagger
 Embedding -> TransformerBlock -> per-token Linear head: (N, L) tokens to (N, L, C) logits. More...
 
struct  pulsatrix::TaggingBatch
 Ragged sequences: real tokens 1..kVocab-1 followed by padding (input 0, target -100). More...
 
struct  pulsatrix::FineTuneConfig
 Training settings. The defaults are the recipe's. More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::TaggingRule { pulsatrix::SumWithPrevious , pulsatrix::DifferenceWithPrevious }
 How a token's tag follows from it and the token before it (the first token's predecessor is 0). Tags are modulo kClasses. More...
 

Functions

void pulsatrix::InitTagger (TinyTagger &tagger, uint64_t seed)
 Fills the tagger's parameters reproducibly on every platform: matrices uniform in +-1/sqrt(fan_in), normalization gains 1, biases 0.
 
TaggingBatch pulsatrix::MakeTaggingBatch (TaggingRule rule, int64_t batch_size, uint64_t seed, DeviceBackend *backend=nullptr)
 A deterministic, platform-independent batch; lengths are 5 to kSeqLen.
 
std::vector< float > pulsatrix::TrainTagger (TinyTagger &tagger, TaggingRule rule, const FineTuneConfig &config, DeviceBackend *backend)
 Trains every parameter of tagger on rule with the whole v1.1 training stack: AdamW with parameter groups, warmup plus cosine, gradient accumulation normalized by the window's token count, and gradient clipping.
 
float pulsatrix::TaggingAccuracy (TinyTagger &tagger, TaggingRule rule, int64_t num_sequences)
 Fraction of real tokens tagged correctly on num_sequences fresh sequences.
 

Detailed Description

A full fine-tuning recipe (roadmap TRN-6): pretrain a tiny transformer tagger on one rule, save it, reload it and fine-tune every parameter on a related rule.

Note
Real pretrained models (SmolLM2, ResNet18) arrive with v1.2's import work (IO-3 to IO-5). Until then the "pretrained" model is one this recipe trains itself; the fine-tuning half is unchanged when a real checkpoint replaces it.
It is per-token tagging rather than next-token prediction because attention has no causal mask yet; with bidirectional attention a language-model target would be visible in the input.