pulsatrix 1.1.0
Loading...
Searching...
No Matches
generation.hpp
Go to the documentation of this file.
1
6#pragma once
7
8#include <cstdint>
9#include <functional>
10#include <limits>
11#include <optional>
12#include <vector>
13
15#include "pulsatrix/module.hpp"
17
18namespace pulsatrix {
19
25using NextTokenLogitsFn = std::function<std::vector<float>(const std::vector<int64_t>& tokens)>;
26
30 int64_t max_new_tokens = 32;
32 int64_t min_new_tokens = 0;
34 std::vector<int64_t> eos_token_ids;
36 bool do_sample = false;
38 float temperature = 1.0f;
40 int64_t top_k = 0;
43 float top_p = 1.0f;
45 float min_p = 0.0f;
48 float repetition_penalty = 1.0f;
51 std::optional<uint64_t> seed;
52};
53
55enum class FinishReason {
57 Eos,
59 Length,
60};
61
65 std::vector<int64_t> tokens;
67 std::vector<int64_t> new_tokens;
71 std::vector<float> log_probs;
73};
74
83void ProcessLogits(std::vector<float>& logits, const std::vector<int64_t>& tokens, int64_t num_new_tokens,
84 const GenerationConfig& config);
85
95[[nodiscard]] GenerationResult Generate(const NextTokenLogitsFn& model, const std::vector<int64_t>& prompt,
96 const GenerationConfig& config = {});
97
108[[nodiscard]] NextTokenLogitsFn MakeNextTokenLogits(std::vector<Module*> layers, DeviceBackend* backend);
109
115using BlockOutputHook = std::function<Tensor(int64_t layer, int64_t start, int64_t prompt_length, const Tensor& hidden)>;
116
136 std::vector<TransformerBlock*> blocks,
137 std::vector<Module*> head, DeviceBackend* backend,
138 int64_t max_length, float embed_scale = 1.0f,
139 BlockOutputHook after_block = {});
140
141} // namespace pulsatrix
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
Embedding lookup table, rank-2 input (N, L) of float-encoded indices -> rank-3 output (N,...
Definition embedding_module.hpp:39
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
Lookup-table (row-select) layer – y = W[index], batched (N, L) -> (N, L, embedding_dim).
Abstract base every layer subclasses – NVI forward(), pure-virtual LRP contract.
Definition acquisition_functions.hpp:16
NextTokenLogitsFn MakeCachedNextTokenLogits(EmbeddingModule &embedding, std::vector< TransformerBlock * > blocks, std::vector< Module * > head, DeviceBackend *backend, int64_t max_length, float embed_scale=1.0f, BlockOutputHook after_block={})
A NextTokenLogitsFn for embedding -> transformer blocks -> head layers that keeps a KV cache per bloc...
NextTokenLogitsFn MakeNextTokenLogits(std::vector< Module * > layers, DeviceBackend *backend)
A NextTokenLogitsFn for a stack of pulsatrix modules that maps token ids (1, L) to logits (1,...
GenerationResult Generate(const NextTokenLogitsFn &model, const std::vector< int64_t > &prompt, const GenerationConfig &config={})
Generates up to max_new_tokens after prompt, one at a time: process the next-token logits,...
FinishReason
Why generation stopped.
Definition generation.hpp:55
std::function< std::vector< float >(const std::vector< int64_t > &tokens)> NextTokenLogitsFn
The model as generation sees it: the token sequence so far in, the logits for the next token out (one...
Definition generation.hpp:25
std::function< Tensor(int64_t layer, int64_t start, int64_t prompt_length, const Tensor &hidden)> BlockOutputHook
Called on each block's output in MakeCachedNextTokenLogits(): block layer's hidden states (1,...
Definition generation.hpp:115
void ProcessLogits(std::vector< float > &logits, const std::vector< int64_t > &tokens, int64_t num_new_tokens, const GenerationConfig &config)
Applies the configured logit processing, in Hugging Face's order: repetition penalty,...
Decoding settings. The defaults decode greedily. Field names follow Hugging Face's GenerationConfig.
Definition generation.hpp:29
int64_t max_new_tokens
Definition generation.hpp:30
int64_t top_k
Keep only the top_k most likely tokens; 0 keeps all.
Definition generation.hpp:40
bool do_sample
false: always the most likely token (greedy). true: sample.
Definition generation.hpp:36
float top_p
Keep the smallest set of most likely tokens whose probability reaches top_p; 1 keeps all.
Definition generation.hpp:43
int64_t min_new_tokens
EOS cannot be generated before this many new tokens.
Definition generation.hpp:32
float temperature
Divides the logits; below 1 sharpens, above 1 flattens. Must be positive.
Definition generation.hpp:38
float min_p
Drop tokens less likely than min_p times the most likely one; 0 keeps all.
Definition generation.hpp:45
std::optional< uint64_t > seed
The sampling seed; unset draws one from the global stream (next_seed(), FND-7), so generation is repr...
Definition generation.hpp:51
float repetition_penalty
CTRL's penalty on every token already in the sequence, prompt included: positive logits are divided b...
Definition generation.hpp:48
std::vector< int64_t > eos_token_ids
Generation stops after any of these is generated (it is kept in the output).
Definition generation.hpp:34
A generated continuation.
Definition generation.hpp:63
std::vector< int64_t > new_tokens
Only the new tokens, EOS included if generated.
Definition generation.hpp:67
std::vector< float > log_probs
Each new token's log-probability under the model itself (log-softmax of the raw logits,...
Definition generation.hpp:71
std::vector< int64_t > tokens
The prompt followed by the new tokens.
Definition generation.hpp:65
FinishReason finish_reason
Definition generation.hpp:72
Pre-LN transformer block – Phase 3's literal exit-gate deliverable, third and last composition missio...