25using NextTokenLogitsFn = std::function<std::vector<float>(
const std::vector<int64_t>& tokens)>;
51 std::optional<uint64_t>
seed;
83void ProcessLogits(std::vector<float>& logits,
const std::vector<int64_t>& tokens, int64_t num_new_tokens,
136 std::vector<TransformerBlock*> blocks,
138 int64_t max_length,
float embed_scale = 1.0f,
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
Embedding lookup table, rank-2 input (N, L) of float-encoded indices -> rank-3 output (N,...
Definition embedding_module.hpp:39
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
Lookup-table (row-select) layer – y = W[index], batched (N, L) -> (N, L, embedding_dim).
Abstract base every layer subclasses – NVI forward(), pure-virtual LRP contract.
Definition acquisition_functions.hpp:16
NextTokenLogitsFn MakeCachedNextTokenLogits(EmbeddingModule &embedding, std::vector< TransformerBlock * > blocks, std::vector< Module * > head, DeviceBackend *backend, int64_t max_length, float embed_scale=1.0f, BlockOutputHook after_block={})
A NextTokenLogitsFn for embedding -> transformer blocks -> head layers that keeps a KV cache per bloc...
NextTokenLogitsFn MakeNextTokenLogits(std::vector< Module * > layers, DeviceBackend *backend)
A NextTokenLogitsFn for a stack of pulsatrix modules that maps token ids (1, L) to logits (1,...
GenerationResult Generate(const NextTokenLogitsFn &model, const std::vector< int64_t > &prompt, const GenerationConfig &config={})
Generates up to max_new_tokens after prompt, one at a time: process the next-token logits,...
FinishReason
Why generation stopped.
Definition generation.hpp:55
std::function< std::vector< float >(const std::vector< int64_t > &tokens)> NextTokenLogitsFn
The model as generation sees it: the token sequence so far in, the logits for the next token out (one...
Definition generation.hpp:25
std::function< Tensor(int64_t layer, int64_t start, int64_t prompt_length, const Tensor &hidden)> BlockOutputHook
Called on each block's output in MakeCachedNextTokenLogits(): block layer's hidden states (1,...
Definition generation.hpp:115
void ProcessLogits(std::vector< float > &logits, const std::vector< int64_t > &tokens, int64_t num_new_tokens, const GenerationConfig &config)
Applies the configured logit processing, in Hugging Face's order: repetition penalty,...
Decoding settings. The defaults decode greedily. Field names follow Hugging Face's GenerationConfig.
Definition generation.hpp:29
int64_t max_new_tokens
Definition generation.hpp:30
int64_t top_k
Keep only the top_k most likely tokens; 0 keeps all.
Definition generation.hpp:40
bool do_sample
false: always the most likely token (greedy). true: sample.
Definition generation.hpp:36
float top_p
Keep the smallest set of most likely tokens whose probability reaches top_p; 1 keeps all.
Definition generation.hpp:43
int64_t min_new_tokens
EOS cannot be generated before this many new tokens.
Definition generation.hpp:32
float temperature
Divides the logits; below 1 sharpens, above 1 flattens. Must be positive.
Definition generation.hpp:38
float min_p
Drop tokens less likely than min_p times the most likely one; 0 keeps all.
Definition generation.hpp:45
std::optional< uint64_t > seed
The sampling seed; unset draws one from the global stream (next_seed(), FND-7), so generation is repr...
Definition generation.hpp:51
float repetition_penalty
CTRL's penalty on every token already in the sequence, prompt included: positive logits are divided b...
Definition generation.hpp:48
std::vector< int64_t > eos_token_ids
Generation stops after any of these is generated (it is kept in the output).
Definition generation.hpp:34
A generated continuation.
Definition generation.hpp:63
std::vector< int64_t > new_tokens
Only the new tokens, EOS included if generated.
Definition generation.hpp:67
std::vector< float > log_probs
Each new token's log-probability under the model itself (log-softmax of the raw logits,...
Definition generation.hpp:71
std::vector< int64_t > tokens
The prompt followed by the new tokens.
Definition generation.hpp:65
FinishReason finish_reason
Definition generation.hpp:72