pulsatrix 1.1.0
Loading...
Searching...
No Matches
generation.hpp File Reference

Text generation from a language model: greedy decoding and seeded sampling with temperature, top-k, top-p and min-p, repetition penalty, and EOS handling (LLM-4). More...

#include <cstdint>
#include <functional>
#include <limits>
#include <optional>
#include <vector>
#include "pulsatrix/embedding_module.hpp"
#include "pulsatrix/module.hpp"
#include "pulsatrix/transformer_block.hpp"
Include dependency graph for generation.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

struct  pulsatrix::GenerationConfig
 Decoding settings. The defaults decode greedily. Field names follow Hugging Face's GenerationConfig. More...
 
struct  pulsatrix::GenerationResult
 A generated continuation. More...
 

Namespaces

namespace  pulsatrix
 

Typedefs

using pulsatrix::NextTokenLogitsFn = std::function< std::vector< float >(const std::vector< int64_t > &tokens)>
 The model as generation sees it: the token sequence so far in, the logits for the next token out (one per vocabulary entry). How they're computed is up to the caller: a full forward pass over the sequence (MakeNextTokenLogits), or a cached step (LLM-5).
 
using pulsatrix::BlockOutputHook = std::function< Tensor(int64_t layer, int64_t start, int64_t prompt_length, const Tensor &hidden)>
 Called on each block's output in MakeCachedNextTokenLogits(): block layer's hidden states (1, L, d) for positions [start, start + L), in a sequence whose prompt is prompt_length long; returns what the next block reads. CausalLM uses it to run its interventions (TRN-9).
 

Enumerations

enum class  pulsatrix::FinishReason { pulsatrix::Eos , pulsatrix::Length }
 Why generation stopped. More...
 

Functions

void pulsatrix::ProcessLogits (std::vector< float > &logits, const std::vector< int64_t > &tokens, int64_t num_new_tokens, const GenerationConfig &config)
 Applies the configured logit processing, in Hugging Face's order: repetition penalty, the EOS ban before min_new_tokens, then (when sampling) temperature, top-k, top-p and min-p. Removed tokens get -infinity. With do_sample false only the first two apply.
 
GenerationResult pulsatrix::Generate (const NextTokenLogitsFn &model, const std::vector< int64_t > &prompt, const GenerationConfig &config={})
 Generates up to max_new_tokens after prompt, one at a time: process the next-token logits, take the most likely token (greedy) or draw one from their softmax (sampling), append it, and stop after an EOS token.
 
NextTokenLogitsFn pulsatrix::MakeNextTokenLogits (std::vector< Module * > layers, DeviceBackend *backend)
 A NextTokenLogitsFn for a stack of pulsatrix modules that maps token ids (1, L) to logits (1, L, vocab) (for example EmbeddingModule, causal TransformerBlocks, an RMSNorm and a TiedLMHeadModule): it runs the whole sequence through the modules and returns the last position's logits.
 
NextTokenLogitsFn pulsatrix::MakeCachedNextTokenLogits (EmbeddingModule &embedding, std::vector< TransformerBlock * > blocks, std::vector< Module * > head, DeviceBackend *backend, int64_t max_length, float embed_scale=1.0f, BlockOutputHook after_block={})
 A NextTokenLogitsFn for embedding -> transformer blocks -> head layers that keeps a KV cache per block (LLM-5): each call runs only the tokens after the longest prefix it has already processed, so generating n tokens costs one pass over the prompt and then one position per token, instead of n passes over the whole sequence. A call with a different history (a new prompt, or a shorter or edited one) truncates the caches to the shared prefix and continues from there.
 

Detailed Description

Text generation from a language model: greedy decoding and seeded sampling with temperature, top-k, top-p and min-p, repetition penalty, and EOS handling (LLM-4).