Text generation from a language model: greedy decoding and seeded sampling with temperature, top-k, top-p and min-p, repetition penalty, and EOS handling (LLM-4).
More...
|
| using | pulsatrix::NextTokenLogitsFn = std::function< std::vector< float >(const std::vector< int64_t > &tokens)> |
| | The model as generation sees it: the token sequence so far in, the logits for the next token out (one per vocabulary entry). How they're computed is up to the caller: a full forward pass over the sequence (MakeNextTokenLogits), or a cached step (LLM-5).
|
| |
| using | pulsatrix::BlockOutputHook = std::function< Tensor(int64_t layer, int64_t start, int64_t prompt_length, const Tensor &hidden)> |
| | Called on each block's output in MakeCachedNextTokenLogits(): block layer's hidden states (1, L, d) for positions [start, start + L), in a sequence whose prompt is prompt_length long; returns what the next block reads. CausalLM uses it to run its interventions (TRN-9).
|
| |
|
| void | pulsatrix::ProcessLogits (std::vector< float > &logits, const std::vector< int64_t > &tokens, int64_t num_new_tokens, const GenerationConfig &config) |
| | Applies the configured logit processing, in Hugging Face's order: repetition penalty, the EOS ban before min_new_tokens, then (when sampling) temperature, top-k, top-p and min-p. Removed tokens get -infinity. With do_sample false only the first two apply.
|
| |
| GenerationResult | pulsatrix::Generate (const NextTokenLogitsFn &model, const std::vector< int64_t > &prompt, const GenerationConfig &config={}) |
| | Generates up to max_new_tokens after prompt, one at a time: process the next-token logits, take the most likely token (greedy) or draw one from their softmax (sampling), append it, and stop after an EOS token.
|
| |
| NextTokenLogitsFn | pulsatrix::MakeNextTokenLogits (std::vector< Module * > layers, DeviceBackend *backend) |
| | A NextTokenLogitsFn for a stack of pulsatrix modules that maps token ids (1, L) to logits (1, L, vocab) (for example EmbeddingModule, causal TransformerBlocks, an RMSNorm and a TiedLMHeadModule): it runs the whole sequence through the modules and returns the last position's logits.
|
| |
| NextTokenLogitsFn | pulsatrix::MakeCachedNextTokenLogits (EmbeddingModule &embedding, std::vector< TransformerBlock * > blocks, std::vector< Module * > head, DeviceBackend *backend, int64_t max_length, float embed_scale=1.0f, BlockOutputHook after_block={}) |
| | A NextTokenLogitsFn for embedding -> transformer blocks -> head layers that keeps a KV cache per block (LLM-5): each call runs only the tokens after the longest prefix it has already processed, so generating n tokens costs one pass over the prompt and then one position per token, instead of n passes over the whole sequence. A call with a different history (a new prompt, or a shorter or edited one) truncates the caches to the shared prefix and continues from there.
|
| |
Text generation from a language model: greedy decoding and seeded sampling with temperature, top-k, top-p and min-p, repetition penalty, and EOS handling (LLM-4).