pulsatrix 1.1.0
Loading...
Searching...
No Matches
lm_scoring.hpp File Reference

Prompt/completion data for a causal language model (LLM-10): read it from JSONL, batch it with the prompt masked out of the loss, and score it, as log P(completion | prompt) or as the probabilities of a few single-token choices ("P(True)"). More...

#include <cstdint>
#include <string>
#include <vector>
#include "pulsatrix/causal_lm.hpp"
#include "pulsatrix/json.hpp"
#include "pulsatrix/tensor.hpp"
#include "pulsatrix/text_tokenizer.hpp"
Include dependency graph for lm_scoring.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::PromptCompletion
 One example: the prompt's token ids, then the completion's. More...
 
struct  pulsatrix::CausalLmBatch
 A right-padded training batch. More...
 
struct  pulsatrix::ChoiceScores
 Scores of single-token choices after each prompt. Indexed [example][choice]. More...
 

Namespaces

namespace  pulsatrix
 

Functions

std::vector< JsonValue > pulsatrix::ReadJsonl (const std::string &path)
 Reads a JSONL file: one JSON value a line. Blank lines are skipped.
 
std::vector< PromptCompletion > pulsatrix::TokenizePromptCompletions (const std::vector< JsonValue > &rows, const TextTokenizer &tokenizer, const std::string &prompt_key="prompt", const std::string &completion_key="completion", bool add_special_tokens=false)
 Tokenizes each row's prompt and completion strings (objects with those keys). The two are tokenized separately, so a completion like " Yes" keeps its own tokens whatever the prompt ends with.
 
CausalLmBatch pulsatrix::CollatePromptCompletions (const std::vector< PromptCompletion > &examples, DeviceBackend *backend, int64_t pad_id=0)
 Builds a batch for TokenCrossEntropyLoss: position i predicts token i + 1, and only the completion's tokens count. The completion's last token is never an input (nothing follows it), so each row is one token shorter than prompt + completion.
 
std::vector< double > pulsatrix::CompletionLogProbs (CausalLM &model, const std::vector< PromptCompletion > &examples, DeviceBackend *backend, int64_t batch_size=8)
 log P(completion | prompt) under the model, summed over the completion's tokens, in double. Runs forward passes only, batch_size examples at a time, grouped by length.
 
ChoiceScores pulsatrix::ScoreChoices (CausalLM &model, const std::vector< std::vector< int64_t > > &prompts, const std::vector< int64_t > &choice_tokens, DeviceBackend *backend, int64_t batch_size=8)
 Scores the next token after each prompt against a fixed set of single-token choices, such as " Yes" and " No". For two choices, choice_log_probs[n][0] - choice_log_probs[n][1] is the log-odds of the first.
 
int64_t pulsatrix::SingleTokenId (const TextTokenizer &tokenizer, const std::string &text)
 The id of a string that must tokenize to exactly one token, such as " Yes".
 

Detailed Description

Prompt/completion data for a causal language model (LLM-10): read it from JSONL, batch it with the prompt masked out of the loss, and score it, as log P(completion | prompt) or as the probabilities of a few single-token choices ("P(True)").

Fine-tuning a model to answer a question trains only the answer: CollatePromptCompletions() sets every target outside the completion to TokenCrossEntropyLoss::kIgnoreIndex, so the loss is the completion's log loss and nothing else. Scoring reads the same positions back.

Batches are right-padded. Causal attention never lets a real token attend to the padding after it, so padding changes no real position's logits (tests/lm_scoring_test.cpp checks a padded batch against each example alone).

The scorers copy only the logit rows they need to the host, gathered on the device: a batch's full logits are (N, L, vocab), about 400 MB for SmolLM2 at N = 8, L = 256.