41[[nodiscard]] std::vector<JsonValue>
ReadJsonl(
const std::string& path);
54 const std::string& prompt_key =
"prompt",
55 const std::string& completion_key =
"completion",
56 bool add_special_tokens =
false);
99 const std::vector<int64_t>& choice_tokens,
DeviceBackend* backend,
100 int64_t batch_size = 8);
A Llama-family decoder-only language model built from a Hugging Face config, and loading its Hugging ...
Token embedding, num_hidden_layers pre-norm transformer blocks, a final RMSNorm and a language-model ...
Definition causal_lm.hpp:82
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each r...
Definition text_tokenizer.hpp:166
A small strict JSON value, parser and deterministic writer (RFC 8259).
Definition acquisition_functions.hpp:16
ChoiceScores ScoreChoices(CausalLM &model, const std::vector< std::vector< int64_t > > &prompts, const std::vector< int64_t > &choice_tokens, DeviceBackend *backend, int64_t batch_size=8)
Scores the next token after each prompt against a fixed set of single-token choices,...
int64_t SingleTokenId(const TextTokenizer &tokenizer, const std::string &text)
The id of a string that must tokenize to exactly one token, such as " Yes".
std::vector< JsonValue > ReadJsonl(const std::string &path)
Reads a JSONL file: one JSON value a line. Blank lines are skipped.
std::vector< double > CompletionLogProbs(CausalLM &model, const std::vector< PromptCompletion > &examples, DeviceBackend *backend, int64_t batch_size=8)
log P(completion | prompt) under the model, summed over the completion's tokens, in double....
std::vector< PromptCompletion > TokenizePromptCompletions(const std::vector< JsonValue > &rows, const TextTokenizer &tokenizer, const std::string &prompt_key="prompt", const std::string &completion_key="completion", bool add_special_tokens=false)
Tokenizes each row's prompt and completion strings (objects with those keys). The two are tokenized s...
CausalLmBatch CollatePromptCompletions(const std::vector< PromptCompletion > &examples, DeviceBackend *backend, int64_t pad_id=0)
Builds a batch for TokenCrossEntropyLoss: position i predicts token i + 1, and only the completion's ...
A right-padded training batch.
Definition lm_scoring.hpp:59
std::vector< int64_t > lengths
each row's real length, |prompt| + |completion| - 1
Definition lm_scoring.hpp:62
int64_t target_tokens
completion tokens in the batch, the loss's token count
Definition lm_scoring.hpp:63
Tensor targets
(N, L): the next token where it is a completion token, else kIgnoreIndex
Definition lm_scoring.hpp:61
Tensor ids
(N, L) token ids as floats: prompt, then completion but its last token
Definition lm_scoring.hpp:60
Scores of single-token choices after each prompt. Indexed [example][choice].
Definition lm_scoring.hpp:85
std::vector< double > choice_mass
the probability the choices take together
Definition lm_scoring.hpp:88
std::vector< std::vector< double > > choice_log_probs
the same, renormalized over the choices
Definition lm_scoring.hpp:87
std::vector< std::vector< double > > log_probs
log P(choice | prompt) over the whole vocabulary
Definition lm_scoring.hpp:86
One example: the prompt's token ids, then the completion's.
Definition lm_scoring.hpp:32
std::vector< int64_t > prompt
Definition lm_scoring.hpp:33
std::vector< int64_t > completion
Definition lm_scoring.hpp:34
N-dimensional tensor – owns a buffer via DeviceBackend*, RAII (Rule of Five).
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...