25using DenoiserFn = std::function<std::vector<float>(
const std::vector<int64_t>& tokens)>;
73 std::optional<uint64_t>
seed;
92 std::vector<MaskedDiffusionWrite>
writes;
105 std::vector<MaskedDiffusionStep>
steps;
A Llama-family decoder-only language model built from a Hugging Face config, and loading its Hugging ...
Token embedding, num_hidden_layers pre-norm transformer blocks, a final RMSNorm and a language-model ...
Definition causal_lm.hpp:82
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
A masked language model: token embeddings (with ESM's token-dropout scaling, when the config has it),...
Definition encoder_lm.hpp:109
Masked-language-model encoders: ESM-2 protein language models (PLM-2), Hugging Face EsmForMaskedLM ch...
Definition acquisition_functions.hpp:16
UnmaskingSchedule
How the sampler chooses which masked positions to fill at each step.
Definition masked_diffusion.hpp:28
@ Remasking
DPLM's reparameterized decoding (reparam-uncond-deterministic-linear, its default): at step t of T th...
@ Confidence
LLaDA's low-confidence remasking: every step fills a fixed share of the positions that started masked...
@ Ancestral
MDLM's ancestral sampler (ddpm_cache): time runs linearly from 1 to 1e-5, and at each step every mask...
std::function< std::vector< float >(const std::vector< int64_t > &tokens)> DenoiserFn
The model as the sampler sees it: the current sequence (masked positions included) in,...
Definition masked_diffusion.hpp:25
MaskedDiffusionResult SampleMaskedDiffusion(const DenoiserFn &model, const std::vector< int64_t > &tokens, const MaskedDiffusionConfig &config)
Fills every masked position of tokens over the configured steps.
DenoiserFn MakeDenoiser(EncoderLM &model, DeviceBackend *backend)
A DenoiserFn for an EncoderLM: one forward pass over the sequence, (1, L) to logits (1,...
Sampling settings. The defaults fill greedily by confidence.
Definition masked_diffusion.hpp:59
bool do_sample
false: each filled position takes its most likely token. true: sample it.
Definition masked_diffusion.hpp:66
int64_t mask_token_id
The token that marks a masked position. Must be set.
Definition masked_diffusion.hpp:63
float temperature
Divides the logits before sampling. Must be positive.
Definition masked_diffusion.hpp:68
std::optional< uint64_t > seed
The seed for sampling and for the ancestral schedule's draws; unset draws one from the global stream ...
Definition masked_diffusion.hpp:73
std::vector< int64_t > suppressed_token_ids
Tokens never generated. The mask token never is (MDLM's SUBS parameterization).
Definition masked_diffusion.hpp:70
int64_t steps
Denoising steps. Must be positive.
Definition masked_diffusion.hpp:61
UnmaskingSchedule schedule
Definition masked_diffusion.hpp:64
A sampled sequence and how it was produced.
Definition masked_diffusion.hpp:99
std::vector< int64_t > tokens
The final sequence: no masked position remains.
Definition masked_diffusion.hpp:101
std::vector< MaskedDiffusionStep > steps
Every step that filled something (under Remasking, every step), in order. Each step's input is the pr...
Definition masked_diffusion.hpp:105
std::vector< int64_t > last_write_step
For each position, the index into steps of the step that last wrote it, or -1 for a position that was...
Definition masked_diffusion.hpp:110
One step that filled at least one position (under Remasking, any step).
Definition masked_diffusion.hpp:87
std::vector< MaskedDiffusionWrite > writes
The positions filled from the model's output, in position order. A position is only ever written whil...
Definition masked_diffusion.hpp:92
std::vector< int64_t > input
The sequence the model saw at this step.
Definition masked_diffusion.hpp:89
std::vector< int64_t > remasked
The positions that held a token in input and are masked again after this step, in position order (Rem...
Definition masked_diffusion.hpp:95
One position filled at one step.
Definition masked_diffusion.hpp:77
int64_t token
Definition masked_diffusion.hpp:79
float probability
The model's probability of the token (softmax of the raw logits, before any processing),...
Definition masked_diffusion.hpp:83
int64_t position
Definition masked_diffusion.hpp:78