|
pulsatrix 1.1.0
|
Dataset/IterableDataset/DataLoader/Transform/Compose/CollateFn core, plus concrete modality implementations (tabular CSV, image, text, audio, a reduced-scope video frame-directory stub) and generic dataset validation (descriptive statistics, missingness/outlier detection), all built on the Deep Learning Modules and Layers Tensor. More...
Files | |
| file | audio_collate.hpp |
| AudioPadCollate – zero-pads variable-length waveforms into one batch Tensor. | |
| file | audio_folder_dataset.hpp |
| Directory-of-class-subfolders audio Dataset, mirroring ImageFolderDataset. | |
| file | audio_transforms.hpp |
| Sample-level audio Transforms – linear-interpolation resampling. | |
| file | bounded_queue.hpp |
| Fixed-capacity thread-safe blocking queue – the staged-pipeline backbone. | |
| file | byte_level_bpe.hpp |
The components byte-level BPE tokenizers are built from (TOK-2): NFC, regex and digit splitting, byte-level mapping, BPE, template post-processing and byte-level decoding, each following Hugging Face tokenizers. | |
| file | collate.hpp |
| Batch assembly – Batch, CollateFn, DefaultCollate. | |
| file | csv_dataset.hpp |
| Dataset over a CSV file's numeric feature/label columns – Phase 1's tabular reference case. | |
| file | csv_reader.hpp |
| Minimal hand-rolled CSV parser – RFC-4180-ish, whole-file-at-once. | |
| file | data_loader.hpp |
| Orchestrates sampling, fetch, and collation into batches. | |
| file | data_thread_pool.hpp |
| Minimal generic thread pool for CPU-side data pipeline work. | |
| file | dataset.hpp |
| Random-access dataset abstraction – Sample, Dataset (size()/get()). | |
| file | dataset_validator.hpp |
| Generic per-field descriptive statistics + missingness/outlier detection over any Dataset. | |
| file | image_decoder.hpp |
| Decodes image files (PNG/JPEG/BMP/etc.) into Tensors via stb_image. | |
| file | image_folder_dataset.hpp |
| Directory-of-class-subfolders image Dataset, mirroring torchvision's ImageFolder. | |
| file | image_transforms.hpp |
| Sample-level image Transforms – resize, center-crop, normalize, horizontal flip. | |
| file | iterable_dataset.hpp |
| Streaming dataset abstraction – reset()/next() for sources with no random access. | |
| file | mnist_dataset_adapter.hpp |
| Adapts a pre-loaded MnistDataset onto the generic Dataset interface. | |
| file | protein_sequences.hpp |
Protein sequence input (PLM-2): ESM's tokenizer from its vocab.txt, and FASTA files. | |
| file | protein_structure.hpp |
| Protein structures (PLM-4): PDB and mmCIF coordinates, residue distances and true contacts, to check a protein language model's predicted contacts against. | |
| file | sampler.hpp |
| Index-order abstraction for DataLoader – SequentialSampler, ShuffleSampler. | |
| file | sentencepiece_bpe.hpp |
The components SentencePiece-style BPE tokenizers are built from (TOK-3): spaces as ▁ (Replace, Prepend or Metaspace), byte fallback in BpeModel (BpeOptions::byte_fallback), and the decoder chain that undoes them, each following Hugging Face tokenizers. Gemma 3, Llama 2, Mistral and TinyLlama tokenize this way. | |
| file | text_collate.hpp |
| PadCollate – right-pads variable-length token sequences into one batch Tensor. | |
| file | text_dataset.hpp |
| Line-delimited corpus Dataset – tokenizes and indexes each line into a Tensor. | |
| file | text_tokenizer.hpp |
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and decoder, as in Hugging Face tokenizers, with added tokens split out first. encode() returns ids, token strings, offsets into the input and a special- token mask; decode() returns text. | |
| file | tokenizer.hpp |
| Minimal deterministic whitespace/punctuation word-level tokenizer. | |
| file | tokenizer_components.hpp |
| Pipeline components for TextTokenizer (TOK-1), and the word-level, byte and character tokenizers built from them. | |
| file | tokenizer_json.hpp |
Loads a Hugging Face tokenizer.json into a TextTokenizer (TOK-2). | |
| file | tokenizer_parity.hpp |
Compares a TextTokenizer with Hugging Face tokenizers on a reference corpus (TOK-2), as written by tools/tokenizers/make_tokenizer_reference.py. | |
| file | transform.hpp |
| Sample-level preprocessing abstraction – Transform, Compose, TransformDataset. | |
| file | unicode_regex.hpp |
A small backtracking regex engine over Unicode code points, for the pre-tokenizer patterns in tokenizer.json files (TOK-2). std::regex can't match Unicode categories. | |
| file | video_frame_directory_dataset.hpp |
| Directory-of-pre-extracted-frames video Dataset (reduced-scope stub, no codec decode). | |
| file | video_transforms.hpp |
| Frame-sampling Transform – selects a fixed number of evenly-spaced frames. | |
| file | vocabulary.hpp |
| Token<->index lookup with a reserved <unk> fallback, plus a frequency-ranked builder. | |
| file | wav_reader.hpp |
| Hand-rolled 16-bit PCM WAV decoder. | |
| file | word_scores.hpp |
| Per-word scores from per-token scores (TOK-4): explanations are made per token but read per word, so relevance, attributions or probe outputs are merged into the words of the input using the tokenizer's offsets. | |
Dataset/IterableDataset/DataLoader/Transform/Compose/CollateFn core, plus concrete modality implementations (tabular CSV, image, text, audio, a reduced-scope video frame-directory stub) and generic dataset validation (descriptive statistics, missingness/outlier detection), all built on the Deep Learning Modules and Layers Tensor.