pulsatrix 1.1.0
Loading...
Searching...
No Matches
Data Loading, Transformation & Validation

Dataset/IterableDataset/DataLoader/Transform/Compose/CollateFn core, plus concrete modality implementations (tabular CSV, image, text, audio, a reduced-scope video frame-directory stub) and generic dataset validation (descriptive statistics, missingness/outlier detection), all built on the Deep Learning Modules and Layers Tensor. More...

Files

file  audio_collate.hpp
 AudioPadCollate – zero-pads variable-length waveforms into one batch Tensor.
 
file  audio_folder_dataset.hpp
 Directory-of-class-subfolders audio Dataset, mirroring ImageFolderDataset.
 
file  audio_transforms.hpp
 Sample-level audio Transforms – linear-interpolation resampling.
 
file  bounded_queue.hpp
 Fixed-capacity thread-safe blocking queue – the staged-pipeline backbone.
 
file  byte_level_bpe.hpp
 The components byte-level BPE tokenizers are built from (TOK-2): NFC, regex and digit splitting, byte-level mapping, BPE, template post-processing and byte-level decoding, each following Hugging Face tokenizers.
 
file  collate.hpp
 Batch assembly – Batch, CollateFn, DefaultCollate.
 
file  csv_dataset.hpp
 Dataset over a CSV file's numeric feature/label columns – Phase 1's tabular reference case.
 
file  csv_reader.hpp
 Minimal hand-rolled CSV parser – RFC-4180-ish, whole-file-at-once.
 
file  data_loader.hpp
 Orchestrates sampling, fetch, and collation into batches.
 
file  data_thread_pool.hpp
 Minimal generic thread pool for CPU-side data pipeline work.
 
file  dataset.hpp
 Random-access dataset abstraction – Sample, Dataset (size()/get()).
 
file  dataset_validator.hpp
 Generic per-field descriptive statistics + missingness/outlier detection over any Dataset.
 
file  image_decoder.hpp
 Decodes image files (PNG/JPEG/BMP/etc.) into Tensors via stb_image.
 
file  image_folder_dataset.hpp
 Directory-of-class-subfolders image Dataset, mirroring torchvision's ImageFolder.
 
file  image_transforms.hpp
 Sample-level image Transforms – resize, center-crop, normalize, horizontal flip.
 
file  iterable_dataset.hpp
 Streaming dataset abstraction – reset()/next() for sources with no random access.
 
file  mnist_dataset_adapter.hpp
 Adapts a pre-loaded MnistDataset onto the generic Dataset interface.
 
file  protein_sequences.hpp
 Protein sequence input (PLM-2): ESM's tokenizer from its vocab.txt, and FASTA files.
 
file  protein_structure.hpp
 Protein structures (PLM-4): PDB and mmCIF coordinates, residue distances and true contacts, to check a protein language model's predicted contacts against.
 
file  sampler.hpp
 Index-order abstraction for DataLoader – SequentialSampler, ShuffleSampler.
 
file  sentencepiece_bpe.hpp
 The components SentencePiece-style BPE tokenizers are built from (TOK-3): spaces as ▁ (Replace, Prepend or Metaspace), byte fallback in BpeModel (BpeOptions::byte_fallback), and the decoder chain that undoes them, each following Hugging Face tokenizers. Gemma 3, Llama 2, Mistral and TinyLlama tokenize this way.
 
file  text_collate.hpp
 PadCollate – right-pads variable-length token sequences into one batch Tensor.
 
file  text_dataset.hpp
 Line-delimited corpus Dataset – tokenizes and indexes each line into a Tensor.
 
file  text_tokenizer.hpp
 The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and decoder, as in Hugging Face tokenizers, with added tokens split out first. encode() returns ids, token strings, offsets into the input and a special- token mask; decode() returns text.
 
file  tokenizer.hpp
 Minimal deterministic whitespace/punctuation word-level tokenizer.
 
file  tokenizer_components.hpp
 Pipeline components for TextTokenizer (TOK-1), and the word-level, byte and character tokenizers built from them.
 
file  tokenizer_json.hpp
 Loads a Hugging Face tokenizer.json into a TextTokenizer (TOK-2).
 
file  tokenizer_parity.hpp
 Compares a TextTokenizer with Hugging Face tokenizers on a reference corpus (TOK-2), as written by tools/tokenizers/make_tokenizer_reference.py.
 
file  transform.hpp
 Sample-level preprocessing abstraction – Transform, Compose, TransformDataset.
 
file  unicode_regex.hpp
 A small backtracking regex engine over Unicode code points, for the pre-tokenizer patterns in tokenizer.json files (TOK-2). std::regex can't match Unicode categories.
 
file  video_frame_directory_dataset.hpp
 Directory-of-pre-extracted-frames video Dataset (reduced-scope stub, no codec decode).
 
file  video_transforms.hpp
 Frame-sampling Transform – selects a fixed number of evenly-spaced frames.
 
file  vocabulary.hpp
 Token<->index lookup with a reserved <unk> fallback, plus a frequency-ranked builder.
 
file  wav_reader.hpp
 Hand-rolled 16-bit PCM WAV decoder.
 
file  word_scores.hpp
 Per-word scores from per-token scores (TOK-4): explanations are made per token but read per word, so relevance, attributions or probe outputs are merged into the words of the input using the tokenizer's offsets.
 

Detailed Description

Dataset/IterableDataset/DataLoader/Transform/Compose/CollateFn core, plus concrete modality implementations (tabular CSV, image, text, audio, a reduced-scope video frame-directory stub) and generic dataset validation (descriptive statistics, missingness/outlier detection), all built on the Deep Learning Modules and Layers Tensor.