pulsatrix 1.1.0
Loading...
Searching...
No Matches
Deep Learning Modules and Layers

Core tensor/graph infrastructure, layers, optimizers (SGD with momentum, Adam, AdamW, parameter groups, learning-rate schedules, gradient clipping), losses, checkpoints and safetensors I/O, seeding and deterministic mode, benchmarking, and device backends with their caching allocator – the building blocks every other group is composed from. More...

Files

file  activation_module.hpp
 A pointwise activation chosen at construction: ReLU, tanh, sigmoid, SiLU, exact GELU or GELU's tanh approximation (PLM-1).
 
file  adam_optimizer.hpp
 Adam optimizer – operates uniformly across any Module's parameters().
 
file  adaptive_avg_pool2d_module.hpp
 Average pooling to a fixed output size, as PyTorch's AdaptiveAvgPool2d (KS-9).
 
file  assert.hpp
 PULSATRIX_ASSERT – debug-only invariant check for programmer errors, distinct from throw (used for caller-facing contract violations). See cpp_style_guide/context_style_project_conventions.md's assert-vs-throw table.
 
file  attnlrp_parity.hpp
 AttnLRP parity with LXT on a real language model (LLM-7): pulsatrix's relevance against LXT's own AttnLRP on the same Hugging Face model and tokens.
 
file  autograd.hpp
 Reverse-mode autodiff – walks a ComputationGraph backward, accumulating gradients via per-node backward functions supplied by the caller.
 
file  avg_pool2d_module.hpp
 2D average pooling, non-overlapping windows (stride == kernel), no padding.
 
file  batch_norm_module.hpp
 Batch normalization (Ioffe & Szegedy, 2015) – per-channel statistics computed across the batch and spatial dimensions jointly, unlike GroupNormModule's per-(batch-row, group) statistics.
 
file  bce_with_logits_loss.hpp
 Binary cross-entropy on raw logits – combined sigmoid + BCE, numerically stable.
 
file  benchmark.hpp
 Timing, benchmark reports (pulsatrix.benchmark.v1) and regression comparison.
 
file  born_again.hpp
 Born-again networks (Furlanello et al., ICML 2018; roadmap KD-4): a student the size of its teacher, trained on the teacher's outputs, then a teacher for the next generation.
 
file  caching_allocator.hpp
 A caching device-memory allocator (roadmap HIP-3): freed blocks are kept and reused for later requests of the same size class instead of going back to the driver.
 
file  calibration_loss.hpp
 Brier score – a proper-scoring-rule calibration loss (Brier, 1950).
 
file  causal_lm.hpp
 A Llama-family decoder-only language model built from a Hugging Face config, and loading its Hugging Face weights by name (IO-4).
 
file  checkpoint.hpp
 Saving and resuming models (roadmap IO-2): parameters and buffers in one safetensors file, optimizer state in a sibling file, and a format version with a migration table.
 
file  computation_graph.hpp
 Owns and exposes graph structure – the interpretability substrate every explainer (Phase 2+) walks.
 
file  conv2d_module.hpp
 2D convolution – implemented via im2col + DeviceBackend::gemm (no new backend primitive).
 
file  cpu_backend.hpp
 CPU implementation of DeviceBackend – the first, reference DeviceBackend implementation.
 
file  cross_entropy_loss.hpp
 Softmax + negative log-likelihood classification loss.
 
file  cublas_check.hpp
 PULSATRIX_CUBLAS_CHECK – converts a cuBLAS call failure into a thrown C++ exception, parallel to PULSATRIX_CUDA_CHECK (cuda_check.hpp). Only compiled when PULSATRIX_ENABLE_CUDA is set.
 
file  cuda_backend.hpp
 CUDA implementation of DeviceBackend. Only compiled when PULSATRIX_ENABLE_CUDA is set.
 
file  cuda_check.hpp
 PULSATRIX_CUDA_CHECK – converts a CUDA runtime failure into a thrown C++ exception at the DeviceBackend boundary, per gpu_backend_programming/context_gpu_cuda_kernel_mechanics.md's Error Checking pattern. Only compiled when PULSATRIX_ENABLE_CUDA is set.
 
file  determinism.hpp
 One global seed for everything that isn't given its own, and a deterministic mode that forbids nondeterministic computation (roadmap FND-7).
 
file  device_backend.hpp
 Abstract interface isolating vendor-specific memory/compute operations from Tensor/ComputationGraph.
 
file  distillation_loss.hpp
 Knowledge distillation (roadmap KD-1): a student learns from a teacher's softened outputs (Hinton, Vinyals and Dean, 2015), optionally mixed with the hard labels.
 
file  dropout_module.hpp
 Inverted dropout – the first module whose forward behavior genuinely differs between training and inference (Module::is_training()).
 
file  embedding_module.hpp
 Lookup-table (row-select) layer – y = W[index], batched (N, L) -> (N, L, embedding_dim).
 
file  encoder_block.hpp
 A configurable transformer encoder layer (PLM-1): LayerNorm or RMSNorm, before or after each sublayer, and a plain or gated MLP. ESM-2's layer, BERT's layer and the Llama-style pre-norm layer are all settings of it.
 
file  encoder_lm.hpp
 Masked-language-model encoders: ESM-2 protein language models (PLM-2), Hugging Face EsmForMaskedLM checkpoints loaded, run, trained and explained natively, and MDLM masked diffusion language models (DIFF-2).
 
file  feature_distillation.hpp
 Distilling a teacher's intermediate features, not only its outputs (roadmap KD-3): FitNets' hint loss (Romero et al., ICLR 2015) and attention transfer (Zagoruyko and Komodakis, ICLR 2017).
 
file  feed_forward_module.hpp
 The plain transformer MLP, Linear -> activation -> Linear, of BERT, ESM and the original Transformer (PLM-1). The gated variant is SwiGLUModule.
 
file  flatten_module.hpp
 Reshape-only Module – flattens every non-batch dim of a (N, ...) input to (N, flattened_features), for chaining Conv2DModule's batched (N,C,H,W) output into a LinearModule's batched (N, in_features) input.
 
file  generation.hpp
 Text generation from a language model: greedy decoding and seeded sampling with temperature, top-k, top-p and min-p, repetition penalty, and EOS handling (LLM-4).
 
file  golden_logits.hpp
 The golden-logit harness: a loaded model's logits against Hugging Face's on real text (LLM-6).
 
file  grad_clipping.hpp
 Global gradient-norm clipping (roadmap TRN-3).
 
file  group_norm_module.hpp
 Group normalization (Wu & He, 2018) – rank-3 (C, H, W), matching Conv2DModule's convention, unlike RMSNormModule/LayerNormModule's rank-1 feature-vector scope.
 
file  gru_module.hpp
 Single-layer GRU – gated recurrence with a reset-gated candidate.
 
file  hf_model.hpp
 Hugging Face checkpoints: config.json and single or sharded safetensors (IO-5).
 
file  hip_backend.hpp
 HIP/ROCm implementation of DeviceBackend. Only compiled when PULSATRIX_ENABLE_HIP is set.
 
file  hip_check.hpp
 PULSATRIX_HIP_CHECK – converts a HIP runtime failure into a thrown C++ exception at the DeviceBackend boundary, per gpu_backend_programming/context_gpu_cuda_kernel_mechanics.md's Error Checking pattern. Only compiled when PULSATRIX_ENABLE_HIP is set.
 
file  hipblas_check.hpp
 PULSATRIX_HIPBLAS_CHECK – converts a hipBLAS call failure into a thrown C++ exception, parallel to PULSATRIX_HIP_CHECK (hip_check.hpp). Only compiled when PULSATRIX_ENABLE_HIP is set.
 
file  host_guard.hpp
 PULSATRIX_REQUIRE_HOST – always-on guard for code paths that dereference Tensor::data() on the host.
 
file  ia3.hpp
 IA³ (Liu et al., 2022): fine-tune a frozen model by learning one rescaling vector per adapted Linear layer, on its output (keys, values) or its input (the MLP's down projection), and save and load it as a PEFT adapter.
 
file  kl_divergence_loss.hpp
 VAE KL-divergence-to-standard-normal loss term.
 
file  kv_cache.hpp
 Preallocated key/value cache for incremental attention (LLM-5).
 
file  layer_norm_module.hpp
 Layer normalization (Ba et al., 2016) – mean-centered/scaled RMSNormModule sibling.
 
file  layer_student.hpp
 DistilBERT's recipe for a smaller language model (Sanh et al., 2019; roadmap KD-4): a student with some of the teacher's layers, started from those layers' weights, and a cosine loss that aligns the two models' hidden states.
 
file  linear_module.hpp
 Dense/fully-connected layer – the reference Module implementation.
 
file  lm_scoring.hpp
 Prompt/completion data for a causal language model (LLM-10): read it from JSONL, batch it with the prompt masked out of the loss, and score it, as log P(completion | prompt) or as the probabilities of a few single-token choices ("P(True)").
 
file  lora.hpp
 LoRA (Hu et al., 2021): fine-tune a frozen model through a low-rank update on each of its Linear layers, merge it into the weights or take it out again, and save and load it as a PEFT adapter.
 
file  loreft.hpp
 LoReFT (Wu et al., 2024): fine-tune a frozen language model by editing its hidden states in a learned low-rank subspace, at a few positions after chosen layers, and explain the edit with LRP.
 
file  lr_scheduler.hpp
 Learning-rate schedules (warmup, constant, linear, cosine) and a scheduler that applies one to an optimizer and its parameter groups (roadmap TRN-4).
 
file  lstm_module.hpp
 Single-layer LSTM – this codebase's first gated recurrent module.
 
file  mamba_module.hpp
 Mamba/S6 selective-state-space recurrence with the MambaLRP relevance rule.
 
file  masked_diffusion.hpp
 Sampling from a masked diffusion language model (DIFF-2): start from a sequence with masked positions and fill them over a number of steps, recording the whole trajectory so it can be explained afterwards (DIFF-3). DPLM's remasking sampler (DIFF-4) also takes positions back.
 
file  max_pool2d_module.hpp
 2D max pooling with a stride and padding (non-overlapping windows by default).
 
file  metrics_sink.hpp
 Keeps monitoring/visualization tools out of the training core – same OCP/DIP pattern as DeviceBackend/ExplainerContext.
 
file  mnist_classifier_example.hpp
 Real MNIST classifier: Conv2DModule -> ReluModule -> FlattenModule -> LinearModule -> CrossEntropyLoss, mirroring XorNetwork's training-loop pattern and grad_cam_mnist_demo.cpp's network shape exactly.
 
file  mnist_loader.hpp
 Parses real MNIST IDX/ubyte files (fetched by tools/fetch_mnist.py) into Tensor images and integer labels.
 
file  module.hpp
 Abstract base every layer subclasses – NVI forward(), pure-virtual LRP contract.
 
file  mse_loss.hpp
 Mean squared error loss.
 
file  multihead_attention_module.hpp
 Multi-head scaled dot-product attention – this codebase's first Module composed out of other real Modules, plus AttnLRP's Eq. 15 bilinear relevance rule.
 
file  node.hpp
 Computation graph node – op type, shape, optional label, parent/child edges.
 
file  noise_schedule.hpp
 DDPM noise schedules (linear and cosine): precomputed beta/alpha/alpha_bar, forward noising, and DDPM and DDIM reverse steps.
 
file  op_type.hpp
 Closed set of operation categories every graph Node is tagged with.
 
file  param_groups.hpp
 Optimizer parameter groups: a learning rate and weight decay per set of parameters, chosen by name (roadmap TRN-1).
 
file  protein_training.hpp
 Training protein language models (PLM-7): ESM-2's masked-LM data pipeline (masking, cropping, cluster-weighted sampling), its initialization, a training step and evaluation, and heads for fine-tuning on per-protein and per-residue labels.
 
file  relu_module.hpp
 ReLU activation – the second Module subclass, following LinearModule's pattern.
 
file  reparameterize.hpp
 VAE reparameterization trick: z = mu + exp(log_sigma) * epsilon.
 
file  residual_module.hpp
 Generic skip-connection wrapper – Phase 4's only mission, the ResNet-style residual block generalized past a fixed Conv-BN-ReLU stack to any Module.
 
file  retnet_module.hpp
 RetNet retention mechanism (recurrent mode), with an original derived LRP rule (no published rule exists for RetNet – see the class-level note).
 
file  rms_norm_module.hpp
 RMS normalization layer (Zhang & Sennrich, 2019) – this codebase's first normalization Module, establishing the Tier 1 pattern for GroupNorm/LayerNorm.
 
file  rnn_module.hpp
 Vanilla (Elman) recurrent layer – this codebase's first recurrent module.
 
file  rope_module.hpp
 Rotary Position Embedding – fixed per-position pair rotation, epsilon-rule LRP.
 
file  rwkv_module.hpp
 RWKV-4 time-mixing (WKV linear-attention) recurrence, with an original derived LRP rule adapting MambaLRP's detach-the-gate technique to the WKV quotient (see the class-level note).
 
file  safetensors.hpp
 Native safetensors reader and writer (roadmap IO-1): the one file format pulsatrix reads and writes for weights.
 
file  sequential_module.hpp
 Model container – chains a sequence of existing Modules.
 
file  sgd_optimizer.hpp
 Stochastic gradient descent – operates uniformly across any Module's parameters().
 
file  shape.hpp
 Tensor dimension arithmetic – rank, element count, per-dimension access.
 
file  sinusoidal_timestep_embedding.hpp
 Fixed sinusoidal encoding of a diffusion timestep, for conditioning a denoiser.
 
file  soft_decision_tree.hpp
 A soft decision tree (Frosst and Hinton, "Distilling a Neural Network Into a Soft Decision Tree", 2017): an interpretable student for a network teacher (roadmap KD-3).
 
file  softmax_module.hpp
 Rank-agnostic softmax over the last axis, with AttnLRP's Eq. 13 DTD relevance rule.
 
file  swiglu_module.hpp
 SwiGLU gated feedforward block – second module composed from real LinearModule sub-objects, plus a diagonal specialization of AttnLRP's Eq. 15 bilinear rule.
 
file  tagger_finetune_example.hpp
 A full fine-tuning recipe (roadmap TRN-6): pretrain a tiny transformer tagger on one rule, save it, reload it and fine-tune every parameter on a related rule.
 
file  tensor.hpp
 N-dimensional tensor – owns a buffer via DeviceBackend*, RAII (Rule of Five).
 
file  tied_lm_head_module.hpp
 Language-model head that reuses an EmbeddingModule's table as its weight (LLM-2).
 
file  token_cross_entropy_loss.hpp
 Cross-entropy over a batch of tokens, with padding ignored and a caller-chosen normalizer for exact gradient accumulation (roadmap TRN-5).
 
file  top_k.hpp
 Top-k selection along a tensor's last dimension, on any device (roadmap FND-3).
 
file  transformer_block.hpp
 Pre-LN transformer block – Phase 3's literal exit-gate deliverable, third and last composition mission (RMSNorm x2, MultiHeadAttentionModule, SwiGLUModule, plus the resolved residual-split LRP rule).
 
file  vision_models.hpp
 torchvision's ResNet and VGG, built from pulsatrix layers with torchvision's parameter names, so their published ImageNet weights load once converted to safetensors (KS-9).
 
file  xor_training_example.hpp
 Phase 1's training-loop proof: Linear(2,4)->ReLU->Linear(4,1) learning XOR.
 

Detailed Description

Core tensor/graph infrastructure, layers, optimizers (SGD with momentum, Adam, AdamW, parameter groups, learning-rate schedules, gradient clipping), losses, checkpoints and safetensors I/O, seeding and deterministic mode, benchmarking, and device backends with their caching allocator – the building blocks every other group is composed from.