Skip to content

GPU Profiling (HIP)

scripts/profile_hip.sh runs a command under rocprofv3, groups its GPU kernels by op type and writes a CSV of where the time went. It also reports the GPU's busy time over the run's wall span. When that is low, the GPU is waiting on launches and synchronization, not computing. Measure before optimizing: every HIP performance item on the roadmap is judged against these numbers. To measure whether a change made training or explanations faster or slower, and to compare two builds, use the benchmark suite.

Running it

rocprofv3 is in the ROCm container, so run the script there:

# build the fixed training workloads once
scripts/rocm-build.sh 'cmake -S . -B build-hip-release -DCMAKE_BUILD_TYPE=Release -DPULSATRIX_ENABLE_HIP=ON \
    && cmake --build build-hip-release -j"$(nproc)" --target hip_profile_workloads'

# profile one: <ops.csv> -- <command>
scripts/rocm-build.sh 'scripts/profile_hip.sh profile/cnn.csv -- ./build-hip-release/hip_profile_workloads cnn 20'

Any command works in place of hip_profile_workloads, such as your own training program. The CSV has one row per op: op, calls, total_us, mean_us, percent. Op names come from the kernel name (batch_norm_forward_kernel becomes batch_norm_forward). All of hipBLAS's GEMM kernels are grouped as gemm (hipBLAS), and HIP runtime helpers such as copies appear as runtime: ....

hip_profile_workloads has three fixed training workloads, each a full training step (forward, loss, backward, AdamW), and one reduction microbenchmark:

  • mlp: Linear(256→512) → ReLU → Linear(512→512) → ReLU → Linear(512→10), batch 64.
  • cnn: Conv2D(3→16) → BatchNorm → ReLU → Conv2D(16→32, stride 2) → BatchNorm → ReLU → Flatten → Linear, on 32 images of 3×32×32.
  • tagger: the transformer tagger from the full fine-tuning recipe.
  • reduce: DeviceBackend::dot and sum over 16M floats, a microbenchmark for HIP-5.

Other programs on the GPU skew timings

Step times and kernel times shift when other programs use the GPU; a local LLM server is enough to triple them. Compare two builds by interleaving their runs (old, new, old, new...) and taking medians, so both see the same load. The allocator counters (hip_profile_workloads prints them) don't depend on load.

Baseline (gfx1151, ROCm 7.2.4, 20 steps)

Measured 2026-10-04 on the Radeon 8060S (Strix Halo), Release build, before any of the HIP performance work.

Workload Step time GPU busy Biggest costs (share of kernel time)
mlp about 1.0 ms 12.9% hipBLAS GEMM 56%, fill 11%, column_sums 8%, copies 7%
cnn about 27 ms 78.3% batch_norm_forward 34%, batch_norm_backward 31%, hipBLAS GEMM 18%, batch_norm_update_running 14%
tagger 7.8 ms 11.6% hipBLAS GEMM 38%, copies 15%, fill 14%, column_sums 12%

What it shows:

  • The small models are bound by launch and sync overhead. The mlp and tagger GPUs are busy only 12–13% of the time. The tagger launches 14,457 kernels in 20 steps, and every op waits for the stream to finish (HIP-4).
  • BatchNorm dominates the CNN. Its three kernels take 79% of the kernel time, because each runs one thread per channel: 16 or 32 threads for 16,384 or 4,096 elements per channel (HIP-2).
  • Fills and copies are a large share. Many of them are temporaries a caching allocator and fused kernels would avoid (HIP-3, HIP-6).

Results

HIP-5: dot and sum across many blocks

They used to run on a single 256-thread block. Now a first pass of up to 1024 blocks writes per-block partial sums, and one block adds those in a fixed order: still deterministic, no atomics. On the reduce workload (16M floats):

Op Before After Effective bandwidth after
dot (reads 128 MB) 17.7 ms 0.61 ms (29×) 211 GB/s
sum (reads 64 MB) 17.2 ms 0.30 ms (58×) 216 GB/s

Both now run at the measured memory bandwidth of the gfx1151 (about 212 GB/s), so there is nothing left to gain on these two ops.

HIP-2: BatchNorm as parallel per-channel reductions

BatchNorm ran one thread per channel over every element of that channel. Now each per-channel sum is a deterministic two-stage reduction: several blocks per channel produce partials, which one thread per channel then adds in order. Normalization and the input gradient run one thread per element. On the cnn workload:

Before After
Step time about 27 ms about 10 ms (2.7×)
BatchNorm kernels (20 steps) 342 ms, 79% of kernel time about 3 ms
GPU busy 78.3% 45.8%

GEMM is now 84% of the CNN's kernel time. The drop in GPU busy time means launch and sync overhead is now the larger cost, which is HIP-4's target.

Results now differ from the CPU in the last bits. The GPU adds each channel's values by a tree, while the CPU adds them one at a time. Both are deterministic. Softmax, layer and RMS norms, and column sums keep their kernels: at current shapes each call takes about 5 µs, which is launch overhead rather than missing parallelism (the roadmap's falsifier for HIP-2).

HIP-3: a caching allocator

Every tensor used to be a fresh hipMalloc, and every free a hipFree, which waits for the GPU to finish. Now HIPBackend keeps freed blocks and reuses them for the same size class (512-byte steps below 1 MiB, then steps of an eighth of a power of two). Over 20 steps:

Workload Allocations hipMalloc calls now Median step time, interleaved A/B
mlp 618 48 4% faster
cnn 1,124 86 12% faster
tagger 5,359 156 7% faster

The gains are modest because every op still synchronizes the stream. Removing those syncs is HIP-4. Cap the memory a backend holds with set_memory_budget() or PULSATRIX_HIP_MEMORY_BUDGET_MB, and return cached blocks with empty_cache().

HIP-4: no per-op synchronization

Every HIP op used to wait for the GPU to finish before returning: 59 synchronizations, one per kernel or hipBLAS call. Now ops are queued on the backend's stream, and the host waits only where it must:

  • Copies to the host, because the caller reads the result next. This includes dot, sum and reading a loss value.
  • Copies from the host, because the source is often a temporary buffer.
  • Releasing memory to the driver.

Interleaved A/B against HIP-3:

Workload Before After Speedup
mlp 0.70 ms/step 0.35 ms/step 2.0×
tagger 5.35 ms/step 2.84 ms/step 1.9×
cnn 7.28 ms/step 5.24 ms/step 1.4×

Set PULSATRIX_HIP_SYNC_DEBUG=1 before creating the backend to restore the per-op waits. A failing kernel otherwise reports its error at the next wait, not at the op that caused it.

After HIP-4 the small models were still only about 20% busy, because of host-side work inside each step: reading the loss back, and losses that validated their targets on the host. HIP-6 (below) moved the target checks to the device and fused the optimizer step.

HIP-7: Conv2D in batch chunks

Conv2DModule unfolds its input into patches (im2col) and multiplies them by the kernel. The patches take N x C*kh*kw x out_h*out_w floats: 7.4 MB per 224x224 image in ResNet18's first layer, 115 MB in VGG16's widest. Every convolution used to keep its whole batch's patches from forward() for backward() and LRP, so they added up across the network.

Now a layer keeps them only when they fit its workspace budget, Conv2DModule::set_max_workspace_bytes() (16 MiB by default). A larger batch runs in chunks of examples, and backward() and propagate_relevance() rebuild each chunk's patches from the cached input. The results are bit-identical for every budget; the tests check forward, the three gradients and all five LRP rules, on the CPU and the GPU (still after HIP-6's batched GEMMs).

The budget is per layer: call set_max_workspace_bytes() on each Conv2DModule. Lower it if a large batch runs out of memory; SIZE_MAX keeps every patch, as before HIP-7.

Explaining ResNet18 (EpsilonPlus, batch 32, 224x224). Before: every layer keeps all its patches. After: the 16 MiB default. GPU times are from three runs each in ABBA order (old, new, new, old, so drift affects both builds; see Benchmarks), all within 0.01 s of each other. CPU times are single runs.

Device Peak memory before After Time before After
gfx1151 (the HIP allocator's peak memory in use) 5.75 GB 2.40 GB 0.70 s 0.60 s
CPU, Release (peak RSS) 5.3 GB 2.3 GB 14.5 s 14.5 s

Rebuilding the patches is cheap next to the GEMMs, and a smaller working set is friendlier to the caches, so the chunked layers are no slower.

HIP-6: fused kernels

Fewer launches and fewer waits in each training step, with the same numbers:

  • One launch for the optimizer. AdamOptimizer (and AdamW, and Adam with L2 decay) updates every parameter with one adam_step_multi call: on the GPU, one kernel per 32 parameter tensors, with nothing uploaded first. The weight decay is folded into the same kernel.
  • One wait for gradient clipping. ClipGradNorm used to read back one dot per parameter (19 waits per tagger step). The squared norms now go to one device buffer and come back together; the host still adds them in double, in the same order.
  • Softmax plus cross-entropy with the targets checked on the device. TokenCrossEntropyLoss::forward() used to copy the targets to the host to validate them, then upload the decoded indices and weights. One row kernel now decodes and checks them, computes the softmax and the loss terms, and counts tokens and bad targets; one read back of four numbers returns the loss. A bad target is reported with the same message as before.
  • Bias plus ReLU. SequentialModule runs each LinearModule followed by a ReluModule as one kernel that writes both the Linear output (which the ReLU's backward and LRP read) and the ReLU output. A test checks that the fused layers give the same outputs, gradients and relevance, bit for bit, as the same layers called one by one.
  • Batched GEMMs in Conv2D (a HIP-7 follow-up). The kernel against every example's patches is one strided-batched GEMM (one call over many equally spaced matrices), in forward, backward and LRP, instead of one GEMM per example. Each example's kernel and bias gradients still go to their own buffer and are added in example order, so they sum as before.

The fused ops are bit-identical to the unfused ones on the CPU (they are the same calls) and on the GPU (they share the device functions; the tests compare them with EXPECT_EQ). The batched GEMM's rounding can differ from one GEMM per example on the GPU, within the CPU-GPU tolerance every test already uses.

The benchmark suite (Benchmarks) on gfx1151, four ABBA rounds. Before is the commit with HIP-7 only:

Benchmark Before After Change
train.mlp 0.356 ms 0.302 ms −15%
train.tagger 2.345 ms 1.952 ms −17%
train.cnn 5.302 ms 1.445 ms −73%
explain.* within ±2.5%

LRP conservation and Integrated Gradients' completeness are unchanged to every printed digit. On the CPU, where the new ops run the calls they replace, every benchmark stays within ±2.5%. Kernel launches per training step (hip_profile_workloads, 20 steps): mlp 73 to 53, tagger 754 to 692, cnn 531 to 137.

Zero-fills of new tensors are now the largest cost after GEMM (13% of mlp's and 19% of tagger's kernel time): most of those tensors are overwritten in full right after.

ROCm 10.0.0 evaluation

ROCm 10.0.0 (2026-08-26) is the first release whose notes list gfx1151 (Ryzen AI Max). This evaluation is roadmap HIP-9. The image is docker/Dockerfile.rocm built on rocm/dev-ubuntu-24.04:10.0.0-full. It ships HIP 7.15 and AMD clang 23, against 7.2.4's HIP 7.2 and clang 22.

Measured 2026-10-04 on the Radeon 8060S, host kernel 7.0.0, same commit for both versions:

Check ROCm 7.2.4 ROCm 10.0.0
HIP Debug tests 2402/2402 2402/2402, also with PULSATRIX_HIP_SYNC_DEBUG=1
scripts/profile_hip.sh (rocprofv3) works works (rocprofv3 1.3.5)
Benchmark suite, 6 ABBA rounds baseline no regressions; every time within ±2%
LRP conservation and IG completeness baseline bit-identical
Benchmark (pulsatrix_bench, HIP) 7.2.4 10.0.0 Change p
train.mlp 0.366 ms 0.371 ms +1.6% 0.066
train.cnn 5.248 ms 5.290 ms +0.8% 0.242
train.tagger 2.229 ms 2.268 ms +1.8% 0.013
explain.lrp_epsilon.mlp 0.290 ms 0.288 ms -0.5% 0.409
explain.lrp_epsilon_plus.convnet 0.690 ms 0.690 ms 0.0% 0.758
explain.integrated_gradients.mlp 7.306 ms 7.222 ms -1.2% 0.934

Two differences from 7.2.4 matter for building against it:

  • Library path. ROCm 10's image keeps its libraries under /opt/rocm/core-10.0/lib, reached through /opt/rocm/lib, and doesn't register that directory with the dynamic loader. Programs then fail to start with libhipblas.so.3: cannot open shared object file. The Dockerfile registers it. Outside the container, add /opt/rocm/lib to /etc/ld.so.conf.d/ or LD_LIBRARY_PATH.
  • hipBLASLt. ROCm 10's hipBLAS links hipBLASLt. Whether GEMMs go through it on gfx1151 is HIP-8's question.

After this evaluation, ROCm 10.0.0 became the default container; PULSATRIX_ROCM_VERSION=7.2.4 still selects the previous one. The HIP results above were measured on 7.2.4, before the switch.