multihead_attention_module.hpp File Reference

Multi-head scaled dot-product attention – this codebase's first Module composed out of other real Modules, plus AttnLRP's Eq. 15 bilinear relevance rule. More...

#include <optional>
#include <memory>
#include <vector>
#include "pulsatrix/linear_module.hpp"
#include "pulsatrix/module.hpp"
#include "pulsatrix/rms_norm_module.hpp"
#include "pulsatrix/rope_module.hpp"
#include "pulsatrix/softmax_module.hpp"

Include dependency graph for multihead_attention_module.hpp:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
class	pulsatrix::MultiHeadAttentionModule
	softmax(Q @ K^T / sqrt(head_dim)) @ V, multi-head, with optional RoPE and optional QK-Norm. Shape (N, L, d_model) -> (N, L, d_model). More...

Namespaces
namespace	pulsatrix

Detailed Description

Multi-head scaled dot-product attention – this codebase's first Module composed out of other real Modules, plus AttnLRP's Eq. 15 bilinear relevance rule.