swiglu_module.hpp File Reference

SwiGLU gated feedforward block – second module composed from real LinearModule sub-objects, plus a diagonal specialization of AttnLRP's Eq. 15 bilinear rule. More...

#include <optional>
#include "pulsatrix/linear_module.hpp"
#include "pulsatrix/module.hpp"

Include dependency graph for swiglu_module.hpp:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
class	pulsatrix::SwiGLUModule
	down_proj(silu(gate_proj(x)) * up_proj(x)), the gated feedforward block used in place of a plain two-linear-layer MLP in most modern transformers. Rank-agnostic over (..., d_model) -> (..., d_model), matching MultiHeadAttentionModule's I/O contract so Phase 3 Mission 5 (TransformerBlock) can chain them directly. More...

Namespaces
namespace	pulsatrix

Detailed Description

SwiGLU gated feedforward block – second module composed from real LinearModule sub-objects, plus a diagonal specialization of AttnLRP's Eq. 15 bilinear rule.