rwkv_module.hpp File Reference

RWKV-4 time-mixing (WKV linear-attention) recurrence, with an original derived LRP rule adapting MambaLRP's detach-the-gate technique to the WKV quotient (see the class-level note). More...

#include <optional>
#include <initializer_list>
#include <vector>
#include "pulsatrix/module.hpp"

Include dependency graph for rwkv_module.hpp:

Go to the source code of this file.

Classes
class	pulsatrix::RWKVModule
	Core RWKV-4 time-mixing block (Peng et al. 2023, arXiv:2305.13048), input (N, L, d_model) -> output (N, L, d_model). More...

Namespaces
namespace	pulsatrix

Detailed Description

RWKV-4 time-mixing (WKV linear-attention) recurrence, with an original derived LRP rule adapting MambaLRP's detach-the-gate technique to the WKV quotient (see the class-level note).