pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::KVCache Class Reference

One attention layer's keys and values for the positions processed so far, in buffers allocated once for max_length positions, so generation never reallocates or copies the past. More...

#include <kv_cache.hpp>

Public Member Functions

 KVCache (int64_t batch, int64_t num_kv_heads, int64_t head_dim, int64_t max_length, DeviceBackend *backend)
 
int64_t batch () const
 
int64_t num_kv_heads () const
 
int64_t head_dim () const
 
int64_t max_length () const
 
int64_t length () const
 Positions cached so far; the next forward_cached input starts at this position.
 
void reset ()
 Forgets every position (the buffers stay allocated).
 
void truncate (int64_t length)
 Keeps only the first length positions, so a sequence can continue from a shared prefix.
 
Tensor & keys ()
 (batch, num_kv_heads, max_length, head_dim); positions from length() on are unused.
 
Tensor & values ()
 
const Tensor & keys () const
 
const Tensor & values () const
 
void advance (int64_t count)
 Records count more positions as written. Called by forward_cached.
 

Detailed Description

One attention layer's keys and values for the positions processed so far, in buffers allocated once for max_length positions, so generation never reallocates or copies the past.

Keys are stored after QK-Norm and RoPE, at the K/V head count (not repeated for grouped-query attention). MultiHeadAttentionModule::forward_cached writes new positions and advances length(); everything before length() is attended to.

Constructor & Destructor Documentation

◆ KVCache()

pulsatrix::KVCache::KVCache ( int64_t  batch,
int64_t  num_kv_heads,
int64_t  head_dim,
int64_t  max_length,
DeviceBackend *  backend 
)
Exceptions
std::invalid_argumentif any size is not positive.

Member Function Documentation

◆ advance()

void pulsatrix::KVCache::advance ( int64_t  count)

Records count more positions as written. Called by forward_cached.

Exceptions
std::invalid_argumentif that would pass max_length().

◆ batch()

int64_t pulsatrix::KVCache::batch ( ) const
inline

◆ head_dim()

int64_t pulsatrix::KVCache::head_dim ( ) const
inline

◆ keys() [1/2]

Tensor & pulsatrix::KVCache::keys ( )
inline

(batch, num_kv_heads, max_length, head_dim); positions from length() on are unused.

◆ keys() [2/2]

const Tensor & pulsatrix::KVCache::keys ( ) const
inline

◆ length()

int64_t pulsatrix::KVCache::length ( ) const
inline

Positions cached so far; the next forward_cached input starts at this position.

◆ max_length()

int64_t pulsatrix::KVCache::max_length ( ) const
inline

◆ num_kv_heads()

int64_t pulsatrix::KVCache::num_kv_heads ( ) const
inline

◆ reset()

void pulsatrix::KVCache::reset ( )
inline

Forgets every position (the buffers stay allocated).

◆ truncate()

void pulsatrix::KVCache::truncate ( int64_t  length)

Keeps only the first length positions, so a sequence can continue from a shared prefix.

Exceptions
std::invalid_argumentif length is negative or longer than what is cached.

◆ values() [1/2]

Tensor & pulsatrix::KVCache::values ( )
inline

◆ values() [2/2]

const Tensor & pulsatrix::KVCache::values ( ) const
inline

The documentation for this class was generated from the following file: