One attention layer's keys and values for the positions processed so far, in buffers allocated once for max_length positions, so generation never reallocates or copies the past.
More...
#include <kv_cache.hpp>
One attention layer's keys and values for the positions processed so far, in buffers allocated once for max_length positions, so generation never reallocates or copies the past.
Keys are stored after QK-Norm and RoPE, at the K/V head count (not repeated for grouped-query attention). MultiHeadAttentionModule::forward_cached writes new positions and advances length(); everything before length() is attended to.
◆ KVCache()
| pulsatrix::KVCache::KVCache |
( |
int64_t |
batch, |
|
|
int64_t |
num_kv_heads, |
|
|
int64_t |
head_dim, |
|
|
int64_t |
max_length, |
|
|
DeviceBackend * |
backend |
|
) |
| |
- Exceptions
-
| std::invalid_argument | if any size is not positive. |
◆ advance()
| void pulsatrix::KVCache::advance |
( |
int64_t |
count | ) |
|
Records count more positions as written. Called by forward_cached.
- Exceptions
-
◆ batch()
| int64_t pulsatrix::KVCache::batch |
( |
| ) |
const |
|
inline |
◆ head_dim()
| int64_t pulsatrix::KVCache::head_dim |
( |
| ) |
const |
|
inline |
◆ keys() [1/2]
| Tensor & pulsatrix::KVCache::keys |
( |
| ) |
|
|
inline |
(batch, num_kv_heads, max_length, head_dim); positions from length() on are unused.
◆ keys() [2/2]
| const Tensor & pulsatrix::KVCache::keys |
( |
| ) |
const |
|
inline |
◆ length()
| int64_t pulsatrix::KVCache::length |
( |
| ) |
const |
|
inline |
Positions cached so far; the next forward_cached input starts at this position.
◆ max_length()
| int64_t pulsatrix::KVCache::max_length |
( |
| ) |
const |
|
inline |
◆ num_kv_heads()
| int64_t pulsatrix::KVCache::num_kv_heads |
( |
| ) |
const |
|
inline |
◆ reset()
| void pulsatrix::KVCache::reset |
( |
| ) |
|
|
inline |
Forgets every position (the buffers stay allocated).
◆ truncate()
| void pulsatrix::KVCache::truncate |
( |
int64_t |
length | ) |
|
Keeps only the first length positions, so a sequence can continue from a shared prefix.
- Exceptions
-
| std::invalid_argument | if length is negative or longer than what is cached. |
◆ values() [1/2]
| Tensor & pulsatrix::KVCache::values |
( |
| ) |
|
|
inline |
◆ values() [2/2]
| const Tensor & pulsatrix::KVCache::values |
( |
| ) |
const |
|
inline |
The documentation for this class was generated from the following file: