pulsatrix::DQNAgent Class Reference

The epsilon-greedy behaviour policy of Mnih et al. 2015: with probability epsilon act uniformly at random, otherwise take argmax_a Q(observation, a). More...

#include <dqn_agent.hpp>

Inheritance diagram for pulsatrix::DQNAgent:

Collaboration diagram for pulsatrix::DQNAgent:

Public Member Functions
	DQNAgent (Module *q_network, int64_t action_dim, float epsilon, DeviceBackend *backend, uint32_t seed=42)
	Constructs an epsilon-greedy policy over a Q-network.
Tensor	act (const Tensor &observation) override
	Chooses an action epsilon-greedily, advancing the internal LCG.
Tensor	act_greedy (const Tensor &observation)
	The purely greedy (evaluation-time) policy: argmax with no exploration at all.
void	set_epsilon (float epsilon)
	Sets the exploration probability – the hook an epsilon-decay schedule drives.
float	epsilon () const
	The current exploration probability.
int64_t	action_dim () const
	Number of discrete actions, as passed to the constructor.
Public Member Functions inherited from pulsatrix::Agent
virtual	~Agent ()=default

Detailed Description

The epsilon-greedy behaviour policy of Mnih et al. 2015: with probability epsilon act uniformly at random, otherwise take argmax_a Q(observation, a).

Note

Takes a Module* rather than defining a Q-network type of its own. A Q-network is just "some module mapping (1, observation_dim) to (1, action_dim)" – a SequentialModule of LinearModule/ReluModule builds one with no new class, and a DQNAgent-specific subclass would only hard-code an architecture choice that belongs to the training loop. Not owned; must outlive this agent – this codebase's universal non-owned-pointer convention.

No optimizer, no replay buffer, no target network, no loss. This is purely the policy half of DQN, exactly as Agent's interface defines it. The update half (sample a batch, compute targets via ComputeDQNTarget/ComputeDoubleDQNTarget, DQNLoss, backward, optimizer.step) lives in the training loop, the same split the GAN building blocks already established.

Exploration randomness comes from an internal deterministic LCG seeded at construction – never \<random\>, whose engine outputs beyond mt19937 are implementation-defined. Two agents with the same seed and the same observations act identically, which is the only thing that makes an epsilon-greedy policy testable. Same generator and constants as CartPoleEnv and ReplayBuffer.

Constructor & Destructor Documentation

DQNAgent()

pulsatrix::DQNAgent::DQNAgent	(	Module *	q_network,
		int64_t	action_dim,
		float	epsilon,
		DeviceBackend *	backend,
		uint32_t	seed = 42
	)

Constructs an epsilon-greedy policy over a Q-network.

Parameters

q_network	Network mapping a (1, observation_dim) observation to (1, action_dim) Q-values. Not owned; must outlive this agent.
action_dim	Number of discrete actions. Must be >= 1.
epsilon	Initial exploration probability, in [0, 1].
backend	Backend to allocate the returned action Tensor through. Not owned; must outlive this agent.
seed	Seed for the internal deterministic LCG.

Exceptions

std::invalid_argument

if q_network is null, action_dim <= 0, or epsilon is outside [0, 1] – external boundaries.

Member Function Documentation

act()

Tensor pulsatrix::DQNAgent::act ( const Tensor &  observation )

overridevirtual

Chooses an action epsilon-greedily, advancing the internal LCG.

Parameters

observation

Observation, shape (1, observation_dim) – whatever q_network accepts.

Returns

The chosen action index encoded as a (1, 1) float Tensor, identical to the encoding Environment::step() expects for a discrete environment.

Exceptions

std::invalid_argument

if q_network's output is not (1, action_dim) – external boundary: the network and action_dim are two independent constructor arguments a caller can genuinely mismatch, and an unchecked argmax would then read past the output row.

Note

Exactly one LCG draw is consumed for the explore/exploit coin flip, plus a second draw for the action index if and only if that flip explores. The greedy branch therefore leaves the stream one step further along than it found it, which keeps an epsilon=0 agent's stream position well-defined rather than frozen.

Host boundary (GPU-native-kernels Mission 7): action selection (the epsilon coin flip and the argmax) is host logic. The network may run on any device; its (1, action_dim) Q-values are copied to the host once per greedy call, and the action is returned through this agent's own backend.

Implements pulsatrix::Agent.

act_greedy()

Tensor pulsatrix::DQNAgent::act_greedy

(

const Tensor &

observation

)

The purely greedy (evaluation-time) policy: argmax with no exploration at all.

Parameters

observation

Observation, shape (1, observation_dim).

Returns

The greedy action index as a (1, 1) float Tensor.

Exceptions

std::invalid_argument

if q_network's output is not (1, action_dim).

Note

Consumes no randomness and advances no state, so calling it never perturbs a subsequent act() sequence – an evaluation rollout interleaved with training cannot change the training run's trajectory. That is what makes this a separate method rather than "call act() with epsilon temporarily set to 0".

Not const despite its name's suggestion of purity: Module::forward() is non-const (every module caches forward state for its own backward()), so running the network necessarily mutates it. Marking this const would require either a const_cast or a mutable member – both worse lies than the missing keyword.

action_dim()

int64_t pulsatrix::DQNAgent::action_dim ( ) const

inline

Number of discrete actions, as passed to the constructor.

epsilon()

float pulsatrix::DQNAgent::epsilon ( ) const

inline

The current exploration probability.

set_epsilon()

void pulsatrix::DQNAgent::set_epsilon

(

float

epsilon

)

Sets the exploration probability – the hook an epsilon-decay schedule drives.

Parameters

epsilon

New exploration probability, in [0, 1].

Exceptions

std::invalid_argument

if epsilon is outside [0, 1] – external boundary, same classification as the constructor's identical check.

Note

The decay schedule itself deliberately lives in the training loop, not here: a linear/exponential/step schedule is a training hyperparameter, not a property of the policy, and baking one in would force every caller into that one choice.

---

The documentation for this class was generated from the following file:

• include/pulsatrix/dqn_agent.hpp