DDPG Agent for Battery Control: Scalar action vs Full trajectory?

I’m implementing a Reinforcement Learning solution to control a home battery, similar to a Model Predictive Control (MPC) approach. The observation includes the current state of charge (SoC) and N-step forecasts for PV generation, electrical demand, import price, and export price.

In MPC, I calculate an optimal charge/discharge trajectory over the prediction horizon and output the entire plan. Now, I’m trying to implement the same using a DDPG agent in MATLAB.

My questions:

Should the DDPG agent output a scalar action (charging/discharging power) for each timestep, which is then used to update the SoC based on the sampling time,

or should the agent output a full trajectory, where I execute only the first action but discard the remaining ones, while still using the full trajectory for the reward calculation?

My thoughts:

In MPC, I get the entire optimal trajectory for charging and discharging over the horizon. Initially, I considered using the same approach with the DDPG agent. However, I’m wondering if this is necessary because the value function already accounts for downstream benefits (future prices/loads) since they are included in the state, right?

But if the agent returns just one action for the next state, it seems like this would lead to a result similar to what I would get if I had no prediction horizon at all.

Thanks in advance for any suggestions.

[Umar]

Hi @Alexander,

In DDPG, the agent usually outputs a scalar action per timestep, not a full trajectory. By including N-step forecasts in your state vector, the value function already accounts for future effects, so single-step actions suffice. After each action, update the SoC, compute the reward, and train your actor and critic networks via experience replay—this effectively captures the lookahead of MPC without needing the full trajectory.

Reference:

• Deep Deterministic Policy Gradient (DDPG) Agent https://www.mathworks.com/help/reinforcement-learning/ug/ddpg-agents.html