Custom DQN enviroment & Loss function.

Hi all. I am working on my thesis project which involves the use of the DQN applied to the dismantling of networks (Undirected Graph). The first problem is to create an environment in which the actions are the removal of individual node.

The second problem concerns the loss function used by the DQN. I would like to know if there is a way to modify the loss function by adding a penalty. I am attaching the function that I would like to use by the Routine DQN of matlab function:

I also add that the Q (s, a) values used will be calculated upstream through another algorithm, so I would need to use those values.

Answer 1

[Aditya]

Creating a custom environment for a DQN (Deep Q-Network) that involves the dismantling of networks (graphs) and modifying the loss function to include a penalty can be achieved in MATLAB. Let's break down the process into steps to address both of your concerns.

1. Creating a Custom Environment for Network Dismantling

For your thesis project, you'll need to define an environment that represents the undirected graph and actions that correspond to the removal of individual nodes. MATLAB's Reinforcement Learning Toolbox allows you to create custom environments by defining the necessary components such as observations, actions, and the reward mechanism.

2. Modifying the Loss Function in DQN

To modify the loss function used by the DQN algorithm in MATLAB, especially to add a penalty, you might have to customize the training loop or the part of the code where the loss is computed. The DQN's loss function is typically the mean squared error (MSE) between the predicted Q-values and the target Q-values. To add a penalty, you would adjust the computation of the target Q-values or directly modify the loss calculation.

If you have specific Q-values calculated upstream and want to use them along with a penalty in the loss function, you could manually compute the loss and perform the gradient update steps. Here's a conceptual sketch of how you might implement this:

1. Compute Q-Values: Use your algorithm to compute the Q-values for the current state-action pairs.
2. Compute Target Q-Values: For the next state, use your algorithm to compute the Q-values and then apply your custom penalty to these values.
3. Compute Loss: Calculate the loss using the modified target Q-values and the Q-values from the current state-action pairs. If you're adding a penalty, it could be a function of the action taken or the resulting state.
4. Update the Network: Use the computed loss to perform a gradient descent step on the DQN's neural network parameters.

Since modifying the core DQN algorithm in MATLAB's Reinforcement Learning Toolbox might require extensive customization, consider implementing the critical parts of the DQN (such as the computation of Q-values, the loss function, and the update step) manually if the toolbox does not offer the flexibility you need for your specific modifications.