Definition of transitions and rewards in a MDP environment for a reinforcement learning problem.

Good morning,

I have an environment with 50 states and 5 actions. From state 1, I transition to states 2 to 25 with action 1; from state 1, I transition to states 26 to 49 with action 2; from states 2 to 25, I transition to state 50 with action 3; from states 26 to 49, I transition to state 50 with action 4. I define the matrices MDP.T as follows:

MDP.T(1,2:25,1) = 1/24;

MDP.T(1,26:49,2) = 1/24;

MDP.T(2:25,50,3) = 1;

MDP.T(26:49,50,4) = 1;

MDP.T(50,50,5) = 0;

The code works only if I consider also these other transitions, which are not feasible in my example.

MDP.T(1,1,3) = 1;

MDP.T(1,1,4) = 1;

MDP.T(2:25,2:25,1) = 1/24;

MDP.T(2:25,2:25,2) = 1/24;

MDP.T(2:25,2:25,4) = 1/24;

MDP.T(2:25,2:25,5) = 1/24;

MDP.T(26:49,26:49,1) = 1/24;

MDP.T(26:49,26:49,2) = 1/24;

MDP.T(26:49,26:49,3) = 1/24;

MDP.T(26:49,26:49,5) = 1/24;

May you help me to understand how to define only feabile transitions?

Thank you in advance

[Emmanouil Tzorakoleftherakis]

Not really sure I understand what you mean. If you use the initial T matrix, are you getting any error? Can you share some code that reproduces the problem?

[Bianca Grieco]

Dear Sir, I am implementing a reinforcement learning method in MATLAB for my thesis. Prior to my previous message, I am writing to ask if you could provide me with information regarding both the maximum dimensions of each MDP.T matrix and the total number of MDP.T matrices (and consequently actions) that can be handled by the MATLAB RL toolbox. Thank you in advance for your response.

[Bianca Grieco]

Dear Sir Tzorakoleftherakis, have you had time to check the answer respect to the maximum dimension of the transition matrix? Thank you in advance.

Answer 1

[arushi]

Hi Bianca,

As per my understanding, it seems that you want to define a transition matrix for an MDP with 50 states and 5 actions, where certain transitions are possible based on the current state and the action taken. The code does not work if you do not consider the other transitions. In some MDP implementations, especially when using certain toolboxes or libraries, it's necessary to define transitions for all state-action pairs, even if some actions are not applicable in certain states. This is typically done to avoid undefined behavior in the MDP solver or to comply with the requirements of the specific MDP framework being used.

To handle this in the code, you can assign a self-transition (transition back to the same state) with a probability of 1 for actions that are not applicable in each state. This effectively means that if you take an infeasible action in a state, you stay in the same state with certainty.

Hope this helps.