dtmc

Create discrete-time Markov chain

Description

dtmc creates a discrete-time, finite-state, time-homogeneous Markov chain from a specified state transition matrix.

After creating a dtmc object, you can analyze the structure and evolution of the Markov chain, and visualize the Markov chain in various ways, by using the object functions.

Creation

Syntax

mc = dtmc(P)

mc = dtmc(P,'StateNames',stateNames)

Description

mc = dtmc(P) creates the discrete-time Markov chain object mc specified by the state transition matrix P.

mc = dtmc(P,'StateNames',stateNames) optionally associates the names stateNames to the states.

Input Arguments

P — State transition matrix

nonnegative numeric matrix

State transition matrix, specified as a numStates-by-numStates nonnegative numeric matrix.

P(i,j) is either the theoretical probability of a transition from state i to state j or an empirical count of observed transitions from state i to state j. P can be fully specified (all elements are nonnegative numbers), partially specified (elements are a mix of nonnegative numbers and NaN values), or unknown (completely composed of NaN values).

dtmc normalizes each row of P without any NaN values to sum to 1, then stores the normalized matrix in the property P.

Data Types: double

Properties

You can set writable property values when you create the model object by using name-value pair argument syntax, or after you create the model object by using dot notation. For example, for the two-state model mc, to label the first and second states Depression and Recession, respectively, enter:

mc.StateNames = ["Depression" "Recession"];

P — Normalized transition matrix

nonnegative numeric matrix

This property is read-only.

Normalized transition matrix, specified as a numStates-by-numStates nonnegative numeric matrix.

If x is a row vector of length numStates specifying a distribution of states at time t (x sums to 1), then x*P is the distribution of states at time t + 1.

NaN entries indicate estimable transition probabilities. The estimate function of msVAR treats the known elements of P as equality constraints during optimization.

Data Types: double

NumStates — Number of states

positive scalar

This property is read-only.

Number of states, specified as a positive scalar.

Data Types: double

StateNames — State labels

string(1:numStates) (default) | string vector | cell vector of character vectors | numeric vector

State labels, specified as a string vector, cell vector of character vectors, or numeric vector of length numStates. Elements correspond to rows and columns of P.

Example: ["Depression" "Recession" "Stagnant" "Boom"]

Data Types: string

Object Functions

dtmc objects require a fully specified transition matrix P.

Determine Markov Chain Structure

asymptotics	Determine Markov chain asymptotics
isergodic	Check Markov chain for ergodicity
isreducible	Check Markov chain for reducibility
classify	Classify Markov chain states
lazy	Adjust Markov chain state inertia
subchain	Extract Markov subchain

Describe Markov Chain Evolution

redistribute	Compute Markov chain redistributions
simulate	Simulate Markov chain state walks

Visualize Markov Chain

distplot	Plot Markov chain redistributions
eigplot	Plot Markov chain eigenvalues
graphplot	Plot Markov chain directed graph
simplot	Plot Markov chain simulations

Examples

Create Markov Chain Using Matrix of Transition Probabilities

Consider this theoretical, right-stochastic transition matrix of a stochastic process.

$$P=\left[\begin{array}{cccc}0.5& 0.5& 0& 0\\ 0.5& 0& 0.5& 0\\ 0& 0& 0& 1\\ 0& 0& 1& 0\end{array}\right].$$

Element $$P_{ij}$$ is the probability that the process transitions to state j at time t + 1 given that it is in state i at time t, for all t.

Create the Markov chain that is characterized by the transition matrix P.

P = [0.5 0.5 0 0; 0.5 0 0.5 0; 0 0 0 1; 0 0 1 0];
mc = dtmc(P);

mc is a dtmc object that represents the Markov chain.

Display the number of states in the Markov chain.

numstates = mc.NumStates

numstates = 4

Plot a directed graph of the Markov chain.

figure;
graphplot(mc);

Observe that states 3 and 4 form an absorbing class, while states 1 and 2 are transient.

Create Markov Chain Using Matrix of Observed Transition Counts

Consider this transition matrix in which element $$(i,j)$$ is the observed number of times state i transitions to state j.

$$P=\left[\begin{array}{cccc}16& 2& 3& 13\\ 5& 11& 10& 8\\ 9& 7& 6& 12\\ 4& 14& 15& 1\end{array}\right].$$

For example, $$P_{32}=7$$ implies that state 3 transitions to state 2 seven times.

P = [16 2  3  13;
     5  11 10 8;
     9  7  6  12;
     4  14 15 1];

Create the Markov chain that is characterized by the transition matrix P.

mc = dtmc(P);

Display the normalized transition matrix stored in mc. Verify that the elements within rows sum to 1 for all rows.

mc.P

ans = 4×4

    0.4706    0.0588    0.0882    0.3824
    0.1471    0.3235    0.2941    0.2353
    0.2647    0.2059    0.1765    0.3529
    0.1176    0.4118    0.4412    0.0294

sum(mc.P,2)

ans = 4×1

     1
     1
     1
     1

Plot a directed graph of the Markov chain.

figure;
graphplot(mc);

Label Markov Chain States

Consider the two-state business cycle of the US real gross national product (GNP) in [3] p. 697. At time t, real GNP can be in a state of expansion or contraction. Suppose that the following statements are true during the sample period.

• If real GNP is expanding at time t, then the probability that it will continue in an expansion state at time t + 1 is $$p_{11}=0.90$$.

• If real GNP is contracting at time t, then the probability that it will continue in a contraction state at time t + 1 is $$p_{22}=0.75$$.

Create the transition matrix for the model.

p11 = 0.90;
p22 = 0.75;

P = [p11 (1 - p11); (1 - p22) p22];

Create the Markov chain that is characterized by the transition matrix P. Label the two states.

mc = dtmc(P,'StateNames',["Expansion" "Contraction"])

mc = 
  dtmc with properties:

             P: [2x2 double]
    StateNames: ["Expansion"    "Contraction"]
     NumStates: 2

Plot a directed graph of the Markov chain. Indicate the probability of transition by using edge colors.

figure;
graphplot(mc,'ColorEdges',true);

Create Markov Chain from Random Transition Matrix

To help you explore the dtmc object functions, mcmix creates a Markov chain from a random transition matrix using only a specified number of states.

Create a five-state Markov chain from a random transition matrix.

rng(1); % For reproducibility
mc = mcmix(5)

mc = 
  dtmc with properties:

             P: [5x5 double]
    StateNames: ["1"    "2"    "3"    "4"    "5"]
     NumStates: 5

mc is a dtmc object.

Plot the eigenvalues of the transition matrix on the complex plane.

figure;
eigplot(mc)

This spectrum determines structural properties of the Markov chain, such as periodicity and mixing rate.

Create Markov Chain with Unknown Transition Matrix Entries

Consider a Markov-switching autoregression (msVAR) model for the US GDP containing four economic regimes: depression, recession, stagnation, and expansion. To estimate the transition probabilities of the switching mechanism, you must supply a dtmc model with an unknown transition matrix entries to the msVAR framework.

Create a 4-regime Markov chain with an unknown transition matrix (all NaN entries). Specify the regime names.

P = nan(4);
statenames = ["Depression" "Recession" ...
    "Stagnation" "Expansion"];

mcUnknown = dtmc(P,'StateNames',statenames)

mcUnknown = 
  dtmc with properties:

             P: [4x4 double]
    StateNames: ["Depression"    "Recession"    "Stagnation"    "Expansion"]
     NumStates: 4

mcUnknown.P

ans = 4×4

   NaN   NaN   NaN   NaN
   NaN   NaN   NaN   NaN
   NaN   NaN   NaN   NaN
   NaN   NaN   NaN   NaN

Suppose economic theory states that the US economy never transitions to an expansion from a recession or depression. Create a 4-regime Markov chain with a partially known transition matrix representing the situation.

P(1,4) = 0;
P(2,4) = 0;

mcPartial = dtmc(P,'StateNames',statenames)

mcPartial = 
  dtmc with properties:

             P: [4x4 double]
    StateNames: ["Depression"    "Recession"    "Stagnation"    "Expansion"]
     NumStates: 4

mcPartial.P

ans = 4×4

   NaN   NaN   NaN     0
   NaN   NaN   NaN     0
   NaN   NaN   NaN   NaN
   NaN   NaN   NaN   NaN

The estimate function of msVAR treats the known elements of mcPartial.P as equality constraints during optimization.

For more details on Markov-switching dynamic regression models, see msVAR.

Alternatives

You also can create a Markov chain object using mcmix.

References

[1] Gallager, R.G. Stochastic Processes: Theory for Applications. Cambridge, UK: Cambridge University Press, 2013.

[2] Haggstrom, O. Finite Markov Chains and Algorithmic Applications. Cambridge, UK: Cambridge University Press, 2002.

[3] Hamilton, J. D. Time Series Analysis. Princeton, NJ: Princeton University Press, 1994.

[4] Norris, J. R. Markov Chains. Cambridge, UK: Cambridge University Press, 1997.