Markov Chain Model

Discrete state-space processes characterized by transition matrices

A discrete state-space Markov process, or Markov chain, is represented by a directed graph and described by a right-stochastic transition matrix P. The distribution of states at time t+1 is the distribution of states at time t multiplied by P. The structure of P determines the evolutionary trajectory of the chain, including asymptotics.

For an overview of the Markov chain analysis tools, see Markov Chain Modeling.

Functions

expand all

 dtmc Create discrete-time Markov chain mcmix Create random Markov chain with specified mixing structure
 asymptotics Determine Markov chain asymptotics isergodic Check Markov chain for ergodicity isreducible Check Markov chain for reducibility classify Classify Markov chain states lazy Adjust Markov chain state inertia subchain Extract Markov subchain
 hitprob Compute Markov chain hitting probabilities hittime Compute Markov chain hitting times redistribute Compute Markov chain redistributions simulate Simulate Markov chain state walks
 distplot Plot Markov chain redistributions eigplot Plot Markov chain eigenvalues graphplot Plot Markov chain directed graph simplot Plot Markov chain simulations

Topics

Discrete-Time Markov Chains

Markov chains are discrete-state Markov processes described by a right-stochastic transition matrix and represented by a directed graph.

Markov Chain Modeling

The dtmc class provides basic tools for modeling and analysis of discrete-time Markov chains. The class supports chains with a finite number of states that evolve in discrete time with a time-homogeneous transition structure.

Create and Modify Markov Chain Model Objects

Create a Markov chain model object from a state transition matrix of probabilities or observed counts, and create a random Markov chain with a specified structure.

Visualize Markov Chain Structure and Evolution

Visualize the structure and evolution of a Markov chain model by using dtmc plotting functions.

Work with State Transitions

This example shows how to work with transition data from an empirical array of state counts, and create a discrete-time Markov chain (dtmc) model characterizing state transitions.

Determine Asymptotic Behavior of Markov Chain

Compute the stationary distribution of a Markov chain, estimate its mixing time, and determine whether the chain is ergodic and reducible.

Compare Markov Chain Mixing Times

Compare the estimated mixing times of several Markov chains with different structures.

Identify Classes in Markov Chain

Programmatically and visually identify classes in a Markov chain.

Simulate Random Walks Through Markov Chain

Generate and visualize random walks through a Markov chain.

Compute State Distribution of Markov Chain at Each Time Step

Compute and visualize state redistributions, which show the evolution of the deterministic state distributions over time from an initial distribution.