Discrete-time_Markov_chain | TriviaQuestions.online

⭐ Core Definition: Discrete-time Markov chain

In probability, a discrete-time Markov chain (DTMC) is a sequence of random variables, known as a stochastic process, in which the value of the next variable depends only on the value of the current variable, and not any variables in the past. For instance, a machine may have two states, A and E. When it is in state A, there is a 40% chance of it moving to state E and a 60% chance of it remaining in state A. When it is in state E, there is a 70% chance of it moving to A and a 30% chance of it staying in E. The sequence of states of the machine is a Markov chain. If we denote the chain by $X_{0},X_{1},X_{2},...$ then $X_{0}$ is the state which the machine starts in and $X_{10}$ is the random variable describing its state after 10 transitions. The process continues forever, indexed by the natural numbers.

An example of a stochastic process which is not a Markov chain is the model of a machine which has states A and E and moves to A from either state with 50% chance if it has ever visited A before, and 20% chance if it has never visited A before (leaving a 50% or 80% chance that the machine moves to E). This is because the behavior of the machine depends on the whole history—if the machine is in E, it may have a 50% or 20% chance of moving to A, depending on its past values. Hence, it does not have the Markov property.

↓ Menu

>>>PUT SHARE BUTTONS HERE<<<

👉 Discrete-time Markov chain in the context of Markov chain Monte Carlo

In statistics, Markov chain Monte Carlo (MCMC) is a class of algorithms used to draw samples from a probability distribution. Given a probability distribution, one can construct a Markov chain whose elements' distribution approximates it – that is, the Markov chain's equilibrium distribution matches the target distribution. The more steps that are included, the more closely the distribution of the sample matches the actual desired distribution.

Markov chain Monte Carlo methods are used to study probability distributions that are too complex or too high dimensional to study with analytic techniques alone. Various algorithms exist for constructing such Markov chains, including the Metropolis–Hastings algorithm.

↓ Explore More Topics

In this Dossier

⭐ Core Definition: Discrete-time Markov chain
👉 Discrete-time Markov chain in the context of Markov chain Monte Carlo
Discrete-time Markov chain in the context of Equilibrium distribution
Discrete-time Markov chain in the context of Continuous-time Markov chain

Discrete-time Markov chain in the context of Equilibrium distribution

In probability theory and statistics, a Markov chain or Markov process is a stochastic process describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event. Informally, this may be thought of as, "What happens next depends only on the state of affairs now." A countably infinite sequence, in which the chain moves state at discrete time steps, gives a discrete-time Markov chain (DTMC). A continuous-time process is called a continuous-time Markov chain (CTMC). Markov processes are named in honor of the Russian mathematician Andrey Markov.

Markov chains have many applications as statistical models of real-world processes. They provide the basis for general stochastic simulation methods known as Markov chain Monte Carlo, which are used for simulating sampling from complex probability distributions, and have found application in areas including Bayesian statistics, biology, chemistry, economics, finance, information theory, physics, signal processing, and speech processing.

↑ Return to Menu

Discrete-time Markov chain in the context of Continuous-time Markov chain

A continuous-time Markov chain (CTMC) is a continuous stochastic process in which, for each state, the process will change state according to an exponential random variable and then move to a different state as specified by the probabilities of a stochastic matrix. An equivalent formulation describes the process as changing state according to the least value of a set of exponential random variables, one for each possible state it can move to, with the parameters determined by the current state.

An example of a CTMC with three states $\{0,1,2\}$ is as follows: the process makes a transition after the amount of time specified by the holding time—an exponential random variable $E_{i}$ , where i is its current state. Each random variable is independent and such that $E_{0}\sim {\text{Exp}}(6)$ , $E_{1}\sim {\text{Exp}}(12)$ and $E_{2}\sim {\text{Exp}}(18)$ . When a transition is to be made, the process moves according to the jump chain, a discrete-time Markov chain with stochastic matrix: