Stochastic Processes for Graduate Students

A stochastic process is a collection of random variables indexed by time, and it is how mathematicians model anything that evolves under uncertainty: queues, stock prices, epidemics, particle motion and network traffic. MS and PhD students in mathematics, statistics, finance and engineering meet these ideas in coursework and in thesis research. This guide walks through the core processes with worked examples you can follow line by line.

The Basic Setup

A stochastic process is a family \(\{X_t : t \in T\}\) of random variables on a probability space \((\Omega, \mathcal{F}, P)\). The index set \(T\) is usually time:

• Discrete time, \(T = \{0, 1, 2, \dots\}\): random walks, Markov chains, time series.
• Continuous time, \(T = [0, \infty)\): Poisson processes, Brownian motion, diffusions.

The state space is the set of values \(X_t\) can take. For a fixed outcome \(\omega\), the function \(t \mapsto X_t(\omega)\) is a sample path. The information available up to time \(t\) is captured by a filtration \(\mathcal{F}_t\), an increasing family of σ-algebras.

Markov Chains

A discrete-time process is a Markov chain if the future depends on the past only through the present:

$$ P(X_{n+1} = j \mid X_n = i, X_{n-1}, \dots, X_0) = P(X_{n+1} = j \mid X_n = i) = p_{ij}. $$

The numbers \(p_{ij}\) form the transition matrix \(P\), whose rows sum to 1. The \(n\)-step transition probabilities are the entries of \(P^n\) (the Chapman–Kolmogorov equations).

A distribution \(\pi\) is stationary if \(\pi P = \pi\). For an irreducible, aperiodic chain on a finite state space, \(\pi\) is unique and \(P^n\) converges to a matrix whose every row equals \(\pi\). Moreover the mean return time to state \(i\) is \(1/\pi_i\).

Worked Example 1: A Two-State Machine

A machine is either Working (state 1) or Broken (state 2), checked daily. A working machine breaks with probability 0.1; a broken one is repaired by the next day with probability 0.5. So

$$ P = \begin{pmatrix} 0.9 & 0.1 \\ 0.5 & 0.5 \end{pmatrix}. $$

Step 1: Two-step probabilities.

$$ P^2 = \begin{pmatrix} 0.9(0.9) + 0.1(0.5) & 0.9(0.1) + 0.1(0.5) \\ 0.5(0.9) + 0.5(0.5) & 0.5(0.1) + 0.5(0.5) \end{pmatrix} = \begin{pmatrix} 0.86 & 0.14 \\ 0.70 & 0.30 \end{pmatrix}. $$

So a machine working today is working in two days with probability 0.86.

Step 2: Stationary distribution. Solve \(\pi P = \pi\) with \(\pi_1 + \pi_2 = 1\). The first equation is \(\pi_1 = 0.9\pi_1 + 0.5\pi_2\), so \(0.1\pi_1 = 0.5\pi_2\), giving \(\pi_1 = 5\pi_2\). Then \(6\pi_2 = 1\):

$$ \pi = \left(\tfrac56, \tfrac16\right) \approx (0.833, 0.167). $$

Step 3: Interpret. In the long run the machine works about 83.3% of days. The mean time between visits to "Working" is \(1/\pi_1 = 1.2\) days.

Sanity check: rows of \(P^2\) are already moving towards \((0.833, 0.167)\), as the convergence theorem predicts.

Gambler's Ruin and Absorption

Many problems ask for the probability of hitting one boundary before another. In a random walk on \(\{0, 1, \dots, N\}\) that moves up with probability \(p\) and down with \(q = 1 - p\), let \(h_i\) be the probability of reaching \(N\) before 0 starting from \(i\). First-step analysis gives \(h_i = p\,h_{i+1} + q\,h_{i-1}\), with \(h_0 = 0\), \(h_N = 1\). The solution is

$$ h_i = \frac{1 - r^i}{1 - r^N}, \quad r = \frac{q}{p} \ne 1, \qquad h_i = \frac{i}{N} \text{ when } p = q. $$

Quick calculation: with \(p = 0.4\), \(i = 2\), \(N = 5\), we get \(r = 1.5\), \(r^2 = 2.25\), \(r^5 = 7.59375\), so

$$ h_2 = \frac{1 - 2.25}{1 - 7.59375} = \frac{-1.25}{-6.59375} \approx 0.190. $$

A slightly unfavourable game cuts the chance of success from the fair value \(2/5 = 0.4\) to about 0.19.

The Poisson Process

A counting process \(N(t)\) is a Poisson process with rate \(\lambda\) if \(N(0) = 0\), it has independent increments, and

$$ P\big(N(t+s) - N(s) = k\big) = e^{-\lambda t}\frac{(\lambda t)^k}{k!}, \qquad k = 0, 1, 2, \dots $$

Equivalently, the times between events are independent exponential random variables with mean \(1/\lambda\). The Poisson process is the standard model for arrivals at a bank counter, calls to a helpline or requests to a web server.

Worked Example 2: Customer Arrivals

Customers arrive at a bank branch as a Poisson process with rate \(\lambda = 3\) per hour.

Part (a). Probability of exactly 4 arrivals in 2 hours.

Step 1. The count in 2 hours is Poisson with mean \(\lambda t = 3 \times 2 = 6\).

Step 2.

$$ P(N(2) = 4) = e^{-6}\frac{6^4}{4!} = e^{-6}\cdot\frac{1296}{24} = 54\,e^{-6} \approx 54 \times 0.002479 \approx 0.134. $$

Part (b). Probability of at least 3 arrivals in 2 hours.

$$ P(N(2) \ge 3) = 1 - e^{-6}\left(1 + 6 + \frac{36}{2}\right) = 1 - 25e^{-6} \approx 1 - 0.062 = 0.938. $$

Part (c). Probability the first customer takes more than 30 minutes. The first arrival time \(T_1\) is exponential with rate 3 per hour, so

$$ P(T_1 > 0.5) = e^{-3 \times 0.5} = e^{-1.5} \approx 0.223. $$

Sanity check: the expected wait for the first customer is \(1/3\) hour, or 20 minutes, so a probability of about 22% of waiting over 30 minutes is reasonable.

Martingales

A process \(M_n\) adapted to \(\mathcal{F}_n\) with \(E|M_n| < \infty\) is a martingale if

$$ E[M_{n+1} \mid \mathcal{F}_n] = M_n. $$

It models a fair game: your expected future fortune equals your current fortune. The Optional Stopping Theorem says that, under conditions such as a bounded stopping time, \(E[M_\tau] = E[M_0]\). Applied to the fair random walk, it gives the gambler's ruin answer \(h_i = i/N\) in one line, because \(i = E[X_\tau] = N\cdot h_i + 0\cdot(1 - h_i)\).

Brownian Motion and Itô Calculus

Standard Brownian motion \(B_t\) satisfies \(B_0 = 0\), has independent increments, \(B_t - B_s \sim N(0, t - s)\), and has continuous paths. Its paths are continuous everywhere but differentiable nowhere, so ordinary calculus fails. Itô calculus replaces it, using the rule \((dB_t)^2 = dt\). Itô's lemma for a smooth \(f\) states

$$ df(B_t) = f'(B_t)\,dB_t + \tfrac12 f''(B_t)\,dt. $$

With \(f(x) = x^2\): \(d(B_t^2) = 2B_t\,dB_t + dt\). Taking expectations (the Itô integral has mean zero) gives \(E[B_t^2] = t\), matching the variance of \(N(0, t)\). This is the foundation of the Black–Scholes model and of stochastic differential equations in physics and biology.

Software for Simulation and Research

For thesis work, simulation is essential. R (packages such as markovchain) and Python (NumPy, SciPy) can simulate chains and estimate stationary distributions; MATLAB is also common in engineering departments. A short Python check of the stationary distribution:

python import numpy as np P = np.array([[0.9, 0.1], [0.5, 0.5]]) print(np.linalg.matrix_power(P, 50)[0]) # approx [0.8333 0.1667]

Did You Know?

• Andrey Markov introduced his chains around 1906 and later analysed the sequence of vowels and consonants in Pushkin's "Eugene Onegin".
• Louis Bachelier modelled stock prices with Brownian motion in his 1900 thesis, five years before Einstein's famous paper on the topic.
• Google's original PageRank algorithm is the stationary distribution of a Markov chain on web pages.

Key Takeaways

• Markov chains are governed by a transition matrix; their long-run behaviour is the stationary distribution \(\pi P = \pi\).
• First-step analysis and martingales solve hitting and ruin problems elegantly.
• Poisson processes model random arrivals, with exponential gaps between events.
• Brownian motion requires Itô calculus, where \((dB_t)^2 = dt\) creates the extra second-order term.

If you are working on stochastic models for coursework or a thesis, Amathify's experts and PhD scholars can provide step-by-step solutions and assignment help. Message us on WhatsApp or use the Ask a Question form.