Probability

can be

= L
7. The important notion of the variance of a random variable ~ indicates

the amount of scatter of the values

around E~.

of~

Definition 5. The variance (also called the dispersion) of the random variable (denoted by V ~) is

~

V~ = E(~- E~) 2 .

The number Since

CJ

= + jVf, is called the standard deviation.

we have

Clearly V~

~

V(a

0. It follows from the definition that

+ b~) =

where a and bare constants.

b 2 V~,

In particular, Va = 0, V(b~) = b 2 V~. Let ~ and 11 be random variables. Then V(~

+ rJ) = =

E((~- E~) Vi;+ V17

+ ('1-

+

E1J)) 2

2E((- E()(Y/- E17).

Write cov(~,

'1) =

E(~- E~)(IJ

- ErJ).

This number is called the covariance of~ and '1· If V ~ > 0 and V11 > 0, then

( ~ ) = cov(~, '1) JV~·VIJ p ,IJ is called the correlation coefficient of~ and '1· It is easy to show (see Problem 7 below) that if p(~, rJ) = ± 1, then~ and 11 are linearly dependent:

11 =a~+ b, with a > 0 if p(~, rJ) = 1 and a < 0 if p(~, rJ) = -1. We observe immediately that if~ and 11 are independent, so are and rJ - ErJ. Consequently by Property (5) of expectations, cov(~,

rJ) =

E(~

- EO· E(IJ - ErJ)

= 0.

~

-

E~

42

I. Elementary Probability Theory

Using the notation that we introduced for covariance, we have V(~

+ Yf)

= V~

+ VYf +

(10)

2cov(~, Yf);

if~ and Yf are independent, the variance of the sum of the variances, V(~ + Y/) = V~ + VYf.

~

+ Yf is equal to the sum (11)

It follows from (10) that (11) is still valid under weaker hypotheses than the independence of~ and Yf· In fact, it is enough to suppose that ~ and Yf are uncorrelated, i.e. cov(~, ry) = 0.

Remark. If ~ and Yf are uncorrelated, it does not follow in general that they are independent. Here is a simple example. Let the random variable o: take the values 0, n/2 and n with probability t. Then ~ = sin o: and Y/ = cos o: are uncorrelated; however, they are not only stochastically dependent (i.e., not independent with respect to the probability P): P{~ = 1, Yf = 1} = 0 ¥ ~ = P{~ = 1}P{ry = 1},

but even functionally dependent: ~ 2 + Yf 2 = 1. Properties (10) and (11) can be extended in the obvious way to any number of random variables: (12)

In particular, if ,;; 1 ,

••. ,

.;;n are pairwise independent (pairwise uncorrelated

is sufficient), then (13)

5. If ~ is a Bernoulli random variable, taking the values 1 and 0 with probabilities p and q, then

ExAMPLE

V~

= E(~- E~)2 = (~- p)2 = (1 - p)2 p

+ p2q =

pq.

It follows that if~ 1 ,

... , ~n are independent identically distributed Bernoulli random variables, and sn = ~ 1 + ... + ~n, then

vsn =

npq.

(14)

8. Consider two random variables ~ and Yf. Suppose that only ~ can be observed. If~ and Yf are correlated, we may expect that knowing the value of~ allows us to make some inference about the values of the unobserved variable Yf· Any function}= f(~) of~ is called an estimator for Yf· We say that an estimator f* = f*(~) is best in the mean-square sense if E(Yf - /*(0) 2 = inf E(Yf - /(~)) 2 • f

43

§4. Random Variables and Their Properties

Let us show how to find a best estimator in the class of linear estimators a + b~. We consider the function g(a, b) = E(17 - (a + b~)) 2 . Differentiating g(a, b) with respect to a and b, we obtain A.(~) =

og(a, b)= - 2E['7 -(a+ oa og(a, b) ob

= - 2E[('7-

b~)],

(a+ be))e],

whence, setting the derivatives equal to zero, we find that the best meansquare linear estimator is ..t*(~) = a* + b*~. where

a*

= E17 - b*E~,

b*

= cov(~, 17) v~

.

(15)

In other words, (16) The number E(17 - ..1.*(~)) 2 is called the mean-square error of observation. An easy calculation shows that it is equal to

Consequently, the larger (in absolute value) the correlation coefficient 17) between ~ and '1· the smaller the mean-square error of observation tl*. In particular, if IP(e. '7)1 = 1 then tl* = 0 (cf. Problem 7). On the other hand, if~ and '1 are uncorrelated (p(~, '7) = 0}, then A.*(~) = E'7, i.e. in the absence of correlation between ~ and '1 the best estimate of '1 in terms of~ is simply E17 (cf. Problem 4). p(~.

9. PROBLEMS 1. Verify the following properties of indicators IA = IA(w):

JAB= IA .JB, ]AuB =]A+ ]B- JAB·

The indicator of Uf=t Ai is 1- n~=l (1- lA,), the indicator of Ui=1 Ai is Df= 1 (1 - I A,), and the indicator o(D= 1 Ai is D= 1 I A,.

where A 6. B is the symmetric difference of A and B, i.e. the set (A\B) u (B\A).

44

I. Elementary Probability Theory

2. Let ~ 1 , ••• ,

~.be

independent random variables and

~min= min(~ 1 ,

.•. ,

~max= max(~!• ... ' ~.).

~.),

Show that n

P{~min;;:: x} = TIP{~;;;:: x}, i=l

< x} = TIPg; < x}.

P{~max

i=l

3. Let ~ ~> ... , ~. be independent Bernoulli random variables such that

0}

P{~; =

P{~;

where ~ is a small Show that

number,~

= 1} =A;~,

> 0, A; > 0.

P{~~ + · · · + ~. = P{~ 1

1 -A;~,

=

+ · · · + ~.

(J/;)~ + 0(~ 2),

1} =

> 1}

= 0(~ 2 ).

4. Show that inL 00
E(~ - a) 2 = V~.

-oo
5.

Let~ be a random variable with distribution function of F~(x), i.e. a point such that

F~(x)

and let me be a median

Show that inf

El~-

-ro
6. Let

P~(x)

=

P{~

= x} and

F~(x)

=

al

P(~::;;

= El~-

m.,l.

x}. Show that X -

b)

Pa~+b(x) = p~ ( -a- ' X-

b)

Fa~+b(x) = F~ ( -a-

for a rel="nofollow"> 0 and - oo < b < oo. If y ;;:: 0, then

F~z(y) Let~+ = max(~,

= F~

+Jy)- F ~ -JY) + P~( -JY).

0). Then

X< 0, 0, X> 0.

X=

45

§5. The Bernoulli Scheme. I. The Law of Large Numbers 7.

Let~ and 'I be random variables with V~ > 0, Vf1 > 0, and let p = p(~, 'I) be their correlation coefficient. Show that IpI ::;; 1. If IpI = 1, find constants a and b such that 'I = a~ + b. Moreover, if p = 1, then

(and therefore a > 0), whereas if p

-1, then

=

(and therefore a < 0). 8.

Let~ and 'I be random variables with coefficient p = p(~, f/). Show that

E~

= Ef/ = 0,

E max<e, f/ 2 )::;; 1 +

9. Use the equation (IndicatorofiV1 to deduce the formula P(B 0 )

=1-

S1

A;)

V~

= Vf/ = 1 and correlation

Jl=Pl.

=bY-

IA),

+ S 2 + · · · ± s. from Problem 4 of §1.

10. Let ~~> ... , ~. be independent random variables, cp 1 = cp 1 (~ 1 , .•• , ~k) and cp 2 = cp 2 ( ~k+ ~> ... , ~.), functions respectively of~ 1 , ••. , ~k and ~k+ 1 , ... , ~ •. Show that the random variables cp 1 and cp 2 are independent. 11. Show that the random variables~ 1 , ... , F~, ..... ~Jx 1 ,

••• ,

~.are

x.)

independent if and only if

= F~,(x 1 )

for all Xt. .•• , x., where F~, ..... ~Jx 1 , •.. , x.) =

P{~ 1

· · ·

F~"(x.)

::;;

x 1, ... , ~.::;; x.}.

12. Show that the random variable ~ is independent of itself (i.e., pendent) if and only if~ = const.

~

13. Under what hypotheses

~independent?

14.

on~

are the random

variables~

and sin

and

~

are inde-

and 'I be independent random variables and 'I # 0. Express the probabilities of the events P{~'l ::;; z} and P{~/'1 ::;; z} in terms of the probabilities P~(x) and Pq(y).

Let~

§5. The Bernoulli Scheme. Large Numbers

I. The Law of

1. In accordance with the definitions given above, a triple (Q, .szl, P)

.91

with =

Q

=

{w: w

{A: A c;; Q},

= (a 1 , ••. , a.), ai = 0, 1}, p(w)

=

pr.a;qn-r.a;

is called a probabilistic model of n independent experiments with two outcomes, or a Bernoulli scheme.

46

I. Elementary Probability Theory

In this and the next section we study some limiting properties (in a sense described below) for Bernoulli schemes. These are best expressed in terms of random variables and of the probabilities of events connected with them. by taking = ai, i = We introduce random variables 1 , ..• , 1, ... , n, where w = (a 1 , ••• , a,). As we saw above, the Bernoulli variables are independent and identically distributed:

e

e,

ei(w)

ei(w)

i = 1, ... , n.

ei

It is natural to think of as describing the result of an experiment at the ith stage (or at time i). Let us put S 0 (w) 0 and

=

k = 1, ... , n. As we found above, ES,

=

np and consequently

s,

(1)

E-=p.

n

In other words, the mean value of the frequency of "success", i.e. S,/n, coincides with the probability p of success. Hence we are led to ask how much the frequency Snfn of success differs from its probability p. We first note that we cannot expect that, for a sufficiently small e > 0 and for sufficiently large n, the deviation of S,/n from p is less than e for all w, i.e. that

wen.

(2)

In fact, when 0 < p < 1,

P{~" = 1} = P{~l = 1, ... , e, = 1} = p", P{~" = o} = P{el = o, ... , e, = o} = q", whence it follows that (2) is not satisfied for sufficiently small e > 0. We observe, however, that when n is large the probabilities of the events {S,/n = 1} and {S,/n = 0} are small. It is therefore natural to expect that the total probability of the events for which I [S,(w)/n] -Pi > e will also be small when n is sufficiently large. We shall accordingly try to estimate the probability of the event {w: i[S,(w)/n] -Pi > e}. For this purpose we need the following inequality, which was discovered by Chebyshev.

47

I. The Law of Large Numbers

§5. The Bernoulli Scheme.

Chebyshev's inequality. Let (Q, d, P) be a probability space and nonnegative random variable. Then

~ = ~(ro)

a

(3)

for all e > 0. PROOF.

We notice that

where I(A) is the indicator of A. Then, by the properties of the expectation,

which establishes (3).

Corollary. If~ is any random variable, we have for e > 0, P{l~l ~ e}

s;

El~l/e,

P{l~l ~ e} = P{~ 2 ~ e2 } s; eeje 2 , P{l~- E~l ~ e}

s; V~je 2 •

In the last of these inequalities, take~ =

Therefore

{I

s.- p p n

(4)

s.;n. Then using (4.14), we obtain

I }s ;ne- s ;4ne- , 1

pq

~e

2

2

(5)

from which we see that for large n there is rather small probability that the frequency S./n of success deviates from the probability p by more than e. For n ~ 1 and 0 s; k s; n, write

Then

P{l s.n - P I~ e} =

L

{k:i(k/n)- PI
P.(k),

and we have actually shown that

L

{k: l(k/n)- PI
P.(k) s;

pq

-2

ne

s;

1 4ne

-2'

(6)

48

I. Elementary Probability Theory

P.(k) I

~

0

np _ ____. ---v-np + ne

-----~

np - ne

n

Figure 6

i.e. we have proved an inequality that could also have been obtained analytically, without using the probabilistic interpretation. It is clear from (6) that

L

P,.(k)-+ 0,

n-+ oo.

(7)

{k:l(k/n)- PI ~e)

We can clarify this graphically in the following way. Let us represent the binomial distribution {P,.(k), 0 ~ k ~ n} as in Figure 6. Then as n increases the graph spreads out and becomes flatter. At the same time the sum of Pik), over k for which np - ne ~ k < np + ne, tends to 1. Let us think of the sequence of random variables S0 , S 1 , .•• , S,. as the path of a wandering particle. Then (7) has the following interpretation. Let us draw lines from the origin of slopes kp, k(p + e), and k(p - e). Then on the average the path follows the kp line, and for every e > 0 we can say that when n is sufficiently large there is a large probability that the point S,. specifying the position of the particle at time n lies in the interval [n(p - e), n(p + e)]; see Figure 7. We would like to write (7) in the following form: n-+ oo,

(8)

k(p +e)

I

s~

lkp

I I I I I I

k(p- e)

..........~,.---,.---..-- - - - - -- -+-+:

~

2

3

4

5

Figure 7

n

k

§5. The Bernoulli Scheme.

I. The Law of Large Numbers

49

However, we must keep in mind that there is a delicate point involved here. Indeed, the form (8) is really justified only if P is a probability on a space (0, d) on which infinitely many sequences of independent Bernoulli random variables ~ 1 , ~ 2 , .•. , are defined. Such spaces can actually be constructed and (8) can be justified in a completely rigorous probabilistic sense (see Corollary 1 below, the end of §4, Chapter II, and Theorem 1, §9, Chapter II). For the time being, if we want to attach a meaning to the analytic statement (7), using the language of probability theory, we have proved only the following. Let (n, .Jil<">, p), n ;;;:: 1, be a sequence of Bernoulli schemes such that

n, •.. ' a~">), al") = 0, 1}, .Jil ={A: A£ n}, p<">( w<">) =

pr.a~

q"- !:.a\"'

and

Sk">(w<"l) = ~\">(w<">)

+ · · · + ek">(w<">),

where, for n :s; 1, ~~nl, •.. , ~~nl are sequences of independent identically distributed Bernoulli random variables. Then

n --t oo. (9) Statements like (7)-(9) go by the name of James Bernoulli's law of large numbers. We may remark that to be precise, Bernoulli's proof consisted in establishing (7), which he did quite rigorously by using estimates for the "tails" of the binomial probabilities P"(k) (for the values of k for which l(k/n)- pi ;;;:: e). A direct calculation of the sum of the tail probabilities of the binomial distribution L{k:l
and

50

I. Elementary Probability Theory

2. The next section will be devoted to precise statements and proofs of these results. For the present we consider the question of the real meaning of the law of large numbers, and of its empirical interpretation. Let us carry out a large number, say N, of series of experiments, each of which consists of "n independent trials with probability p of the event C of interest." Let S~/n be the frequency of event C in the ith series and N, the number of series in which the frequency deviates from p by less than e: N, is the number of i's for which i(S~n)- pi ~e. Then N,/N "' P,

(10)

where P, = P{i(S!/n)- Pi~ e}. It is important to emphasize that an attempt to make (10) precise inevitably involves the introduction of some probability measure, just as an estimate for the deviation of Sn/n from p becomes possible only after the introduction of a probability measure P. 3. Let us consider the estimate obtained above,

p{l sn- pI~ e}= n

L

{k:l
Pn(k)

~ ~. 4ne

(11)

as an answer to the following question that is typical of mathematical statistics: what is the least number n of observations that is guaranteed to have (for arbitrary 0 < p < 1) (12) where ex is a given number (usually small)? It follows from (11) that this number is the smallest integer n for which

n

1 e ex

(13)

~-4 2 •

For example, if ex = 0.05 and e = 0.02, then 12 500 observations guarantee that (12) will hold independently of the value of the unknown parameter p. Later (Subsection 5, §6) we shall see that this number is much overstated; this came about because Chebyshev's inequality provides only a very crude upper bound for P{i(Sn/n)- Pi ~ e}. 4. Let us write

{ I

Sn(w) - p C(n, e) = w: -n-

I ~ e} .

From the law of large numbers that we proved, it follows that for every e > 0 and for sufficiently large n, the probability of the set C(n, e) is close to 1. In this sense it is natural to call paths (realizations) w that are in C(n, e) typical (or (n, e)-typical).

§5. The Bernoulli Scheme.

51

I. The Law of Large Numbers

We ask the following question: How many typical realizations are there, and what is the weight p(w) of a typical realization? For this purpose we first notice that the total number N(O.) of points is 2", and that if p = 0 or 1, the set of typical paths C(n, e) contains only the single path (0, 0, ... , 0) or (1, 1, ... , 1). However, if p =!,it is intuitively clear that "almost all" paths (all except those of the form (0, 0, ... , 0) or (1, 1, ... , 1)) are typical and that consequently there should be about 2n of them. It turns out that we can give a definitive answer to the question whenever 0 < p < 1; it will then appear that both the number of typical realizations and the weights p(w) are determined by a function of p called the entropy. In order to present the corresponding results in more depth, it will be helpful to consider the somewhat more general scheme of Subsection 2 of §2 instead of the Bernoulli scheme itself. Let (p 1 , p 2 , ••• , p,) be a finite probability distribution, i.e. a set of nonnegative numbers satisfying p 1 + · · · + p, = 1. The entropy of this distribution is r

L

H =-

p)npi,

(14)

i= 1

with 0 ·In 0 = 0. It is clear that H 2: 0, and H = 0 if and only if every pi, with one exception, is zero. The function f(x) = -x In x, 0 s x s 1, is convex upward, so that, as know from the theory of convex functions, f(xd

+ -~· + f(x,) s

!(x + ·; · + x} 1

Consequently

H = -

L' Pi In Pi s - r · P1 + r·· · + Pr · In (P1 + ·r· · + Pr) = In r.

i= 1

In other words, the entropy attains its largest value for p 1 = · · · = p, = 1/r (see Figure 8 for H = H(p) in the case r = 2). If we consider the probability distribution (p 1 , p 2 , ••• , p,) as giving the probabilities for the occurrence of events A 1 , A 2 , ... , A, say, then it is quite clear that the "degree of indeterminancy" of an event will be different for H(p)

I

p

2

Figure 8. The function H(p)

=

-pIn p- (1 - p)ln(l - p).

52

I. Elementary Probability Theory

different distributions. If, for example, p 1 = 1, p 2 = · · · = Pr = 0, it is clear that this distribution does not admit any indeterminacy: we can say with complete certainty that the result of the experiment will be A 1 . On the other hand, if p 1 = · · · = Pr = 1/r, the distribution has maximal indeterminacy, in the sense that it is impossible to discover any preference for the occurrence of one event rather than another. Consequently it is important to have a quantitative measure of the indeterminacy of different probability distributions, so that we may compare them in this respect. The entropy successfully provides such a measure of indeterminacy; it plays an important role in statistical mechanics and in many significant problems of coding and communication theory. Suppose now that the sample space is Q

= {w: w = (a 1 ,

..• ,

a.), ai

= 1, ... , r}

and that p(w) = p~1 (w) • · • pv; 0 and n = 1, 2, ... , let us put

I

.

}

{lvi(w) -n- -

Pi I :2: e} ,

{ I

vlw) - Pi < e, 1 = 1, ... , r . C(n, e) = w: -n~

It is clear that r P(C(n, e)) :2: 1 - i~t P

and for sufficiently large n the probabilities P{l(v;(w)/n)- p;l ;::: e} are arbitrarily small when n is sufficiently large, by the law of large numbers applied to the random variables ¢k(w)

= {01' ak: , ak

~·,

k = 1, ... , n.

-r z

Hence for large n the probability of the event C(n, e) is close to 1. Thus, as in the case n = 2, a path in C(n, e) can be said to be typical. If all Pi > 0, then for every w E Q p(w)

= exp{ -n

J

1

(-

vk~w) In Pk)}.

Consequently if w is a typical path, we have

v (w) Pk ) I L - ~k-In n r

k=t

(

I

- H ::::;; -

v (w) L I~kr

k=t

n

I

Pk In Pk ::::;; - e

L In Pk. r

k=t

It follows that for typical paths the probability p(w) is close to e-•H andsince, by the law of large numbers, the typical paths "almost" exhaust Q when n is large-the number of such paths must be of order e"H. These considerations lead up to the following proposition.

§5. The Bernoulli Scheme.

53

I. The Law of Large Numbers

Theorem (Macmillan). Let P; > 0, i = 1, ... , rand 0 < s < 1. Then there is an n0 = n0 (s; p 1 , ••• , p,) such that for all n > n0 (a) en
L

(c) P(C(n, s 1)) =

p(w)

-4

wE

1,

C(n, s 1); n-+ oo,

roe C(n,
where

s 1 is the smaller of sands/{ -2

kt/n Pk}

PROOF. Conclusion (c) follows from the law of large numbers. To establish

the other conclusions, we notice that if we C(n, s) then

k = 1, ... , r, and therefore

L

vk In Pk} < exp{ -n :::; exp{- n(H - ts)}.

p(w) = exp{-

L Pk ln Pk- s1n L ln Pd

Similarly p(w)

> exp{- n(H + ts)}.

Consequently (b) is now established. Furthermore, since P(C(n, s 1)) 2:: N(C(n, s 1 )) · we have

N ( C(n, S1 ))

:::;

min p(w), coeC(n,
P(C(n, sl)) 1 _ n(H+(l/2J•J . ( ) < -n(H+(l/2J•J - e mm pw e roeC(n,
and similarly N(C(n, s 1 )) 2:: P(C(n, s 1)) max p(w)

> P(C(n, s 1))en
meC(n,£ 1 )

Since P(C(n, s 1))-+ 1, n-+ oo, there is an n 1 such that P(C(n, s 1 )) for n > n 1, and therefore N(C(n, s 1 )) 2:: (1 - s) exp{n(H - t)}

= exp{n(H -

s)

+ (tns + ln(l

- s))}.

> 1- s

54

I. Elementary Probability Theory

Let n2 be such that

+ ln(1

!ne

- e) > 0.

for n > n2 • Then when n:;::: n0 = max(nto n2 ) we have N(C(n, e1))

:;::: en
•>.

This completes the proof of the theorem. 5. The law of large numbers for Bernoulli schemes lets us give a simple and elegant proof of Weierstrass's theorem on the approximation of continuous functions by polynomials. Letf = f(p) be a continuous function on the interval [0, 1]. We introduce the polynomials

which are called Bernstein polynomials after the inventor of this proof of Weierstrass's theorem. If ~ 1 , ••• , ~" is a sequence of independent Bernoulli random variables with P{~i = 1} = p, P{~i = 0} = q and S" = ~ 1 +···+~"'then

Ef(~) =

Bip).

Since the function.{ = f(p), being continuous on [0, 1], is uniformly continuous, for every e > 0 we can find b > 0 such that I f(x) - f(y)l :::;; e whenever lx- yl :::;; b. It is also clear that the function is bounded: If(x)l :::;; M < oo. Using this and (5), we obtain lf(p)- Bn(p)l

=I Jo : :;

[J(p)-

f(~) JC~lqn-k I

L

{k:i(k/n)- PiS~)

+

I f(p)

L

{k:i(kfn)-pi>~)

:::;; e + 2M

!(~) Jc~pkqn-k

-

n

lj(p)- J(~) IC~pkqn-k

L

n

{k:i ~J

C~lqn-k :::;;

Hence lim max n-+oo 0Sp:S1

I f(p)

- Bn(p)l = 0,

which is the conclusion of Weierstrass's theorem.

2M M e + - -2 = e + - -2 • 4n<5 2nb

§6. The Bernoulli Scheme.

II. Limit Theorems (Local, De Moivre-Laplace, Poisson)

55

6.PROBLEMS

1. Let ~and 1J be random variables with correlation coefficient p. Establish the following two-dimensional analog of Chebyshev's inequality: 1 P{l~- E~l ~ e~ or 111- EIJI ~ ~ 2 (1 + e

eJVtl}

.Ji="7).

(Hint: Use the result of Problem 8 of §4.)

2. Let f = f(x) be a nonnegative even function that is nondecreasing for positive x. Then for a random variable ~ with I~(w) I ~ C,

EfW- f(e) < P{l;:- E;:l >c.}< Ef(~- E~). f(C) "' "' f(e) In particular, if f(x) = x 2 ,

Eee C

2

2

3. Let ~ 1 , ••• ,

~.be

v~ ~ P{l~- E~l ~c.}~~-

a sequence of independent random variables with

V~; ~

p{ I~ 1 +...n +~. _E(~ 1 +n... +~.)I>- e} <-ne_£_

C. Then

2

(15)

(With the same reservations as in (8), inequality (15) implies the validity of the law of large numbers in more general contexts than Bernoulli schemes.) 4. "Let ~ 1 , ••. , ~.be independent Bernoulli random variables with P{e; = 1} = p > 0, P{e; = -1} = 1 - p. Derive the following inequality of Bernstein: there is a number a > 0 such that

p{ I~·- (2pwhere

s. =

~1

1)

I~ c.}~ 2e-ae2•,

+ · · · + ~. and e > 0.

§6. The Bernoulli Scheme. II. Limit Theorems (Local, De Moivre-.:.Laplace, Poisson) 1. As in the preceding section, let

s. = el + ... + e•.

Then

s.

E-= p, n

(1)

and by (4.14) (2)

56

I. Elementary Probability Theory

It follows from (1) that Sn/n ~ p, where the equivalence symbol ~ has been given a precise meaning in the law of large numbers in terms of an inequality for P{I(Sn/n)- Pi ~ e}. It is natural to suppose that, in a similar way, the relation (3) which follows from (2), can also be given a precise probabilistic meaning involving, for example, probabilities of the form

xeR\ or equivalently

(since ES. = np and VS. If, as before, we write

= npq). O~k~n,

for n

~

1, then

p{ IsnJ'iS,.

ESn

I~

x} =

I

Pn(k).

(4)

{k:j(k-np)/JiiNI:5x}

We set the problem of finding convenient asymptotic formulas, as n --+ oo, for Pn(k) and for their sum over the values of k that satisfy the condition on the right-hand side of (4). The following result provides an answer not only for these values of k (that is, for those satisfying Ik - np I = O(JnPq)) but also for those satisfying Ik - np I = o(npq) 213 •

Local Limit Theorem. Let 0 < p < 1 ; then P.(k) ~ uniformly fork such that

1

J2rffiM

e-
lk- npl = o(npq) 2 13 , i.e. as n--+ P.(k)

sup

{k:jk-npj,;cp(n)} - - - - - - - - -

1

j2nnpq where
=

o(npq) 213 •

(5)

e- (k- np)2/(2npq)

1

oo --+

0,

§6. The Bernoulli Scheme. II. Limit Theorems (Local, De Moivre-Laplace, Poisson)

57

The proof depends on Stirling's formula (2.6)

where R(n)

-+

0 as n-+ oo.

Then if n -+ oo, k

ck = n

-+

oo, n - k -+ oo, we have

n! k!(n- k)!

j2im e-"n"

1 + R(n)

J2nk · 2n(n - k) e-kkk. e-
J2.. Hl _~) ·(m~- ~r · 1 + e(n, k, n- k)

1

~

+ R(k))(1 + R(n- k))

where e = e(n, k, n- k) is defined in an evident way and e-+ 0 as n-+ oo, k -+ oo, n - k -+ oo. Therefore p J.k)

~ C,p'qo-' ~

J

l{l -

1 k(

p)n-k

k) (k)k( k)" k (1 2nn- 1 - 1- n n n n

+ e).

Write P= kjn. Then Pn(k)

1

(p)k(1 _ p)n-k 1- p (1

p

=

J2nnfi(1 - p)

=

1 exp{k ln J2nnfi(1 - p) P

=

1 exp{n ln + J2nnfi(1 - p) n P

~ + (n -

[~ ~

1

---;:::===:======::;= exp{-

J2nnfi(1 - fi)

nH(p)}(1

+

e)

~} · (1 + e)

k) ln 1 -

1- P

(1 - ~) ln 1 - P]} (1 + e) P 1-

n

+ e),

where

x

H(x) = xln-

p

1-x -p

+ (1- x)ln1-.

We are considering values of k such that lk- npl sequently p- p-+ 0, n-+ oo.

=

o(npq) 213 , and con-

58

I. Elementary Probability Theory

Since, for 0 < x < 1,

1- X X H'(x) =In- - I n - - , 1- p p

1 1 H"(x) = - + - - , 1 -·X X H"'( ) X

= -

=~

+ H'(p)(p-

(! + ~)

2 p

q

(p- p) 2

+ (1

+ (p

if we write H(fi) in the form H(p find that for sufficiently large n H(p) = H(p)

1

X2

p)

-

1

X )2 '

- p)) and use Taylor's formula, we

+ -!H"(p)(p-

p) 2

+ O(IP- pl 3 )

+ O(IP- Pl 3 ).

Consequently Pn(k)

=

~ 2n

1 (p- p) 2 exp{pq J2nnfi(1 - p)

+ nO(IP-

pl 3 )} (1

+ s).

Notice that _!!_ (p - p)z = _!!_ (~ - p)z 2pq n 2pq

= (k - np)z 2npq

Therefore l

Pn(k) =

~

e-J(Znpq)(l

+ s'(n, k, n-

k)),

where 1 + s'(n, k, n- k) = (1

+ s(n, k, n-

k))exp{n O(IP- Pl 3 )}

and, as is easily seen,

n --+ oo,

supls'(n, k, n- k)l--+ 0, if the sup is taken over the values of k for which lk- npl:::;; cp(n),

This completes the proof.

cp(n)

= o(npq) 213 •

p(l - p)

p(l - p)

§6. The Bernoulli Scheme. II. Limit Theorems (Local, De Moivre-Laplace, Poisson)

59

Corollary. The conclusion of the local limit theorem can be put in the folio~ equivalent form: For all x E R 1 such that x = o(npq) 116 , and for np + x.Jnpq an integer from the set {0, 1, ... , n}, Pinp

+ x~) ,..., ~ e-x2f2,

(7)

2nnpq

i.e. as n -+ oo, sup {x:lxl~l/l(n))

Pinp+ x~) _ 1 -+ 0, 1 -x2j2 ---;:==e

(8)

~

where t/J(n) = o(npq) 116.

With the reservations made in connection with formula (5.8), we can reformulate these results in probabilistic language in the following way:

P{Sn = k} ,..., p{ S n -np} ~

=x

1

~ 1

e-2f<2npq>,

"'~e

-x2j2

lk- npl

= o(npq) 213 , (9)

x = o(npq)lf6.

,

(10)

(In the last formula np + x~ is assumed to have one of the values 0, 1, ... , n.) If we put tk = (k- np)/~ and t:.tk = tk+ 1 - tk = 1/~, the preceding formula assumes the form

Sn - np = P{---"-==~

tk

}

tltk _ 12 12 ,..., - - e k '

(11)

fo

It is clear that t:.tk = 1/~-+ 0 and the set of points {tk} as it were "fills" the real line. It is natural to expect that (11) can be used to obtain the integral formula

P{a < Sn -

np

~

~

b} ,. ., _1_ fb e-x2Jz dx, fo

- oo < a ~ b < oo.

a

Let us now give a precise statement. 2. For - oo < a ~ b < oo let

Pn(a, b] =

L

Pn(np

+ x~),

a<x~b

where the summation is over those x for which np

+ x~ is an integer.

60

I. Elementary Probability Theory

It follows from the local theorem (see also (11)) that for all tk defined by k = np + tkfipq and satisfying Itk I ~ T < oo,

Pn(np

"PI.f) + tky'r.:=

=

2 /2 ll.tk M: e-tk [1 v 2n

+ e(tk, n)],

(12)

where

n --+ oo.

sup le(tk, n)l--+ 0,

(13)

itkiST

Consequently, if a and b are given so that - T

~ a ~

b ~ T, then

where

L

R~ll(a, b)=

ll.tk

e-l~/2-

a
L

R~2 >(a, b) =

_1_ rb e-x2j2 dx,

j2n Ja

~tk

2

e(tk> n) - - e-tki 2 .

fo

a
From the standard properties of Riemann sums, n --+ oo.

IR~1 >(a, b)l--+ 0,

sup

(15)

-TsasbsT

It also clear that JR~2 >(a, b)l

sup -TsasbsT

~

sup Ie(tk> n)l itkiST

x [

~ JT

y'

2n

-T

e-x 212 dx

+

sup

-TsasbsT

JR~ll(a, b)JJ --+ 0,

(16)

where the convergence of the right-hand side to zero follows from (15) and from

_1_ fT e-x2!2 dx < _1_ f"' e-x2/2 dx = 1

Jbi

-T

-

Jbi

-oo

the value of the last integral being well known.

'

(17)

61

§6. The Bernoulli Scheme. II. Limit Theorems (Local, De Moivre-Laplace, Poisson)

We write


Jin

fx -oo

e- 1212 dt.

Then it follows from (14)-(16) that

n --+ oo.

IPn(a, b] - ((b)-
sup

(18)

-T:Sa:Sb:ST

We now show that this result holds for T = oo as well as for finite T. By (17), corresponding to a given e > 0 we can find a finite T = T(e) such that -1-

IT

for-T

i

> 1-

e-x>f 2 dx

e.

(19)

According to (18), we can find anN such that for all n >Nand T = T(e) we have sup -T:Sa:Sb:ST

IPn(a, b] - ((a))l <

i

e.

(20)

It follows from this and (19) that PnC- T, T]

> 1-

t e,

and consequently Pn(- oo, T]

+ Pn(T, oo)

~

te,

where Pn(- 00, T] = lims!-oo PnCS, T] and Pn(T, oo) = limstoo Pn(T, S]. Therefore for - oo ~ a ~ - T < T ~ b ~ oo,

IPn(a, b] -

- 1-

foe

fb e-x>/2 dx I a

~ IPn(- T, T]- ~IT

+I

v 2n

Pn(a,- T]-

+ 1 foo +-foe ~ !e 4

fo

Pn(-oo,- T]

T

2

e-x 12

e-x>j 2

dx

I

dx

I+ I

-T

i-T e-x>f2

Pn(T, b]-

fo s:

e-x>f2

dx

I

+v~f-T + 1 1 1 1 + + + 4 2 8 8

dx ~ -e

2n - oo -e

e-x 212

-e

dx

PnCT, oo)

-e = e.

By using (18) it is now easy to see that Pn(a, b] tends uniformly to (b)
62

I. Elementary Probability Theory

De Moivre-Laplace Integral Theorem. Let 0 < p < 1, Pn(a, b] =

L

a<xsb

Pn(np

+ xJnpq),

Then 'P"(a,b]-

sup

-cosa
~iba e-x 12 dx,--+O, 2

y 2n

n--+ oo.

(21)

With the same reservations as in (5.8), (21) can be stated in probabilistic language in the following way: sup

-cosa
ESn IP{a < snv-~ vsn

::;; b} -

~ ib e-x212 dx I-+ 0,

v 2n

n --+ oo.

a

It follows at once from this formula that

P{A < S" :s; B} -

- np) -
--+

0,

(22)

as n--+ oo, whenever - oo :s; A < B :s; oo. A true die is tossed 12 000 times. We ask for the probability P that the number of 6's lies in the interval (1800, 2100]. The required probability is

EXAMPLE.

-

P-

(l)k(5)L: c 12 ooo 1800
12 000- k.

An exact calculation of this sum would obviously be rather difficult. However, if we use the integral theorem we find that the probability P in question is (n = 12 000, p = i, a = 1800, b = 2100) 2100- 2000) ( 1800- 2000)
<11(2.449) -
~

0.992,

where the values of <11(2.449) and
xJnM

theintegraltheoremsaysthatP"(a, b] = P{aj;lpq < S"- np :s; bj;lpq} = P{np + aj;lpq < Sn :s; np + bj;lpq} is closely approximated by the integral (1/j2ic)J! e-x2f2 dx.

§6. The Bernoulli Scheme. II. Limit Theorems (Local, De Moivre-Laplace, Poisson)

P.(np

63

+ xJniq)

X

0

Figure9

We write

Then it follows from (21) that sup

-oo:S:x:Soo

n- oo.

IFn(x) -
(23)

It is natural to ask how rapid the approach to zero is in (21) and (23), as n- oo. We quote a result in this direction (a special case of the BerryEsseen theorem: see §6 in Chapter Ill): sup

JFn(x) -
-oo:Sx:Soo

+ q2 C::: v npq

p2

(24)

It is important to recognize that the order of the estimate (1/.}niq) cannot be improved; this means that the approximation of Fn(x) by (x) can be poor for values of p that are close to 0 or 1, even when n is large. This suggests the question of whether there is a better method of approximation for the probabilities of interest when p or q is small, something better than the normal approximation given by the local and integral theorems. In this connection we note that for p =!,say, the binomial distribution {Pn(~)} is symmetric (Figure 10). However, for small p the binomial distribution is asymmetric (Figure 10), and hence it is not reasonable to expect that the normal approximation will be satisfactory. P.(k)

P.(k)

0.3

p = !, n

0.2

= 10

\

I

'

0.1 2

4

6

0.3 0.2

1 ..,.

I

p = ;\, n

= 10

I

0.1 8

2

10

Figure 10

4

6

8

10

64

I. Elementary Probability Theory

4. It turns out that for small values of p the distribution known as the Poisson distribution provides a good approximation to {P,(k)}. Let k_ 0 1 P,(k) = ,p q ' - ' '· · ·' n, 0, k = n + 1, n + 2, . . ,

{ck kn-k

and suppose that pis a function p(n) of n.

Poisson's Theorem. Let p(n) -+ 0, n -+ oo, in such a way that np(n)-+ A, where A.> 0. Then fork= 1, 2, ... , P,(k)-+ nk, where

A.ke-;. nk=~,

n-+ oo,

(25)

k = 0, 1, ....

(26)

The proof is extremely simple. Since p(n) = (A./n) for a given k = 0, 1, ... and sufficiently large n,

P,(k) =

+ o(1/n) by hypothesis,

C~pkqn-k

But

n(n- l)···(n- k =

+

1)[~ + o(~)r

n(n - 1) · · ~~n - k

and

[ 1 --;:;A.

+ 1\A. + o( 1W-+A.\

+ o ( -;:;l)]n-k -+ e-;.,

n-+ oo,

n-+ oo,

which establishes (25). The set of numbers {nk, k = 0, 1, ... } defines the Poisson probability distribution (nk ~ 0, Lk'=o nk = 1). Notice that all the (discrete) distributions considered previously were concentrated at only a finite number of points. The Poisson distribution is the first example that we have encountered of a (discrete) distribution concentrated at a countable number of points. The following result of Prohorov exhibits the rapidity with which P,(k) converges tonk as n-+ oo: if np(n) = A. > 0, then (27) (For the proof of (27), see §7 of Chapter III.)

65

§6. The Bernoulli Scheme. II. Limit Theorems (Local, De Moivre-Laplace, Poisson)

5. Let us return to the De Moivre-Laplace limit theorem, and show how it implies the law of large numbers (with the same reservation that was made in connection with (5.8)). Since

it is clear from (21) that when e > 0 P

{I s I } ___.!!_-

n

p :::;: e - -1-

s·..(iljpq

fo - ,,;nrpq

e-x>J 2 dx--+

0,

n --+ oo,

(28)

whence n--+ oo,

which is the conclusion of the law of large numbers. From (28)

P{l SnpI : :;: e},..., f'..fnipq n fo-.,;nrpq _1_

e-x>/2

dx,

n--+ oo,

(29)

whereas Chebyshev's inequality yielded only

It was shown at the end of §5 that Chebyshev's inequality yielded the estimate 1

n:<::-42 e a

for the number of observations needed for the validity of the inequality

Thus withe= 0.02 and a= 0.05, 12 500 observations were needed. We can now solve the same problem by using the approximation (29). We define the number k(a) by 1 Jk(IX)

--

fo

Since e .J(il!Pq)

e-x>f 2

dx = 1 - a.

-k(<X)

; : : 2e.jn, if we define n as the smallest integer satisfying 2eJn ;;::: k(a)

(30)

we find that (31)

66

I. Elementary Probability Theory

We find from (30) that the smallest integer n satisfying k 2 (rx)

n 2 4s2

guarantees that (31) is satisfied, and the accuracy of the approximation can easily be established by using (24). Taking s = 0.02, rx = 0.05, we find that in fact 2500 observations suffice, rather than the 12 500 found by using Chebyshev's inequality. The values of k(rx) have been tabulated. We quote a number of values of k(rx) for various values of rx: 0(

k(a)

0.50 0.3173 0.10 0.05 0.0454 O.ol 0.0027

0.675 1.000 1.645 1.960 2.000 2.576 3.000

6. The function (x) =

_1_

fo

fx

e-t2/2

dt,

(32)

-oo

which was introduced above and occurs in the De Moivre-Laplace integral theorem, plays an exceptionally important role in probability theory. It is known as the normal or Gaussian distribution on the real line, with the (normal or Gaussian) density

-3 -2 -1

°0.67I 1 1.96 2.58

X

Figure 11. Graph of the normal probability density cp(x).

§6. The Bernoulli Scheme. II. Limit Theorems (Local, De Moivre-Laplace, Poisson)

0.9--

67

-1

I I I

I I I I I

1111 I

-3 -2 -1

°111111

q1J

I

2

3

4

X

0.25111 I 0.5211/ I

o.67

1

:

0.841 I 1.28i

Figure 12. Graph of the normal distribution Cl>(x).

We have already encountered (discrete) distributions concentrated on a finite or countable set of points. The normal distribution belongs to another important class of distributions that arise in probability theory. We have mentioned its exceptional role; this comes about, first of all, because under rather general hypotheses, sums of a large number of independent random variables (not necessarily Bernoulli variables) are closely approximated by the normal distribution (§4 of Chapter III). For the present we mention only some of the simplest properties of cp(x) and cD(x), whose graphs are shown in Figures 11 and 12. The function cp(x) is a symmetric bell-shaped curve, decreasing very rapidly with increasing lxl: thus cp(1) = 0.24197, cp(2) = 0.053991, cp(3) =

0.004432, cp(4)

=

0.000134, cp(5)

=

0.000016. Its maximum is attained at

x = 0 and is equal to (2nr 112 ~ 0.399. The curve cD(x) = (1/fo) J~ao e- 1212 dt approximates 1 very rapidly as x increases: C1>(1) = 0.841345, cD(2) = 0.977250, Cl>(3) = 0.998650, Cl>(4) = 0.999968, Cl>(4, 5) = 0.999997. For tables of cp(x) and cD(x), as well as of other important functions that are used in probability theory and mathematical statistics, see [A1].

7. PROBLEMS 1. Let n = 100, p = / 0 ,

120 , 130 , 1~, 150 • Using tables (for example, those in [A1]) of the binomial and Poisson distributions, compare the values of the probabilities

P{lO <

s,oo ~ 12},

P{33 < S100

~

35},

P{20 < S100 P{40 <

P{so < s,oo

~

22},

s,oo ~ 42},

~52}

with the corresponding values given by the normal and Poisson approximations.

68

I. Elementary Probability Theory

2. Letp = !andZ. = 2S.- n(theexcessofl'soverO 'sinntrials).Showthat suplfoP{Z2 •

=

j } - e-P/ 4 "1--> 0,

n-->

oo.

3. Show that the rate of convergence in Poisson's theorem is given by

§7. Estimating the Probability of Success in the Bernoulli Scheme 1. In the Bernoulli scheme (Q, d, P) with 0, 1) }, d = A: A ~ Q },

n=

{w:w = (xb ... ' x.), X;=

p(w) = p'f.x;qn-'f.x;,

we supposed that p (the probability of success) was known. Let us now suppose that p is not known in advance and that we want to determine it by observing the outcomes of experiments; or, what amounts to the same thing, by observations of the random variables ~1> ••• , ~.,where ~;(co) = x;. This is a typical problem of mathematical statistics, and can be formulated in various ways. We shall consider two of the possible formulations: the problem of estimation and the problem of constructing confidence intervals. In the notation used in mathematical statistics, the unknown parameter is denoted bye, assuming a priori that e belongs to the set 0 = [0, 1]. We say that the set (Q, d, P8 ; 8 E 01 with p8(w) = er. x'(l - 8)"-r.x, is a probabilistic-statistical model (corresponding to" n independent trials" with probability of "success" E 0), and any function T, = T,(w) with values in 0 is called an estimator. If s. = ~ 1 + · · · + ~. and T: = S./n, it follows from the law of large numbers that is consistent, in the sense that (s > 0)

e

T:

P8{/'J!'-

8/

~

s}--+ 0,

n--+ oo.

Moreover, this estimator is unbiased: for every

E0 T: =

e,

(1)

e (2)

where E8 is the expectation corresponding to the probability P8 • The property of being unbiased is quite natural: it expresses the fact that any reasonable estimate ought, at least "on the average," to lead to the desired result. However, it is easy to see that is not the only unbiased estimator. For example, the same property is possessed by every estimator

T:

T,=

b1 x 1

+ · · · + b"", x n

69

§7. Estimating the Probability of Success in the Bernoulli Scheme

where b 1 + · · · + bn = n. Moreover, the law of large numbers (1) is also satisfied by such estimators (at least if Ib;l ~ K < oo; see Problem 2(b), §3, Chapter Ill) and so these estimators T, are just as "good" as In this connection there arises the question of how to compare different unbiased estimators, and which of them to describe as best, or optimal. With the same meaning of "estimator," it is natural to suppose that an estimator is better, the smaller its deviation from the parameter that is being estimated. On this basis, we call an estimator f;, efficient (in the class of unbiased estimators T,) if,

r:.

Oe9,

(3)

where V9 T, is the dispersion ofT,, i.e. E9(T,- 0) 2 • Let us show that the estimator considered above, is efficient. We have

r:,

*_

(Sn) _ VSn _ n0(1 9

VoTn-VoHence to establish that

--2--

n

2

n

0) _ 0(1 - 0)

n

-

n

(4)

r: is efficient, we have only to show that . f v T.

m

()

n ~

0(1 - 0) • n

(5)

This is obvious for 0 = 0 or 1. Let 0 E (0, 1) and

Po(X;) = ox•(l - 0)1-x;. It is clear that n

Po(w) =

01 Po(x;).

i=

Let us write L 9(w) = In p6 (w).

Then

Lo(w)

.

= In 0. LX; + ln(l - O)L(l -

and

oL (w)

----ae- = 9

L(x; - 0) 0(1 - 0)

Since (jJ

and since Tn is unbiased,

(} =Eo T, = L T,(w)p (w). 9

(jJ

X;)

70

I. Elementary Probability Theory

After differentiating with respect to (), we find that

Therefore

1 = E{(1;. _ ()) oL;~w)] and by the Cauchy-Bunyakovskii inequality,

whence 1

2

(6)

Eo[T, - ()] ~ In(())' where I.(())=

[aL;~w)T

is known as Fisher's information. From (6) we can obtain a special case of the Rao-Cramer inequality for unbiased estimators T,: (7)

In the present case

I (()) = E [oLo(w)]z = n

0

()()

- O)]z

E [L(~; 8 ()(1 _ ())

n()(l - ()) [()(1 - ())JZ

n ()(1 - ()) '

which also establishes (5), from which, as we already noticed, there follows the efficiency of the unbiased estimator = Sn/n for the unknown parameter e.

r:

r:

2. It is evident that, in considering as a pointwise estimator for(), we have introduced a certain amount of inaccuracy. It can even happen that the numerical value of calculated from observations of x~> ... , xn differs rather severely from the true value e. Hence it would be advisable to determine the size of the error. It would be too much to hope that r:( w) differs little from the true value () for all sample points w. However, we know from the law of large numbers

r:

71

§7. Estimating the Probability of Success in the Bernoulli Scheme

that for every {J > 0 and for sufficiently large n, the probability of the event {I w)I > {J} will be arbitrarily small. By Chebyshev's inequality

o- r:c

P {IO _ T*l 6

n

b} <

>

-

v6[J2r:

=

oon[J2- O)

and therefore, for every A > 0,

P9{1o- r:1

~ AJ0(1;

O)} ~ 1- ;

2 •

If we take, for example, A = 3, then with P6 -probability greater than 0.888 (1 - (1/3 2 ) = ! ~ 0.8889) the event

10- r:1

~

3)0(1~

O)

will be realized, and a fortiori the event

10since 0(1 - 0) ~ Therefore P6

r:1 ~ 3;:, 2y n

i.

{1 o- r:1 ~ 2~} =

Po{

r: - 2~ ~ o~ r: + 2~} ~ 0.8888.

In other words, we can say with probabilipr greater than 0.8888 that the exact value of 0 is in the interval [T~ - (3/2y'n), T~ + (3/2,fn)]. This statement is sometimes written in the symbolic form

o~ r:

±if.

c~ 88%),

where " ~ 88%" means "in more than 88% of all cases." The interval [T: - (3/2.j1i), + (3/2.j1i)] is an example of what are called confidence intervals for the unknown parameter.

r:

Definition. An interval of the form

where

t/1 1( w) and t/1 i

w) are functions of sample points, is called a corifidence {J (or of significance level {J) if

interval of reliability 1 -

P11 {1/1 1 (w) for all 0 E 9.

~

0

~

t/1 2 (w)}

~

1- b.

72

I. Elementary Probability Theory

The preceding discussion shows that the interval

[ Tn* -

A Tn* + Jn A J Jn' 2

2

has reliability 1 - (1/A 2 ). In point of fact, the reliability of this confidence interval is considerably higher, since Chebyshev's inequality gives only crude estimates of the probabilities of events. To obtain more precise results we notice that

{w: /8- r:1 s ;.,)8(1: 8)} = {w: t/1 (T:, n):::; 8 s 1/Jz(T:, n)}, 1

where t/1 1 = t/1 1(T:,n) and t/1 2 = t/1 2 (T:,n) are the roots of the quadratic equation (8 - T!) 2

= ;.,z 8(1 - 8),

n which describes an ellipse situated as shown in Figure 13. Now let

Then by (6.24) sup /F(J(x)-
s

1

1

v n8(1 - 8)

Therefore if we know a priori that 0 < Ll ::::;; 8 ::::;; 1 - Ll < 1,

where Ll is a constant, then sup /F 0(x)-
()

Figure 13

s

1

r:

Llv n

73

§7. Estimating the Probability of Success in the Bernoulli Scheme

2:: (2<1>(A) - 1) -

2

r:..

~vn

Let A* be the smallest A for which (2<1>(A) - 1) -

2

r:.

~vn

2:: 1 - <5*,

where <5* is a given significance level. Putting <5 = <5* - (2/~Jn), we find that A* satisfies the equation

For large n we may neglect the term 2/~Jn and assume that A* satisfies ( A*) = 1 -

~ <5*.

In particular, if A.* = 3 then c5* = 0.9973 .... Then with probability approximately 0.9973

r:: -

3J8(1

~ 8) ~ () ~ r:: + 3J8(1 ~ 8)

(8)

or, after iterating and then suppressing terms of order O(n- 3 14 ), we obtain

T,.* - 3

T.*(1 - T.*) n

n

n

~ () ~

T,.*

+3

T.n*(1 - T.n*)

n

(9)

Hence it follows that the confidence interval

[r:- 2~, r: + 2~]

(10)

has (for large n) reliability 0.9973 (whereas Chebyshev's inequality only provided reliability approximately 0.8889). Thus we can make the following practical application. Let us carry out a large number N of series of experiments, in each of which we estimate the parameter () after n observations. Then in about 99.73% of the N cases, in each series the estimate will differ from the true value of the parameter by at most 3/2Jn. (On this topic see also the end of §5.)

74

I. Elementary Probability Theory

3. PROBLEMS 1. Let it be known a priori that 8 has a value in the set 8 estimator fore, taking values only in E>o.

0

c;; [0, 1]. Construct an unbiased

2. Under the hypotheses of the preceding problem, find an analog of the Rao-Cramer inequality and discuss the problem of efficient estimators. 3. Under the hypotheses of the first problem, discuss the construction of confidence intervals for e.

§8. Conditional Probabilities and Mathematical Expectations with Respect to Decompositions 1. Let (0, d, P) be a finite probability space and

a decomposition ofO (D; Ed, P(D;) > 0, i = 1, ... , k, and D 1 + · · · + Dk = 0). Also let A be an event from d and P(AID;) the conditional probability of A with respect to D;. With a set of conditional probabilities {P(A ID;), i = 1, ... , k} we may associate the random variable n(w)

=

k

L

P(A ID;)lv,(w)

(1)

i= 1

(cf. (4.5)), that takes the values P(AID;) on the atoms of D;. To emphasize that this random variable is associated specifically with the decomposition fl2, we denote it by P(Aifl2)

or

P(Aifl2)(w)

and call it the conditional probability of the event A with respect to the de-

composition fl2.

This concept, as well as the more general concept of conditional probabilities with respect to au-algebra, which will be introduced later, plays an important role in probability theory, a role that will be developed progressively as we proceed. We mention some of the simplest properties of conditional probabilities: P(A

+ B I fl2)

= P(A I fl2)

+ P(B Ifl2);

(2)

if fl2 is the trivial decomposition consisting of the single set 0 then P(A I0)

=

P(A).

(3)

75

§8. Conditional Probabilities and Mathematical Expectations

The definition of P(A i.@) as a random variable lets us speak of its expectation; by using this, we can write the formula (3.3) for total probability in the following compact form: EP(Ai.@) = P(A).

(4)

In fact, smce P(AI.@)

=

k

L P(AID;)Iv;(m),

i= 1

then by the definition of expectation (see (4.5) and (4.6)) EP(Ai.@)

k

L

=

P(AID;)P(D;)

i= 1

=

k

L

P(AD;)

i= 1

= P(A).

Now let '1 = 1J(m) be a random variable that takes the values y 1, ... , Yk with positive probabilities: k

1J(m) =

L Yivim),

j= 1

where Di = {m: 1J(m) = y). The decomposition.@,= {D 1, ... , Dk} is called the decomposition induced by 'I· The conditional probability P(A !.@,) will be denoted by P(AI'I) or P(AI'I)(m), and called the conditional probability of A with respect to the random variable '1· We also denote by P(AI'I =yi) the conditional probability P(AIDi), where Di = {m: 1J(m) = Yi}. Similarly, if 1J 1, 1J 2 , ••• , '1m are random variables and .@,,,, 2 , ... ,,'" is the decomposition induced by '11> 1J 2 , ••• , '1m with atoms DY!.Y2· .. ··Ym

= {m: '11(m) = Y1• · · ·' 'lm(m) = Ym},

then P(AID~ •. ~, ..... ~~) will be denoted by P(AI17 1 , '7 2 , ••• , '1m) and called the conditional probability of A with respect to '11> 1] 2 , ••• , '1m· 1. Let ~ and 11 be independent identically distributed random variables, each taking the values 1 and 0 with probabilities p and q. For k = 0, 1, 2, let us find the conditional probability P(~ + '1 = ki'l) of the event A = {m: ~ + '1 = k} with respect to '1· To do this, we first notice the following useful general fact: if ~ and 1J are independent random variables with respective values x and y, then

ExAMPLE

P(~

+ 1J =

zi1J = y) = P(~

+y

= z).

(5)

In fact,

P(~ + = fi'l = y) = P(~ + '1 = z, '1 = y) '1

P('l = y)

=

P(~

= P(~

+ y = Z,1J = y) P('l = y) + y = z).

P(~

+ y = z)P(y = 'I) P('l

= y)

76

I. Elementary Probability Theory

Using this formula for the case at hand, we find that P(~

+ 17 =

+ 17 = ki11 = O)J{,=OJ(w) + P(~ + 11 = kl11 = 1)J{.,= 11(w) = P(~ = k)I!,=O!(w) + P{~ = k-

ki17) = P(~

1}J{,= 11(w).

Thus (6)

or equivalently q(1 - 11),

P(~

+ 11 = kl17) = { p(1

- 17)

+ q17,

P11,

2. Let ~ =

~(w)

k = 0, k = 1, k = 2,

be a random variable with values in the set X = { x 1 ,

(7)

... ,

xn}:

l

~

=

L xJ4w),

j= 1

and let f!} = {D 1 , .•. , Dk} be a decomposition. Just as we defined the expectation of~ with respect to the probabilities P(A),j = 1, ... , l. l

Ee = L

xjP(Aj),

(8)

j= 1

it is now natural to define the conditional expectation of ~ with respect to f!} by using the conditional probabilities P(Ail f!}), j = 1, ... , l. We denote this expectation by EW f!}) or E(~l f!}) (w), and define it by the formula l

E(~l f!}) =

L xiP(Ail f!}).

(9)

j= 1

According to this definition the conditional expectation E(~ If!}) (w) is a random variable which, at all sample points w belonging to the same atom Di> takes the same value 1 xiP(AiiD;). This observation shows that the definition of E( ~I D;) could have been expressed differently. In fact, we could first define E(~ ID;), the conditional expectation of~ with respect to D;, by

'LJ=

(10)

and then define E(~lf!})(w)

=

k

L E(~ID;)Iv,(w)

i= 1

(see the diagram in Figure 14).

(11)

77

§8. Conditional Probabilities and Mathematical Expectations (8)

PO

E~

1(3.1) (10)

P(-ID)

E(~ID)

j(l) P(-1

1(11) {9)

~)

E(~l ~)

Figure 14 It is also useful to notice that E(~ID) and EW ~)are independent of the representation of~. The following properties of conditional expectations follow immediately from the definitions: E(a~

+ b17i ~) =

aE(~I ~)

+ bE('71 ~),

a and b constants;

(12) (13)

E(~l!l) = E~;

E(CI

~) =

C constant;

C,

(14)

(15)

The last equation shows, in particular, that properties of conditional probabilities can be deduced directly from properties of conditional expectations. The following important· property generalizes the formula for total probability (5): EE(~I ~)

=

(16)

E~.

For the proof, it is enough to notice that by (5) EE(~I ~) = E

l

l

l

j= 1

j= 1

j= 1

L xiP(Ail ~) = L xiEP(Ail2)) = L xiP(Ai) =

E~.

Let f!} = {D 1 , .•• , Dk} be a decomposition and '7 = 17(w) a random variable. We say that '7 is measurable with respect to this decomposition, or ~-measurable, if f!}~ ~ ~.i.e. '7 = 17(w) can be represented in the form k

'l(W)=

L YJn,(w),

i= 1

where some Y; might be equal. In other words, a random variable is measurable if and only if it takes constant values on the atoms of f!}.

~­

2. If~ is the trivial decomposition,~ = {Q}, then '7 ts ~-measur­ able if and only if 17 C, where C is a constant. Every random variable '7 is measurable with respect to f!}~.

ExAMPLE

=

78

I. Elementary Probability Theory

Suppose that the random variable 1J is

~-measurable.

Then (17)

and in particular

= 1J).

(E(1J I~")

To establish (17) we observe that if~=

1

xi/AJ' then

k

l

~1] =

IJ=

(18)

L L

XiYJAjD;

j= 1 i= 1

and therefore k

l

L L xiy;P(AiDd~)

E(~1JI ~) =

j= 1 i= 1

=

L L

j=l i=l

L L

j= 1 i = 1

On the other hand, since

IJE(~I ge) = =

[t [t

P(AjD;IDm)lvm(w)

xiy;P(AiDdD;)lv;(w)

j= 1 i= 1

= lv; and lv; · lvm = 0, i

YJv;(w)

(19)

xiy;P(AiiD;)lv;(w).

1 YJv/w)J ·

k

=

L L

Jb;

L

m=l

k

l

=

XjYi

k

l

=

k

k

l

lt

=I=

m, we obtain

1 xiP(Ail ge)J

J·mt Lt XjP(AjiDm)] · lvJw)

l

L L Y;XiP(AiiD;) · lv;(w),

i= 1 j= 1

which, with (19), establishes (17). We shall establish another important property of conditional expectations. Let ge 1 and ge 2 be two decompositions, with ~ 1 ~ ge 2 (ge 2 is "finer" than ~ 1 ). Then E[E(~I ~z)l

ge1J

= E(~l

gel).

For the proof, suppose that gel = {Dll, · · ·, D1m}, Then if~ =

LJ= xi AJ' we have 1

l

E(~l gez) =

L

j= 1

xiP(Ail ge 2),

(20)

§8. Conditional Probabilities and Mathematical Expectations

79

and it is sufficient to establish that (21) Since

P(Aj/ ~ 2 ) =

n

L

q=1

P(Aj/D 2 q)lv 2 . ,

we have n

E[P(Aj/ ~2)/ ~ 1 ]

=

L

P(Aj /D 2 q)P(D 2 q/ ~d

q=1

n

m

L

p= 1

lv,p ·

L

q= 1

m

=

L

p=1

P(Aj/D 2 q)P(D 2 q/D 1 p)

L

lv,p ·

P(Aj/D 2 q)P(D 2 q/D 1 P)

{q:D2qSD1p}

m

=

L

p=1

lv,p · P(Aj/D 1 p) = P(Aj/ ~ 1 ),

which establishes (21). When ~ is induced by the random variables 11~> ... , 11k ( ~ = ~~, .... ~J, the conditional expectation E(~I~~~ ..... ~J is denoted by E(~/11 1 , ••• ,11k), or E(( IIJ 1 , ... , l]k){w), and is called the conditional expectation of ( with respect to 11 1 , .•. , 11k· It follows immediately from the definition of E(~ /11) that if~ and 11 are independent, then E(~/11) = E~.

(22)

11·

(23)

From (18) it also follows that

E(11 /11)

=

Property (22) admits the following generalization. Let ~ be independent of ~(i.e. for each D; E ~the random variables~ and lv, are independent). Then E(~/ ~) = E~.

(24)

As a special case of (20) we obtain the following useful formula: (25)

80

I. Elementary Probability Theory

EXAMPLE 3. Let us find E(~ + 11 111) for the random variables~ and 11 considered in Example 1. By (22) and (23),

This result can also be obtained by starting from (8):

E(~

+ 11111) =

EXAMPLE 4. Let variables. Then

~

2

L kP(~ + 11 = k=O

kll1) = p(1- 11)

+ ql1 + 2pl1

=

p

+ 11·

and 11 be independent and identically distributed random

(26) In fact, if we assume for simplicity that ~and 11 take the values 1, 2, ... , m, we find (1 ::; k ::; m, 2 ::; I ::; 2m)

P( ~ = k I~

+ '1

= I) =

P( ~ = k, '1 = I - k) P( ~ = k, ~ + 11 = l) P( ~ + 11 = l) = P( ~ + 11 = l) P(~ =

k)P(11 = I - k) + 11 = l)

P( ~ =

P(IJ =

klc; +

P(11 = k)P(~ = 1 - k) P( ~

+ '1 =

l)

1J = [).

This establishes the first equation in (26). To prove the second, it is enough to notice that

3. We have already noticed in §1 that to each decomposition ~ = {D 1 , ... , Dk} of the finite set Q there corresponds an algebra a(~) of subsets of Q. The converse is also true: every algebra !!J of subsets of the finite space Q generates a decomposition ~ (!!J = a(~)). Consequently there is a oneto-one correspondence between algebras and decompositions of a finite space Q. This should be kept in mind in connection with the concept, which will be introduced later, of conditional expectation with respect to the special systems of sets called a-algebras. For finite spaces, the concepts of algebra and a-algebra coincide. It will turn out that if !!J is an algebra, the conditional expectation E( ~I !!J) of a random variable ~ with respect to !!J (to be introduced in §7 of Chapter II) simply coincides with EW ~), the expectation of ~ with respect to the decomposition ~ such that !!J = a(~). In this sense we can, in dealing with finite spaces in the future, not distinguish between E(~I!!J) and EW ~), understanding in each case that E(~I !!J) is simply defined to be E(~I ~).

§9. Random Walk. I. Probabilities of Ruin and Mean Duration in Coin Tossing

4. PROBLEMS 1. Give an example of random variables which

(Cf. (22).)

81

eand '1 which are not independent but for

e

2. The conditional variance of with respect to !!} is the random variable

Show that

3. Starting from (17), show that for every function f

= f('1) the conditional expectation

E(el'1) has the property

4. Let e and '1 be random variables. Show that inf1 E('1 - f(e)) 2 is attained for J*(e) = E('11 e). (Consequently, the best estimator for '1 in terms of in the mean-square sense, is the conditional expectation E('11 e)).

e.

el .... ' e•.

el .... 'e. are identically e + ··· + e, is the

5. Let 't' be independent random variables, where distributed and r takes the values 1, 2, ... , n. Show that if S, = sum of a random number of the random variables,

1

and ES, = Er · E~ 1 , 6. Establish equation (24).

§9. Random Walk. I. Probabilities of Ruin and Mean Duration in Coin Tossing 1. The value of the limit theorems of §6 for Bernoulli schemes is not just that they provide convenient formulas for calculating probabilities P(Sn = k) and P(A < Sn ::;; B). They have the additional significance of being of a universal nature, i.e. they remain useful not only for independent Bernoulli random variables that have only two values, but also for variables of much more general character. In this sense the Bernoulli scheme appears as the simplest model, on the basis of which we can recognize many probabilistic regularities which are inherent also in much more general models. In this and the next section we shall discuss a number of new probabilistic regularities, some of which are quite surprising. The ones that we discuss are

82

I. Elementary Probability Theory

again based on the Bernoulli scheme, although many results on the nature of random oscillations remain valid for random walks of a more general kind. 2. Consider the Bernoulli scheme (Q, d, P), where n = {(J): (J) = (x 1• ... ' Xn), Xi= ± 1}, d consists of all subsets of Q, and p(w) = pv(w)qn-v(wl, v(w) = (L xi + n)/2. Let ~i(w) = xi, i = 1, ... , n. Then, as we know, the sequence ~ 1, •.• , ~n is a sequence of independent Bernoulli random variables, P(~i =

1) = p,

P(~i

= -1) = q,

p+q=l.

Let us put S0 = 0, Sk = ~ 1 + · · · + ~b 1 s k s n. The sequence S0 , S 1 , .•• , Sn can be considered as the path of the random motion of a particle starting at zero. Here Sk+ 1 = Sk + ~k> i.e. if the particle has reached the point Sk at time k, then at time k + 1 it is displaced either one unit up (with probability p) or one unit down (with probability q). Let A and B be integers, A s 0 s B. An interesting problem about this random walk is to find the probability that after n steps the moving particle has left the interval (A, B). It is also of interest to ask with what probability the particle leaves (A, B) at A or at B. That these are natural questions to ask becomes particularly clear if we interpret them in terms of a gambling game. Consider two players (first and second) who start with respective bankrolls (-A) and B. If ~i = + 1, we suppose that the second player pays one unit to the first; if~; = -1, the first pays the second. Then Sk = ~ 1 + · · · + ~k can be interpreted as the amount won by the first player from the second (if Sk < 0, this is actually the amount lost by the first player to the second) after k turns. At the instant k s nat which for the first time Sk = B (Sk = A) the bankroll of the second (first) player is reduced to zero; in other words, that player is ruined. (If k < n, we suppose that the game ends at time k, although the random walk itself is well defined up to time n, inclusive.) Before we turn to a precise formulation, let us introduce some notation. Let x be an integer in the interval [A, B] and for 0 s k s n lets:= x + Sk, r: = min{O

s ls

k: St =A orB},

(1)

where we agree to take r: = k if A < Sf < B for all 0 s l s k. For each k in 0 s k s n and x E [A, B], the instant r:, called a stopping time (see §11), is an integer-valued random variable defined on the sample space n (the dependence of r: on n is not explicitly indicated). It is clear that for alll < k the set {w: r: = l} is the event that the random walk {Sf, 0 s is k}, starting at time zero at the point x, leaves the interval (A, B) at time l. It is also clear that when l s k the sets {w: r: = l, Sf= A} and {w: r: = l, Sf = B} represent the events that the wandering particle leaves the interval (A, B) at time l through A orB respectively.

§9. Random Walk. I. Probabilities of Ruin and Mean Duration in Coin Tossing

For 0

~

k

~

83

n, we write

die =

L: {w: tic = 1, s~ = A}, O:S:ISk

&lie =

L:

{w:

O:S:!Sk

(2)

tic = 1, Sf = B},

and let

be the probabilities that the particle leaves (A, B), through A orB respectively, during the time interval [0, k]. For these probabilities we can find recurrent relations from which we can successively determine 1X 1(x), ... , 1Xn(x) and Pt (x), ... , Pn(x). Let, then, A < x
= P(&llc) = P(&llciS1 =

x

+

l)P(~ 1

= 1)

+ P(&llciS1 =X- 1)P(~l = -1) = pP(&IIc = X + 1) + qP(&IIcl S1 = X

-

1).

(3)

We now show that

P(&llciS1

=X+ 1) =

P(&llc~D,

To do this, we notice that

&lie=

{w:(x,x

&llc can be represented in the form

+ ~ 1 , ... ,x + ~ 1 + ... + ~k)EBk},

Blc is the set of paths of the form (x, x + x 1 , ... , x + x 1 + · · · xk) with x 1 = ± 1, which during the time [0, k] first leave (A, B) at B (Figure 15). where

A~-----------------------

Figure 15. Example of a path from the set B';.

84

I. Elementary Probability Theory

We represent B~ in the form B~,x+ 1 + B~,x- 1 , where B~·x+ 1 and BV- 1 are the paths in B~ for which x 1 = + 1 or x 1 = -1, respectively. Notice that the paths (x, x + 1, x + 1 + x 2 , ••• , x + 1 + x 2 + · · · + xk) in B~·x+ 1 are in one-to-one correspondence with the paths (x

+ 1, x + 1 + x 2 , ••• , x + 1 + x 2, ••• , x + 1 + X 2 + · · · + xk)

in B~~ ~. The same is true for the paths in B~·x- 1• Using these facts, together with independence, the identical distribution of ~ 1 , ... , ~b and (8.6), we obtain P(Bi~ISf = X

+ 1)

= P(Bi~l~ 1 = 1)

= P{(x,x + ~t>····x + ~ 1 + ··· + ~k)eB~I~t = 1} + 1, x + 1 + ~ 2 , ••• , x + 1 + ~2 + · · · + ~k)eB~~D P{(x + 1,x + 1 + ~1, .•. ,x + 1 + ~1 + ··· + ~k-t)eB~~n

= P{(x =

= P(Bi~~D-

In the same way, P(Bi~ISf

=X-

1) = P(Bi~=D·

Consequently, by (3) with x e (A, B) and k Pk(x) = PPk-t(x

~

n,

+ 1) + qflk-t(x-

1),

(4)

1:::;;, n.

(5)

where

{J1(B)

=

1,

{J1(A)

= 0,

0

~

Similarly (6)

with

1X1(A) = 1,

1X1(B) =

0,

O~l~n.

Since tX 0 (x) = {J 0 (x) = 0, x e (A, B), these recurrent relations can (at least in principle) be solved for the probabilities

1X1(x), ... , IX"(x)

and

{J 1(x), ... , fln(x).

Putting aside any explicit calculation of the probabilities, we ask for their values for large n. For this purpose we notice that since Bi~ _ 1 c: Bi~, k ~ n, we have Pk- 1(x) ~ {Jk(x) ~ 1. It is therefore natural to expect (and this is actually the case; see Subsection 3) that for sufficiently large n the probability fln(x) will be close to the solution {J(x) of the equation {J(x) = p{J(x

+

1)

+ q{J(x -

1)

(7)

§9. Random Walk. I. Probabilities of Ruin and Mean Duration in Coin Tossing

85

with the boundary conditions

f3(B) = 1,

f3(A) = 0,

(8)

that result from a formal approach to the limit in (4) and (5). To solve the problem in (7) and (8), we first suppose that p =1= q. We see easily that the equation has the two particular solutions a and b(qjpy, where a and b are constants. Hence we look for a solution of the form f3(x)

=

a

+ b(qjpy.

(9)

Taking account of (8), we find that for A ::;; x ::;; B f3(x)

= (q/pY - (q/p)A.

(10)

(qjp)B _ (qjp)A

Let us show that this is the only solution of our problem. It is enough to show that all solutions of the problem in (7) and (8) admit the representation (9). Let P(x) be a solution with P(A) = 0, P(B) = 1. We can always find constants aand b such that

a + b(qjp)A =

b(A),

a+

b(q/p)A+l = P(A

a+

b(q/p)A+ 2

+ 1).

Then it follows from (7) that P(A

+ 2)

=

and generally P(x) =

a + b(qjpy.

Consequently the solution (10) is the only solution of our problem. A similar discussion shows that the only solution of a(x)

= pa(x + 1) + qa(x - 1),

XE(A, B)

(11)

with the boundary conditions a(A) = 1,

a(B)

=0

(12)

A::;;x::;;B.

(13)

is given by the formula (p/q)B - (qjpy a(x) = (p/q)B _ (pjp)A'

If p = q = respectively

t, the only solutions f3(x) and a(x) of (7), (8) and (11 ), ( 12) are f3(x)

x-A

= B- A

(14)

and

B-x

a(x) = B- A.

(15)

86

I. Elementary Probability Theory

We note that

a(x)

+ fJ(x) =

1

(16)

for 0 s; p s; 1. We call a(x) and {J(x) the probabilities of ruin for the first and second players, respectively (when the first player's bankroll is x - A, and the second player's is B - x) under the assumption of infinitely many turns, which of course presupposes an infinite sequence of independent Bernoulli random variables ~b ~ 2 , ••. , where~;= + 1 is treated as a gain for the first player, and ~; = -1 as a loss. The probability space (Q, d, P) considered at the beginning of this section turns out to be too small to allow such an infinite sequence of independent variables. We shall see later that such a sequence can actually be constructed and that {J(x) and a(x) are in fact the probabilities of ruin in an unbounded number of steps. We now take up some corollaries of the preceding formulas. If we take A = 0, 0 s; x s; B, then the definition of {J(x) implies that this is the probability that a particle starting at x arrives at B before it reaches 0. It follows from (10) and (14) (Figure 16) that

fJ(x) =

xjB, p = q { (qjpy - 1

= !,

(qjp)B - 1' p =f- q.

(17)

Now let q > p, which means that the game is unfavorable for the first player, whose limiting probability of being ruined, namely a = a(O), is given by

(qjp)B - 1 (q/pl - (qjp)A .

G(=-~----c

Next suppose that the rules of the game are changed: the original bankrolls of the players are still (-A) and B, but the payoff for each player is now !, {J(x)

X

Figure 16. Graph of fl(x), the probability that a particle starting from x reaches B before reaching 0.

§9. Random Walk. I. Probabilities of Ruin and Mean Duration in Coin Tossing

87

rather than 1 as before. In other words, now let P(ei = !) = p, P(ei = -!) = q. In this case let us denote the limiting probability of ruin for the first player by a. 112 • Then (qjp)2B _ 1 r:J.l/2 = (qjp)2B _ (qjp)2A' and therefore =

r:J.l/2

r:J.'

(qjp)B + 1 (qjp)B + (qjp)A > r:t.,

if q > p. Hence we can draw the following conclusion:

if the game is unfavorable to the .first player (i.e., q > p) then doubling the stake decreases the probability of ruin. 3. We now turn to the question of how fast r:t.n(x) and Pn(x) approach their limiting values a.(x) and P(x). Let us suppose for simplicity that x = 0 and put

r:l.n = r:t.n(O), It is clear that

Yn = P{A < Sk < B, 0 where {A < Sk < B, 0

k

~

~

~

k

~

n},

n} denotes the event

n

{A< Sk < B}.

O!>k!>n

Let n = rm, where r and m are integers and

+ ... + em, em+ 1 + ... + e2m•

el = el

e2

=

e, =

em(r-1)+ 1

+ ... + erm•

Then if C = IA I + B, it is easy to see that {A

< Sk < B, 1 ~ k

and therefore, since ( 1 ,

••• ,

~

rm} £ {1( 1 1 < C, ... , I(, I < C},

(,are independent and identically distributed,

Yn ~ P{ let I < C, ... , I(, I < C} = We notice that ciently large m,

Ve

1

r

0 P{l(d < C} =

= m[1 - (p- q) 2 ]. Hence, for 0

P{l( 1 1 < c}:::; where 8 1 < 1, since

(P{I(tl < C})'.

ve 1

(18)

i=l

81,

~ C 2 if P{IC 1 1 ~ C}

= 1.

< p < 1 and suffi(19)

88

I. Elementary Probability Theory

If p = 0 or p = 1, then P{ i(d < C} = 0 for sufficiently large m, and consequently (19) is satisfied for 0 ::::;; p ::::;; 1. It follows from (18) and (19) that for sufficiently large n

Yn::::;; e", where e = e~fm < 1. According to (16), oc

+ f3

(20)

= 1. Therefore

(oc - OCn)

+ ({3 - f3n)

=

Yn,

and since oc ~ ocn, f3 ~ f3n, we have 0 ::::;; oc - ocn ::::;; Yn ::::;; e",

0 ::::;; f3 - /3n ::::;; Yn ::::;; e",

e < 1.

There are similar inequalities for the differences oc(x) - ocn(x) and f3(x)- /3n(x). 4. We now consider the question of the mean duration of the random walk. Let mk(x) = E-r: be the expectation of the stopping time -r:,k::::;; n. Proceeding as in the derivation of the recurrent relations for f3x(x), we find that, for x E (A, B), mk(x)

= E-r: = = =

I

1 :s;l:s;k

I

1 :s;l:s;k

lP(-r: = l)

l· [pP(-rk = ~~~1 = 1)

I [. [pP(-r:~I

1 :s;l :s;k

I

(l

O:s;l:s;k-1

= pmk- 1(x

+

I

+

+

1)[pP(-r:~f = l)

1)

11~1

= -1)]

= l - 1) + qP(-r:=I = l - 1)]

+ qmk- 1(x-

[pP(-r:~f = l)

O:s;l:s;k-1

= pmk-1(x

+ qP(-rk =

+ qP(-r:=:f

1)

+ qP(-r:=:f

+ 1) + qmk- 1(x-

= l)]

1)

= l)]

+ 1.

Thus,forx E (A, B)andO::::;; k::::;; n, thefunctionsmk(x)satisfytherecurrent relations (21)

with m0 (x) = 0. From these equations together with the boundary conditions mk(A)

=

mk(B)

= 0,

we can successively find m 1(x), ... , mn(x). Since mk(x) ::::;; mk+ 1(x), the limit m(x) = lim mix) n-+oo

(22)

§9. Random Walk. I. Probabilities of Ruin and Mean Duration in Coin Tossing

89

exists, and by (21) it satisfies the equation m(x)

+ 1 + pm(x + 1) + qm(x

- 1)

(23)

with the boundary conditions m(A) = m(B) = 0.

(24)

To solve this equation, we first suppose that m(x) < oo,

x

E

(A, B).

(25)

Then if p =I= q there is a particular solution of the form a.j(q - p) and the general solution (see (9)) can be written in the form m(x) = -Xq-p

+ a + b (q)x - . p

Then by using the boundary conditions m(A) = m(B) = 0 we find that 1 m(x) = - - (B{J(x) p-q

+ Aa.(x) -

x],

where fJ(x) and a.(x) are defined by (10) and (13). If p = q = solution of (23) has the form m(x)

=

a

+ bx -

(26)

!, the general

x 2,

and since m(A) = m(B) = 0 we have m(x) = (B - x)(x - A).

(27)

It follows, in particular, that if the players start with equal bankrolls (B = -A), then m(O) = B 2 •

If we take B = 10, and suppose that each turn takes a second, then the (limiting) time to the ruin of one player is rather long: 100 seconds. We obtained (26) and (27) under the assumption that m(x) < oo, x E (A, B). Let us now show that in fact m(x) is finite for all x E (A, B). We consider only the case x = 0; the general case can be analyzed similarly. Let p = q =!.We introduce the random variableS<" defined in terms of the sequence S0 , Sl> ... , Sn and the stopping time •n = .-~by the equation n

s,n =

L Skl{tn=k)(w). k=O

(28)

The descriptive meaning of S
•n·

•n

90

I. Elementary Probability Theory

Let us show that when p = q =

t.

ES,n = 0,

(29)

Es;n =Ern.

(30)

To establish the first equation we notice that n

ES,n =

L E[SkJ{tn=k}(w)]

k=O n

=

n

L E[SnJ{tn=k}(w)] + L E[(Sk- Sn)Jitn=k}(w)]

k=O

= ESn

k=O

+

n

L E[(Sk- Sn)II
k=O

(31)

where we evidently have ESn = 0. Let us show that n

L E[(Sk- Sn)Jitn=k}(w)] = 0.

k=O

To do this, we notice that {tn > k} = {A < S 1 < B, ... , A < Sk < B} when 0:::;; k < n. The event {A< S 1 < B, ... , A< Sk < B} can evidently be written in the form (32) where Ak is a subset of { -1, + 1}k. In other words, this set is determined by just the values of e1, ... , ek and does not depend on ek+t• ... , en. Since

{tn = k} = {tn > k- 1}\{tn > k}, this is also a set of the form (32). It then follows from the independence of 1 , •.. , en and from Problem 9 of §4 that the random variables Sn - Sk and I l
e

E[(Sn- Sk)II
ES;" =

n

n

k=O

k=O

L ESfJ{tn=k} = L E([Sn + (Sk- Sn)] 2I{tn=k}) n

=

L [ES;II
k=O

n

= n-

L (n -

k=O

n

k)P(tn = k) =

L kP(tn = k) = Et".

k=O

§9. Random Walk. I. Probabilities of Ruin and Mean Duration in Coin Tossing

(p

Thus we have (29) and (30) when p = q = + q = 1) it can be shown similarly that

= (p

ES,n

=

E[S,"- r. · E~ 1 ] 2

t.

91

For general p and q (33)

- q) ·Ern,

V~ 1 ·Ern,

(34)

where E~ 1 = p- q, V~ 1 = 1 - (p- q) 2 • With the aid of the results obtained so far we can now show that limn-+ 00 mn(O) = m(O) < 00. If p = q = !, then by (30) (35)

If p =F q, then by (33), E

r.:::;;

max( lA I, B)

(36)

Ip-q I '

from which it is clear that m(O) < oo. We also notice that when p = q =!

Ern=

es;n = A 2 • (X.+ B 2 • fJ. + E[S;I{A<Sn
and therefore

It follows from this and (20) that as n _... oo, Er. converges with exponential rapidity to

m(O) = A 2 lX

B

A

B-A

B-A

+ B 2 {J = A 2 · - - - B 2 · - - =

IABI.

There is a similar result when p =F q: Er. _... m(0) -_ ocA

+ fJB ,

p-q

exponentially fast.

5. PROBLEMS 1. Establish the following generalizations of (33) and (34):

Es:. = x

+ (p

- q)Et~,

E[Stlf- •: · E~tJ 2 = V~ 1 . Et:. 2. Investigate the limits of oc(x), P(x), and m(x) when the level A ! 3. Let p = q =

- oo. tin the Bernoulli scheme. What is the order of EIS.l for large n?

92

I. Elementary Probability Theory

4. Two players each toss their own symmetric coins, independently. Show that the probability that each has the same number of heads after n tosses is r 2" D=o (C=) 2 • Hence deduce the equation D=o (C=>2 = Cin· Let u. be the first time when the number of heads for the first player coincides with the number of heads for the second player (if this happens within n tosses; un = n + 1 if there is no such time). Find Emin(u., n).

§10. Random Walk. II. Reflection Principle. Arcsine Law 1. As in the preceding section, we suppose that ~ 1 , ~ 2 , ••• , ~ 2 " is a sequence of independent identically distributed Bernoulli random variables with

= 1) = p, + · · · + ~k•

P(~j

sk =

~1

1~ k

~

2n;

We define a2n

= min{1

:::; k :::; 2n:

sk = 0},

putting G'2n = 00 if sk =1: 0 for 1 ~ k :::; 2n. The descriptive meaning of a 2 n is clear: it is the time of first return to zero. Properties of this time are studied in the present section, where we assume that the random walk is symmetric, i.e. p = q = !. For 0 :::; k :::; n we write u2k

= P(S 2 k = 0),

f2k

= P(a2n = 2k).

(1)

It is clear that u0 = 1 and Our immediate aim is to show that for 1 :::; k :::; n the probability f 2 k is given by (2)

It is clear that

{a 2 n = 2k} = {S 1 =1: 0, S 2 =1: 0, ... , S 2 k- 1 =1: 0, S 2 k = 0} for 1 :::; k :::; n, and by symmetry !2k

o, ... , s2k-1 =1: o, s2k = O} 2P{s1 > o, ... , s2k-1 > o, s2k = O}.

= P{S1 =

=1:

(3)

93

§10. Random Walk. II. Reflection Principle. Arcsine Law

A sequence (S 0 , ••• , Sk) is called a path of length k; we denote by Lk(A) the number of paths of length k having some specified property A. Then f2k

=

L2n<s1 >

2

o, ... , s2k-1 > o, s2k = o,

and S2k+1 = a2k+1• ... ,S2n = a2k+1

= 2L2k(s1 >

+ ··· + a2n)·2- 2"

o, ... , s2k-1 > o, s2k = O) · 2- 2\

(4)

where the summation is over all sets (a 2k+ 1, ... , a2n) with a; = ± 1. Consequently the determination of the probability f 2 k reduces to calculating the number of paths L 2k(S 1 > 0, ... , S 2k- 1 > 0, S 2k = 0).

Lemma 1. Let a and b be nonnegative integers, a - b > 0 and k = a Then a-b Lk(S1>0, ... ,Sk-1>0,Sk=a-b)=-k-Ck·

+ b. (5)

PRooF. In fact,

Lk(S 1 > 0, ... , Sk- 1 > 0, Sk =a- b) = Lk(S 1 = 1, S2 > 0, ... , Sk- 1 > 0, Sk =a- b) = Lk(S 1 = 1, Sk =a- b) - Lk(S 1 = 1, Sk =a- b; and 3 i, 2 :::;; i :::;; k - 1, such that S; :::;; 0).

(6)

In other words, the number of positive paths (S 1, S2, ... , Sk) that originate at (1, 1) and terminate at (k, a- b) is the same as the total number of paths from (1, 1) to (k, a - b) after excluding the paths that touch or intersect the time axis.* We now notice that

Lk(S 1 = 1, Sk =a- b; 3 i, 2 :::;; i:::;; k- 1, such that S;:::;; 0) =

Lk(S 1 = -1, Sk =a- b),

(7)

i.e. the number of paths from IX = (1, 1) to f3 = (k, a - b), neither touching nor intersecting the time axis, is equal to the total number of paths that connect IX* = (1, -1) with {3. The proof of this statement, known as the reflection principle, follows from the easily established one-to-one correspondence between the paths A= (S 1, ... , Sa, Sa+ 1, ... , Sk) joining IX and {3, and paths B = ( -S 1, ... , -Sa, Sa+~> ... , Sk)joining IX* and f3 (Figure 17); a is the first point where A and B reach zero.

* A path (S 1 , ••• , Sk) is called positive (or nonnegative) if all S1 > 0 (S1 ~ 0); a path is said to touch the time axis if S1 ~ 0 or else S1 :::;; 0, for I s j s k, and there is an i, I :::;; i :::;; k, such

that S 1 = 0; and a path is said to intersect the time axis if there are two times i and j such that 0 and sj < 0.

s, >

94

I. Elementary Probability Theory

fJ

Figure 17. The reflection principle.

From (6) and (7) we find

Lk(S 1 > 0, ... , Sk- 1 > 0, Sk =a- b)

= Lk(S 1 = 1,Sk =a- b)- Lk(S 1 = a-1

a

= ck-1- ck-1 =

-1,Sk =a- b)

a-bca -k- k,

which establishes (5). Turning to the calculation ofj2 k, we find that by (4) and (5) (with a = k, b = k- 1), f2k

o, ... , s2k-1 > o,

= o) · 2- 2k

=

2L2k(sl >

=

2Lzk-1(S1 > O, ... ,Szk- 1 = 1)·2- 2k

s2k

t 1 ck 1 2k-1 = 2k Uz(k-1)· = 2 . rzk . 2k-

Hence (2) is established. We present an alternative proof of this formula, based on the following observation. A straightforward verification shows that

1 2k Uz(k-1) = u2(k-1) -

(8)

Uzk>

At the same time, it is clear that

= 2k} = {a2n > 2(k- 1)}\{a2n > {a2n > 2!} = {S1 =I= 0, ... , S 21 =!= 0}

{azn

2k},

and therefore {a2n = 2k} = {S 1 =I= 0, ... , S2(k- 1) =I= 0}\{S 1 =I= 0, ... , S 2k =I= 0}.

Hence fzk

=

P{S1 =I= 0, ... , Sz(k-1) =I= 0} - P{S1 =I= 0, ... , S2k =I= 0},

95

§10. Random Walk. II. Reflection Principle. Arcsine Law

a

Figure 18

and consequently, because of (8), in order to show that it is enough to show only that

L2kCS1 ¥ 0, ... , S2k ¥ 0)

f 2k = (1/2k)u 2
= L2kCS2k = 0).

(9)

For this purpose we notice that evidently

L2k(S1 ¥ 0, ... , S2k ¥ 0) = 2L2k(St > 0, ... , S2k > 0). Hence to verify (9) we need only establish that

2L2k(St > 0, ... , S2k > 0) = L2k(S1

~

0, ... , S2k ~ 0)

(10)

and

(11) Now (10) will be established if we show that we can establish a one-to-one correspondence between the paths A = (S~> ... , S 2 k) for which at least one S; = 0, and the positive paths B = (S ~> ... , S 2k). Let A = (S ~> ... , S2k) be a nonnegative path for which the first zero occurs at the point a (i.e., Sa = 0). Let us construct the path, starting at (a, 2), (Sa + 2, Sa+ 1 + 2, ... , S 2 k + 2) (indicated by the broken lines in Figure 18). Then the path B = (S 1, ..• , Sa_ 1 , Sa+ 2, ... , S 2 k + 2) is positive. Conversely, let B = (S 1, ..• , S 2k) be a positive path and b the last instant at which Sb = 1 (Figure 19). Then the path

A = (S 1' . . . , Sb, Sb+ 1

-

b

Figure 19

2, ... , Sk - 2)

96

I. Elementary Probability Theory

2k

-m Figure 20

is nonnegative. It follows from these constructions that there is a one-to-one correspondence between the positive paths and the nonnegative paths with at least one Si = 0. Therefore formula (10) is established. We now establish (11). From symmetry and (10) it is enough to show that L2k(S1 > 0, ... , S 2k > 0) + L 2k(S 1 :2:: 0, ... , S 2k :2:: 0 and 3 i, 1 ~ i ~ 2k, such that S; = 0) = L 2 k(S 2 k = 0).

The set of paths (S 2 k = 0) can be represented as the sum of the two sets "t' 1 and "t'2 , where "t' 1 contains the paths (S 0 , ••• , S 2k) that have just one minimum, and "t' 2 contains those for which the minimum is attained at at least two points. Let C 1 E "t' 1 (Figure 20) and let y be the minimum point. We put the path C 1 = (S 0 , St. ... , S 2 k) in correspondence with the path Ct obtained in the following way (Figure 21). We reflect (S 0 , S 1, ••• , S1 ) around the vertical line through the point l, and displace the resulting path to the right and upward, thus releasing it from the point (2k, 0). Then we move the origin to the point (l, -m). The resulting path Ct will be positive. In the same way, if C 2 e "t' 2 we can use the same device to put it into correspondence with a nonnegative path C~.

(2k, 2m)

2k

Figure 21

97

§10. Random Walk. II. Reflection Principle. Arcsine Law

Conversely, let Ct = (S 1 > 0, ... , S 2 k > 0) be a positive path with S 2 k =2m (see Figure 21). We make it correspond to the path C 1 that is obtained in the following way. Let p be the last point at which SP = m. Reflect (SP, ... , S 2 m) with respect to the vertical line x = p and displace the resulting path downward and to the left until its right-hand end coincides with the point (0, 0). Then we move the origin to the left-hand end of the resulting path (this is just the path drawn in Figure 20). The resulting path C 1 = (S 0 , •.• , S 2 k) has a minimum at S 2 k = 0. A similar construction applied to paths (S 1 ~ 0, ... , S 2 k ~ 0 and 3 i, 1 ~ i ~ 2k, with S; = 0) leads to paths for which there are at least two minima and S 2 k = 0. Hence we have established a one-to-one correspondence, which establishes (11). Therefore we have established (9) and consequently also the formula f2k

=

Uz(k-l) -

= (1/2k)u2(k-l) ·

U2k

By Stirling's formula

u2k =

k c2k.

2

-2k

1

k-+

"' - - ,

fo

00.

Therefore

k-+

00.

Hence it follows that the expectation of the first time when zero is reached, namely Emin(a 2 n, 2n)

=

n

L 2kP(a

k=l

2n

= 2k) + 2nu 2 n

n

=

L u2(k-1) + 2nu2n• k= 1

can be arbitrarily large. In addition, 1 u 2 = oo, and consequently the limiting value of the mean time for the walk to reach zero (in an unbounded number of steps) is oo. This property accounts for many of the unexpected properties of the symmetric random walk that we have been discussing. For example, it would be natural to suppose that after time 2n the number of zero net scores in a game between two equally matched players (p = q = !), i.e. the number of instants i at which S; = 0, would be proportional to 2n. However, in fact the number of zeros has order (see [F1]). Hence it follows, in particular, that, contrary to intuition, the "typical" walk (S 0 , S 1 , .•• , S") does not have a sinusoidal character (so that roughly half the time the particle would be on the positive side and half the time on the negative side), but instead must resemble a stretched-out wave. The precise formulation of this statement is given by the arcsine law, which we proceed to investigate.

Lf=

fo

98

I. Elementary Probability Theory

2. Let P2 k, 2n be the probability that during the interval [0, 2n] the particle spends 2k units of time on the positive side.*

Lemma 2. Let u0 = 1 and 0 ::::; k ::::; n. Then (12)

PRooF. It was shown above thatf2k = u2
Uzk =

Since {S 2 k = 0}

!;;;;; {11 2"::::;

L f2r · U2(k-r)·

r=

(13)

1

2k}, we have

{S 2k = 0} = {S 2k = 0} n {O'zn::::; 2k} =

L

1 ::s;l:s;k

{S2k = 0} n {0'2n = 21}.

Consequently Uzk = P(S2k = 0) =

L L

1 ::s;l::s;k

=

P(S2k = 0, O'zn = 21)

P(Szk = OIO'zk = 2l)P(O'zn = 2/).

1 :s;l::s;k

But P(S 2k = OIO'zn = 21) = P(Szk = OIS1 #- 0, ... , S21-1 #- 0, S21 = 0)

+ (~21+1 + .. · + ~zk) = O!S1 #- 0, ... , S21-1 P(S21 + (~21+1 + · · · + ~zk) = OISz, = 0) P(~zt+ 1 + · · · + ~2k = 0) = P(Sz
= P(S21 = =

#- 0, S21 = 0)

Therefore U2k =

L

1 ::s;l::s;k

P(S2(k-1J = O)P(O'zn = 21),

which establishes (13). We turn now to the proof of (12). It is obviously true fork = 0 and k = n. Now let 1 ::::; k ::::; n - 1. If the particle is on the positive side for exactly 2k instants, it must pass through zero. Let 2r be the time of first passage through zero. There are two possibilities: either Sk ;;;:: 0, k ::::; 2r, or Sk ::::; 0, k ::::; 2r. The number of paths of the first kind is easily seen to be (21 22rf:2r) ' 22(n-rJp 2(k-r),2(n-r)

1 22n ' J:2r' p 2(k-r),2(n-r)• = 2'

* We say that the particle is on the positive side in the interval [m - l, m] if one, at least, of the values S,._ 1 and S,. is positive.

99

§10. Random Walk. II. Reflection Principle. Arcsine Law

The corresponding number of paths of the second kind is

Consequently, for 1 s k p2k,2n

=

1

s

n - 1, 1

k

k

2 7 ~1 fzr · p2(k-r),2(n-r) + 2 ,~/2r · P2k,2(n-r)•

(14)

Let us suppose that P Zk, zm = u2 k • u2 m _ Zk holds for m = 1, ... , n - 1. Then we find from (13) and (14) that P2k,2n

= !uzn-2k ·

k

k

r;1

r;1

L fzr ·llzk-2r + !uzk • L fzr ·llzn-2r-2k

This completes the proof of the lemma. Now let y(2n) be the number of time units that the particle spends on the positive axis in the interval [0, 2n]. Then, when x < 1, 1 y(2n) P{ - < - - S

X

2n

2

}

=

L

{k, 1/2 < (2k/2n) s x)

P 2k, 2n ·

Since

ask-+ oo, we have p 2k,2n =

ll

2k

·U

2(n-k)

~

1

--;==== _ k)'

njk(n

as k -+ oo and n - k -+ oo. Therefore

L

P 2k, 2n

-

{k: 1/2 <(2k/2n)Sx}

L

{k: 1/2 <(2kj2n)~x)

__!,_ ·

nn

[~ (1 - ~)]n

n

112

-+

0,

whence

L

{k: l/2<(2k/2n)Sn)

p 2k

2n '

1 n

-

IX 1/2

dt -+ 0, jt(l - t)

But, by symmetry,

L

{k:k/nS 1/2)

Pzk,2n-+!

n -+ oo.

n-+

00,

100

I. Elementary Probability Theory

and

_1 n

Jx 112

dt . vC. 1 = -2 arcsm x - 2· Jt(1 - t) n

Consequently we have proved the following theorem.

Theorem (Arcsine Law). The probability that the fraction of the time spent by the particle on the positive side is at most x tends to 2n- 1 arcsin .jX:

L

P lk, ln ~ 2n- 1 arcsin

.jX.

(15)

{k:k/n!S:x}

We remark that the integrand p(t) in the integral

1

ix

dt n o Jt(1 - t) represents aU-shaped curve that tends to infinity as t Hence it follows that, for large n,

P{o < y(2n) < ~} > p{! < y(2n) < 2n2 2n-

1 2

~

0 or 1.

+ ~}

'

i.e., it is more likely that the fraction of the time spent by the particle on the positive side is close to zero or one, than to the intuitive value !. Using a table of arcsines and noting that the convergence in (15) is indeed quite rapid, we find that

p{Y~nn) :::;; 0.024} ~ 0.1, P{y~:):::;; 0.1} ~ 0.2, P{y~:) : :; 0.2} ~ 0.3,

P{y~:):::;; 0.65} ~ 0.6. Hence if, say, n = 1000, then in about one case in ten, the particle spends only 24 units of time on the positive axis and therefore spends the greatest amount of time, 976 units, on the negative axis. 3.PR.OBLEMS 1. How fast does Emin(a 2., 2n)-+ oo as n-+ oo? 2. Lett.= min{1:::;; k:::;; n: sk·= 1}, where we take t. = oo if Sk < 1 for 1:::;; k:::;; n. What is the limit of Emin(t., n) as n-+ oo for symmetric (p = q =!)and for unsymmetric (p of. q) walks?

101

§II. Martingales. Some Applications to the Random Walk

§11. Martingales. Some Applications to the Random Walk 1. The Bernoulli random walk discussed above was generated by a sequence

~ 1 , ... , ~n of independent random variables. In this and the next section we introduce two important classes of dependent random variables, those that constitute martingales and Markov chains. The theory of martingales will be developed in detail in Chapter VII. Here we shall present only the essential definitions, prove a theorem on the preservation of the martingale property for stopping times, and apply this to deduce the "ballot theorem." In turn, the latter theorem will be used for another proof of proposition (10.5), which was obtained above by applying the reflection principle.

2. Let (Q, .91, P) be a finite probability space and 22 1 ~ 22 2 sequence of decompositions.

~

•••

~

22n a

Definition 1. A sequence of random variables~ 1, ... , ~n is called a martingale (with respect to the decomposition 22 1 ~ 22 2 ~ • • • ~ 22n) if (1) ~k is 22k-measurable, (2) E(~k+ 1l22k) = ~k' 1 :::;; k :::;; n - 1.

In order to emphasize the system of decompositions with respect to which the random variables form a martingale, we shall use the notation (1)

where for the sake of simplicity we often do not mention explicitly that 1 :::;; k :::;; n. When 22k is induced by ~ 1, ... , ~n' i.e.

instead of saying that ~ = (~k• 22k) is a martingale, we simply say that the sequence~ = (~k) is a martingale. Here are some examples of martingales. ExAMPLE

1. Let 111, ... , IJn be independent Bernoulli random variables with P(IJk sk

= 1) = P(IJk

= -1)

= t,

= 1J1 + ... + 1Jk and 22k = 22qJ, ... ,q.·

We observe that the decompositions 22k have a simple structure:

102

I. Elementary Probability Theory

where

v+

= {w: 171 = +1}, ~

2

=

v-

= {w: 171 = -1},

{v++ v+- v-+ '

'

'

v--}

'

where

v+ + =

{w: 171 = + 1,172 = + 1}, ... ' v-- = {w: 'It= -1,172 = -1},

etc. It is also easy to see that ~~~·····~k = ~s~o ... ,sk·

Let us show that (Sb ~k) forms a martingale. In fact, Skis ~k-measurable, and by (8.12), (8.18) and (8.24), E(Sk+tl~k)

= E(Sk + 1'fk+ti~k) = E(Skl~k) + E(17Htl~k) = Sk + E1'fk+t = Sk.

If we put S0 = 0 and take D 0 = {0}, the trivial decomposition, then the

sequence (Sk,

~k)osksn

also forms a martingale.

2. Let 17 1, ... , 1'fn be independent Bernoulli random variables with P(1]; = 1) = p, P(1]; = -1) = q. If p -::/: q, each of the !;lequences ~ = (~k) with

ExAMPLE

where

sk =

'11 + ... + '1n•

is a martingale. EXAMPLE

3. Let 17 be a random variable, ~ 1

~

··· ~

~n•

and (2)

Then the sequence ~ = (~k• ~k) is a martingale. In fact, it is evident that E(17 I~k) is ~k-measurable, and by (8.20)

In this connection we notice that by (8.20)

if~

=

(~k• ~k)

is any martingale, then

~k = E(~k+tl~k) = E[E(~k+21~k+t)l~k]

= E(~k+21~k) = · · · = E(~nl~k).

(3)

Consequently the set of martingales ~ = (~k• ~k) is exhausted by the martingales of the form (2). (We note that for infinite sequences ~ = (~k• ~k)k~t this is, in general, no longer the case; see Problem 7 in §1 of Chapter VII.)

103

§11. Martingales. Some Applications to the Random Walk

EXAMPLE 4. Let '1 ~> ... , '1 nbe a sequence of independent identically distributed random Variables, Sk = '11 + · · · + IJb and E»1 = E»sn' E»2 = E»sn.Sn~t' · · ·, E»n = E»s".s"~ 1 , ... ,s,· Let us show that the sequence~= (~b E»k) with

;:

sn n

;:

sn-1 n-

S1=-,~z=--1 , ... ,;,k=

sn+1-k k'''''~n=S1 n+ 1 -

is a martingale. In the first place, it is clear that E»k ~ E»k+ 1 and ~k is E»kmeasurable. Moreover, we have by symmetry, for j ::::; n - k + 1, (4)

(compare (8.26)). Therefore

(n- k + 1)E(1Jt1E»k) =

n-k+ 1

I

j= 1

E(,.,jiE»k) = E(Sn-k+11E»k) = sn-k+1•

and consequently ;: =

Sk

Sn-k+l

n _ k+ 1

and it follows from Example 3 that~ =

=E( (~b

'11

IE»)

k ,

E»k) is a martingale.

ExAMPLE 5. Let 1J 1, ... , '1n be independent Bernoulli random variables with P(IJ; = +1) = P(IJ; = -1) =

!,

Sk = '7 1 + · · · + '1k· Let A and B be integers, A < 0
n/2, the sequence~=

~k =(cos A.)-k exp{iA.(sk-

B; A)}

is a complex martingale (i.e., the real and imaginary parts of martingales).

(5) ~k

3. It follows from the definition of a martingale that the expectation the same for every k: E~k = E~ 1 .

form

E~k

is

It turns out that this property persists if time k is replaced by a random time. In order to formulate this property we introduce the following definition.

Definition 2. A random variable T = -r(w) that takes the values 1, 2, ... , n is called a stopping time (with respect to a decomposition (E»k) 1 ~k~n· E» 1 ~ E» 2 ~ · · · ~ E»n) if, for k = 1, ... , n, the random variable /{t=kl(w) is E»kmeasurable. If we consider !?)k as the decomposition induced by observations for k steps (for example, E»k = E»~ ...... ~k' the decomposition induced by the

104

I. Elementary Probability Theory

variables 17 1, .•. , 1'/k), then the ~k-measurability of Ift=kl(w) means that the realization or nonrealization of the event {r = k} is determined only by observations for k steps (and is independent of the "future"). If f!Jk = a(~k), then the ~k-measurability of J{t=kl(w) is equivalent to the assumption that {r = k} E f!Jk. (6) We have already introduced specific examples of stopping times: the times

rk, (Jzn introduced in §§9 and 10. Those times are special cases of stopping times of the form rA

= min{O < k:::;

(JA

= min{O:::;

n: ~k E A},

k:::; n: ~kEA},

(7)

which are the times (respectively the first time after zero and the first time) for a sequence ~ 0 , ~ 1 , ... , ~n to attain a point of the set A.

4. Theorem 1. Let ~ = (~k• ~k) 1 s;ks;n be a martingale and r a stopping time with respect to the decomposition (~k) 1 s;ks;n· Then (8)

where n

~t =

L ~kl{t=kl(w)

(9)

k=l

and

(10) PRooF (compare the proof of (9.29)). Let D E ~ 1 . Using (3) and the properties of conditional expectations, we find that

E(~ ID) = E(~Jn) t

P(D)

1

n

= P(D) t~1E[~J{t=tl. In] 1

= P(D) E(~nln) = E(~niD),

105

§ll. Martingales. Some Applications to the Random Walk

and consequently E(e,l~t) = E(enl~t) = e1.

The equation Ee, = Ee 1 then follows in an obvious way. This completes the proof of the theorem.

Corollary. For the martingale (Sk, ~k) 1 sksn of Example 1, and any stopping timer (with respect to (~k)) we have the formulas (11)

ES, = 0,

known as Wald's identities (cf. (9.29) and (9.30); see also Problem 1 and Theorem 3 in §2 of Chapter VII).

5. Let us use Theorem 1 to establish the following proposition. Theorem 2 (Ballot Theorem). Let q 1, ... , '1n be a sequence of independent identically distributed random variables whose values are nonnegative integers, Sk = q 1 + ... + 1'fk, 1 :::; k :::; n. Then P{Sk < kforall k, 1:::; k:::; niSn} = (1-

~r'

(12)

where a+ = max(a, 0). PRooF. On the set {w: Sn ~ n} the formula is evident. We therefore prove (12) for the sample points at which sn < n. Let us consider the martingale = ~k)tsksn introduced in Example 4, With = Sn+1-J(n + 1- k) and ~k = ~Sn+l-k•···•Sn· We define

e (ek,

ek

r = min{1 :::; k ::5; n: ~k ~ 1},

taking ! = n on the set {ek < 1 for all k such that 1 ::5; k ::5; n} = {maxl:s;l:s;n(SI/1) < 1}. It is clear that S1 = 0 on this set, and therefore

e, =en=

{ max S11 < 1} = {max lSISn lSISn

~~ < 1, S" < n} £

{e, = 0}.

(13)

Now let us consider those outcomes for which simultaneously max 1s 1 sn(S1/l) ~ 1 and Sn < n. Write a= n + 1 - r. It is easy to see that

a= max{1 :::; k:::; n: Sk;;:::; k} and therefore (since Sn < n) we have a < n, Sa ~ a, and Sa+ 1 < a + 1. Consequently '1a+ 1 = Sa+ 1 - Sa < (a + 1) - a = 1, i.e. '1a+ 1 = 0. Therefore a :::; sa = sa+ 1 < a + 1, and consequently sa = (J and

e,=

Sn+t-• =Sa=l. n+1-r a

106

I. Elementary Probability Theory

Therefore

{ max~~~ 1,Sn < n} £ {~t = 1}.

(14)

1:51:5n

From (13) and (14) we find that 1 { max S/

1 :51:5n

~ 1, S" < n} = {~t =

1} n {S" <

n}.

Therefore, on the set {S" < n}, we have

where the last equation follows because ~< takes only the two values 0 and 1. Let us notice now that E(~tiSn) = E(~tl~ 1 ), and (by Theorem 1) E(~tl~ 1 ) = ~ 1 = Sn/n. Consequently, on the set {S" < n} we have P{Sk < k

for all k such that 1 :::;; k :::;; n ISn} = 1 - (Sn/n). This completes the proof of the theorem.

We now apply this theorem to obtain a different proof of Lemma 1 of §10, and explain why it is called the ballot theorem. Let ~ 1 , ..• , ~"be independent Bernoulli random variables with P(~ 1 = 1) = P(~; = -1) = Sk

a

+ · · · + ~k

and a, b nonnegative integers such that a - b rel="nofollow"> 0, b = n. We are going to show that

= ~1

+

t,

a-b a+

P{S 1 > 0, ... , S" > OISn =a- b} =--b.

(15)

In fact, by symmetry, P{S 1 > 0, ... , S" > OISn =a- b}

= P{S 1 < 0, ... , S" < OISn = -(a- b)} = P{S 1 + 1 < 1, ... , S" + n < niSn + n = n- (a- b)} = P{1Jt < 1,. ·., '11 + · · · + 1Jn < ni1Jt + · · · + 1Jn = n- (aa-b n

b)}

a-b a+ b'

where we have put 1Jk = ~k + 1 and applied (12). Now formula (10.5) follows from (15) in an evident way; the formula was also established in Lemma 1 of §10 by using the reflection principle.

§II. Martingales. Some Applications to the Random Walk

107

Let us interpret ~i = + 1 as a vote for candidate A and ~i = -1 as a vote for B. Then Sk is the difference between the numbers of votes cast for A and B at the time when k votes have been recorded, and P{S 1 > 0, ... , Sn > OISn =a- b}

is the probability that A was always ahead of B, with the understanding that A received a votes in all, B received b votes, and a - b > 0, a + b = n. According to (15) this probability is (a - b)/n.

6. PROBLEMS 1. Let !?} 0 ~ !!} 1 ~···~!!}.be a sequence of decompositions with !?} 0 = {Q}, and Jet ''lk be !?}k-measurable variables, 1 ~ k ~ n. Show that the sequence~ = (~b !!}k) with k

I

~k =

I= I

['11 - E('ltl !?}I-I)]

is a martingale. 2. Let the random variables '1~o····'1k satisfy E('1ki'11, ••• ,'1k-l) = 0. Show that the sequence~ = (~k) 1 ,;k,;n with ~ 1 = 17 1 and k

~k+ 1 =

I

i=l

'li+lli<'11· ... , 11J,

where fi are given functions, is a martingale. 3. Show that every martingale ~ = (~i> !!}k) has uncorrelated increments: if a< b < c < d then

4. Let ~ = (~ 1 , ... , ~.) be a random sequence such that ~k is !?}k-measurable (!!} ~ !!} 2 ~ · · · ~ !?}.). Show that a necessary and sufficient condition for this sequence to be a martingale (with respect to the system (!!}k)) is that E~t = E~ 1 for every stopping timeT (with respect to (!!}k)). (The phrase "for every stopping time" can be replaced by "for every stopping time that assumes two values.") 5. Show that if~ =

(~k•

!!}k) 1,;k,;n is a martingale and Tis a stopping time, then

for every k. 6. Let~= (~k• !!}k) and '1 = ('lk• !!}k) be two martingales, ~ 1 = 11 1 = 0. Show that

E~n'ln =

I

k=2

E(~k - ~k-1)('1k - 'lk-1)

and in particular that

E~; =

I

k=2

E(~k - ~k-1) 2 •

108

I. Elementary Probability Theory

7. Let 111, ... , Yfn be a sequence of independent identically distributed random variables with E11; = 0. Show that the sequence~ = (~k) with

~k = ~

(.I

l=l

'7;)

2

-

kEYff,

_ exp A.('7 1 + · · · + Yfk) (E exp A.q 1)k

k -

is a martingale. 8. Let '7t, ... , Yfn be a sequence of independent identically distributed random variables taking values in a finite set Y. Let f 0 (y) = P('7 1 = y), y E Y, and let f 1(y) be a nonnegative function with LyeY f 1(y) = 1. Show that the sequence ~ = (~k• !0Z) with

~:J = D~"····~k' ~k =

J; (111) .. . J; ('1d fo('11) · · · foCYfk)'

is a martingale. (The variables ~k, known as likelihood ratios, are extremely important in mathematical statistics.)

§12. Markov Chains. Ergodic Theorem. Strong Markov Property 1. We have discussed the Bernoulli scheme with Q

= {w: W = (xlo ... , Xn), X; = 0, 1},

where the probability p(w) of each outcome is given by (1)

= pxql-x. With these hypotheses, the variables ~ 1 , ... , ~n = X; are independent and identically distributed with X= 0, 1. P(~ 1 = x) = .. · = P(~. = x) = p(x),

with p(x) ~;(w)

with

If we replace (1) by p(w) = P1(x1) · · · Pn(xn),

where P;(x) = pf(1 - p;), 0 :::; p; :::; 1, the random variables ~ 1 , still independent, but in general are differently distributed:

... ,

~.

are

We now consider a generalization that leads to dependent random variables that form what is known as a Markov chain. Let us suppose that

n=

{w: w = (xo. xl, ... ' x.), X; EX},

109

§12. Markov Chains. Ergodic Theorem. Strong Markov Property

where X is a finite set. Let there be given nonnegative functions p0 (x), Pt(x, y), ... , Pn(x, y) such that

L Po(x) =

1,

L Pk(x, y) =

1,

XEX

k=1, ... ,n; yEX.

(2)

yeX

(3)

It is easily verified that Lroe!l p(w) = 1, and consequently the set of numbers p(w) together with the space Q and the collection of its subsets defines a probabilistic model, which it is usual to call a model of experiments that form a Markov chain. Let us introduce the random variables ~ 0 , ~ 1 , ... , ~n with ~;(w) =X;. A simple calculation shows that P(~ 0 = a) = p 0 (a),

P(~o

=

ao, ... , ~k

=

ak)

=

Po(ao)Pt(ao, at)··· Pk(ak-1• ak).

(4)

We now establish the validity of the following fundamental property of conditional probabilities: P{~k+t

=

ak+tl~k

=

ak, ... , ~o

= ao} =

P{~k+t

=

ak+tl~k =ad

(5)

(under the assumption that P(~k = ak, ... , ~ 0 = a0 ) > 0). By (4), P{(k+t

=

ak+tl~k

P{~k+t

=

ak> ·· ., ~o

= ao}

= ak+t' ·· ., ~o = ao}

P{ ~k = ak> ... , ~o = a 0 }

=

Po(ao)Pt(ao, a1) · · · Pk+ 1(ak, ak+ 1) Po(ao) · · · Pk(ak-l' ak)

=

( ) Pk+t abak+t.

In a similar way we verify (6)

which establishes (5). Let !0i = !0~o •... , ~k be the decomposition induced by ~ 0 , ••. , ~k> and P.Bi = a(!0i). Then, in the notation introduced in §8, it follows from (5) that (7)

or

I. Elementary Probability Theory

110 If we use the evident equation P(AB I C)

= P(A I BC)P(B I C),

we find from (7) that P{ ~n

=

an,·.·, ~k+ 1 = ak+11.?4D

= Pgn =

an,···, ~k+ 1 = ak+ll ~k}

(8)

or P{~" =an, ... , ~k+1

= ak+1l~o, ... , ~d = P{~" =a.,···, ~k+1 = ak+ll~k}. (9)

This equation admits the following intuitive interpretation. Let us think of ~k as the position of a particle "at present," (~ 0 , •.. , ~k- 1 ) as the "past," and (~k+ 1- ... , ~.)as the "future." Then (9) says that if the past and the present are given, the future depends only on the present and is independent of how the particle arrived at ~k' i.e. is independent of the past (~ 0 , ... , ~k- 1 ). Let F =(~.=a., ... , ~k+t = ak+t), N = gk =ad, B

= {~k- 1 =

ak-1• · · ·, ~o

= ao}.

Then it follows from (9) that P(FINB) = P(FIN),

from which we easily find that P(FB IN) = P(F I N)P(B IN).

(10)

In other words, it follows from (7) that for a given present N, the future F and the past B are independent. It is easily shown that the converse also holds: if (10) holds for all k = 0, 1, ... , n - 1, then (7) holds for every k = 0, 1, ... , n - 1.

The property of the independence of future and past, or, what is the same thing, the lack of dependence of the future on the past when the present is given, is called the Markov property, and the corresponding sequence of random variables ~ 0 , ••. , ~.is a Markov chain. Consequently if the probabilities p(w) of the sample points are given by (3), the sequence (~ 0 , ... , ~.)with ~;(w) =X; forms a Markov chain. We give the following formal definition.

Definition. Let (Q, d, P) be a (finite) probability space and let~ = (~ 0 , ••. , ~.) be a sequence of random variables with values in a (finite) set X. If (7) is satisfied, the sequence~ = (~ 0 , ... , ~")is called a (finite) Markov chain. The set X is called the phase space or state space of the chain. The set of probabilities (Pn(x)), x EX, with p 0 (x) = P(~ 0 = x) is the initial distribution, and the matrix IIPk(x,y)ll, x, yEX, with p(x,y) = Pgk = Yl~k-t = x} is the matrix of transition probabilities (from state x to state y) at time k = 1, ... , n.

§12. Markov Chains. Ergodic Theorem. Strong Markov Property

Ill

When the transition probabilities Pk(x, y) are independent of k, that is, = is called a homogeneous 0 , ..• , Markov chain with transition matrix llp(x, y)ll-

Pk(x, y) = p(x, y), the sequence

e (e

en)

Notice that the matrix ll(x, y)ll is stochastic: its elements are nonnegative and the sum of the elements in each row is 1: p(x, y) = 1, x EX. We shall suppose that the phase space X is a finite set of integers (X = {0, 1, ... , N}, X = {0, ± 1, ... , ± N}, etc.), and use the traditional notation Pi = p0 (i) and Pii = p(i, j). It is clear that the properties of homogeneous Markov chains completely determine the initial distributions Pi and the transition probabilities Pii· In specific cases we describe the evolution of the chain, not by writing out the matrix IIPiill explicitly, but by a (directed) graph whose vertices are the states in X, and an arrow from state ito state j with the number Pii over it indicates that it is possible to pass from point i to point j with probability Pii· When Pii = 0, the corresponding arrow is omitted.

LY

Pii

~i j EXAMPLE

1. Let X = {0, 1, 2} and

IIPijll

=

(P i). 3·

0

3

The following graph corresponds to this matrix:

.l2

I

2

~a~_f).l3 ·o~2~ ~

Here state 0 is said to be absorbing: if the particle gets into this state it remains there, since p00 = 1. From state 1 the particle goes to the adjacent states 0 or 2 with equal probabilities; state 2 has the property that the particle remains there with probability! and goes to state 0 with probability i ExAMPLE

for

2. Let X= {0, ± 1, ... , ±N}, Po= 1, PNN = P(-N)(-N) = 1, and,

Iii< N,

p, j = i + 1, { Pii = q, j = i - 1, 0 otherwise.

(11)

112

I. Elementary Probability Theory

The transitions corresponding to this chain can be presented graphically in the following way (N = 3):

This chain corresponds to the two-player game discussed earlier, when each player has a bankroll N and at each turn the first player wins + 1 from the second with probability p, and loses (wins -1) with probability q. If we think of state i as the amount won by the first player from the second, then reaching state N or - N means the ruin of the second or first player, respectively. In fact, if 17 1 , 17 2, ... , 'ln are independent Bernoulli random variables with P('l; = + 1) = p, P(1]; = -1) = q, S 0 = 0 and Sk = 17 1 + .. · + 'lk the amounts won by the first player from the second, then the sequence S0 , S 1, ••• , Sn is a Markov chain with p0 = 1 and transition matrix (11 ), since P{Sk+1 =jiSk = ik,Sk-1 = ik-1, ... } = P{Sk = P{Sk

+ '7k+1 + '7k+1

=jiSk = ik,Sk-1 = ik-1, ... } =jiSk = ik} = P{'7k+1 =j- ik}.

This Markov chain has a very simple structure: 0

~

k

~

n- 1,

where 17 1 ,17 2 , ..• , 'ln is a sequence of independent random variables. The same considerations show that if ~ 0 , 17 1 , •.. , 'ln are independent random variables then the sequence ~ 0 , ~ 1 , ••• , ~n with 0

~

k ~ n- 1,

(12)

is also a Markov chain. It is worth noting in this connection that a Markov chain constructed in this way can be considered as a natural probabilistic analog of a (deterministic) sequence x = (x 0 , •.• , xn) generated by the recurrent equations

We now give another example of a Markov chain of the form (12); this example arises in queueing theory. EXAMPLE 3. At a taxi stand let taxis arrive at unit intervals of time (one at a time). If no one is waiting at the stand, the taxi leaves immediately. Let 'lk be the number of passengers who arrive at the stand at time k, and suppose that 1] 1 , •.• , 'ln are independent random variables. Let ~k be the length of the

113

§12. Markov Chains. Ergodic Theorem. Strong Markov Property

waiting line at time k, ~ 0 = 0. Then if ~k = i, at the next time k ~k + 1 of the waiting line is equal to j.=

{1Jk+ 1 i- 1

+ 1Jk+ 1

+ 1 the length

if i = 0, ifizl.

In other words, 0

s

k

s

n- 1,

where a+ = max(a, 0), and therefore the sequence ~ = (~ 0 , Markov chain.

..• ,

~")

is a

4. This example comes from the theory of branching processes. A branching process with discrete times is a sequence of random variables ~ 0 , ~ 1 , .•. , ~"'where ~k is interpreted as the number of particles in existence at time k, and the process of creation and annihilation of particles is as follows: each particle, independently of the other particles and of the "prehistory" of the process, is transformed into j particles with probability pi, j = 0, 1, ... , M. We suppose that at the initial time there is just one particle, ~ 0 = 1. If at time k there are ~k particles (numbered 1, 2, ... , ~k), then by assumption ~k+ 1 is given as a random sum of random variables, EXAMPLE

~k+ 1

=

IJlk)

+ ... + 1]~~,

where 1Jlkl is the number of particles produced by particle number i. It is clear that if ~k = 0 then ~k+ 1 = 0. If we suppose that all the random variables IJ~k>, k 2 0, are independent of each other, we obtain Pgk+1 = ik+1i~k = ik, ~k-1 = ik-t• · · .} = Pgk+t = ik+ti~k = ik} = P{IJ\kl + · · · + IJ!:> = ik+ d. It is evident from this that the sequence ~ 0 , ~ 1 ,

... , ~"is a Markov chain. A particularly interesting case is that in which each particle either vanishes with probability q or divides in two with probability p, p + q = 1. In this case it is easy to calculate that

is given by the formula Pii

=

{c 0

1,."12pi12qi- j/2, ). = 0' ... , 2'I, in all other cases.

2. Let ~ = ( ~k, Ill, IP') be a homogeneous Markov chain with st~rting vectors (rows) Ill = (p;) and transition matrix Ill = IIPiill. It is clear that Pii = P{~1 =j/~o = i} = ... = P{~n =j/~n-1 = i}.

114

I. Elementary Probability Theory

We shall use the notation

for the probability of a transition from state i to state j in k steps, and

for the probability of finding the particle at point j at time k. Also let

Let us show that the transition probabilities pj'> satisfy the Kolmogorov-

Chapman equation

\~ I) = " p\k)p(l! P'1 l.J ra ~J'

(13)

~

or, in matrix form,

(14) The proof is extremely simple: using the formula for total probability and the Markov property, we obtain P!~+IJ

= P(~k+l = j l~o =

i)

=

L P(~k+l = j, ~k = IXI~o = i) ~

=

L P(~k+l = j I~k = 1X)P(~k = IX I~0 = i) = L p~}P!~>. ~

~

The following two cases of (13) are particularly important: the backward equation 1J = "P· P\~+ IJ £...., I~ p
(15)

v

and the forward equation \~+ 1) P IJ

= "p\k)p" . £...., I~ ~}

(16)

~

(see Figures 22 and 23). The forward and backward equations can be written in the following matrix forms

= IP. IP,

(17)

IP.

(18)

IP
§12. Markov Chains. Ergodic Theorem. Strong Markov Property

0

115

I+ I

Figure 22. For the backward equation.

Similarly, we find for the (unconditional) probabilities p)k) that

'\' p(k)p(l) PJ(_k +I) = ~ a: '2.]'

(19)

a

or in matrix form

fl (k +I) = fl (k) • I]J>(I). In particular,

rn
rn
·IP

(forward equation) and

fl (k+ 1) = fl (1).

I]J>(k)

(backward equation). Since IP 0 ) = IP, fl (1) = fl, it follows from these equations that JP(k) =

!Pk,

Consequently for homogeneous Markov chains the k-step transitiOn probabilities p~j) are the elements of the kth powers of the matrix IP, so that many properties of such chains can be investigated by the methods of matrix analysis.

k k

+1

Figure 23. For the forward equation.

116

I. Elementary Probability Theory

5. Consider a homogeneous Markov chain with the two states 0 and 1 and the matrix

EXAMPLE

IFD

= (Poo Pot)· Pu

Pto

It is easy to calculate that p2

Po;(Poo + Pu)) Ptt + PotPto

= ( P~o + Po1P1o Pto(Poo

and (by induction)

+ Pu)

1 (1 - 1-

IFD" _ - 2 - Poo - Pu

Pu 1 - Pu

Poo) 1 - Poo

+ (Poo + Pu

- 1)" ( 1 - Poo 2 - Poo - Pu -(1 - Pu)

-(1 - Poo)) 1- Pu

(under the hypothesis that IPoo + p 11 - 11 < 1). Hence it is clear that if the elements of IFD satisfy Ip00 + p11 - 11 < 1 (in particular, if all the transition probabilities Pii are positive), then as n-+ oo IFD" -+

1 (1 - 1-

2 - Poo - Pu

Pu 1 - Pu

Poo) 1 - Poo '

(20)

and therefore l - p 11 lim (nl = P,o 2 ' n - Poo- Pu

. Pil(n) 1tm "

1 - Poo

= -,-------'--'---

2- Poo- Pu

Consequently if IPoo + p 11 - 11 < 1, such a Markov chain exhibits regular behavior of the following kind: the influence of the initial state on the probability of finding the particle in one state or another eventually becomes negligible (plj> approach limits ni, independent of i and forming a probability distribution: n0 ~ 0, n 1 ~ 0, n0 + n 1 = 1); if also all Pii > 0 then n0 > 0 and n 1 > 0. 3. The following theorem describes a wide class of Markov chains that have the property called ergodicity: the limits ni = limn Pii not only exist, are independent of i, and form a probability distribution (ni ~ 0, Li ni = 1}, but also ni > 0 for allj (such a distribution ni is said to be ergodic).

Theorem 1 (Ergodic Theorem). Let 1P

= IIPiill be the transition matrix of a chain with a .finite state space X = {1, 2, ... , N}.

(a) If there is an n0 such that min p\~ol > 0' lJ i,j

(21)

117

§12. Markov Chains. Ergodic Theorem. Strong Markov Property

then there are numbers n 1,

••• ,

nN

such that (22)

and

n- oo

(23)

for every i e X. if there are numbers n 1, ••. , nN satisfying (22) and (23), there is an n0 such that (21) holds. (c) 1he numbers (n 1 , ... , nN) satisfy the equations

(b) Conversely,

= 1, ... ,N.

j PROOF.

(24)

(a) Let

m(n> = min p!~> ) I] '

M)"> = max Pl~P.

i

i

Since 1) = "P· p
(25)

~

we have 1 > =min p!'!+ 1 > =min" P· pmin" P· min p<") = m(n> m("+ ) I) ' L, I~ ~) L, I~ ~) ) '

i

i

a

i

eX

a

whence m}"> :::;; m)"+ >and similarly M}"> ~ M}"+ 1l. Consequently, to establish (23) it will be enough to prove that 1

M("> - m("> .,... 0 1

Let e

=

1

n .,... oo, j = 1, ... , N.

'

min;, i Pljo> > 0. 'fhen = "p(no>p]p<") P!'!o+n) I) L, I~ ~) I~ )~ ~) ~

+ s "p(">p
~

~)

~

(p(no) - sp(">]p
= "

L,

+ sp(~n) 11 •

~

But p!no) - sp("> > 0 '· therefore la Ja. -

> m(n)." [p!no) - sp(">J P!'!o+n) I) ) L, I~ )~

+ sp(~n) = ))

m("l(1 - s) )

~

and consequently m!no+n) > m("l(1 - s) 1 1

+ sp(~n) )) •

M(no+n> < M(">(1 - s) 1 1

+ sp!~n> )) •

In a similar way Combining these inequalities, we obtain M(no+n> _ m(no+n> < (M("> _ m(">). (1 _ s) 1

1

-

1

1

+ sp(~n) Jl '

118

I. Elementary Probability Theory

and consequently M<J<no+n> _ m<.kno+n> < (M<.n> _ m<.">)(1 _ e)k! 0 J

J

-

J

J

'

k--+ 00.

Thus M~"tll - m~npl --+ 0 for some subsequence np, np --+ oo. But the difference M~nl - m~n) is monotonic in n, and therefore M~nl - m~nl --+ 0, n--+ oo. If we put ni = limn m~">, it follows from the preceding inequalities that lp \~)IJ

7C·I < M(n)m<.n> < (1 - e)[n/no)-1 J J J -

for n ~ n0 , that is, p!j> converges to its limit ni geometrically (i.e., as fast as a geometric progression). It is also clear that m~n> ~ m~no) ~ e > 0 for n ~ n0 , and therefore ni > 0. (b) Inequality (21) follows from (23) and (25). (c) Equation (24) follows from (23) and (25). This completes the proof of the theorem. 4. Equations (24) play a major role in the theory of Markov chains. A nonnegative solution (n 1 , ••• , nN) satisfying I,. n~ = 1 is said to be a stationary or invariant probability distribution for the Markov chain with transition matrix IIPiill. The reason for this terminology is as follows. Let us select an initial distribution (n 1, ... , nN) and take Pi= ni. Then PJ(l)

= "" 1C

p . = 1C.J

£...,~~} ~

and in general p)"> = ni. In other words, if we take (n 1, ... , nN) as the initial distribution, this distribution is unchanged as time goes on, i.e. for any k j = 1, ... ,N.

Moreover, with this initial distribution the Markov chain ~ =(~,Ill, IP) is really stationary: the joint distribution of the vector (~k• ~k+ ~> ... , ~k+ 1) is independent of k for alll (assuming that k + 1 :::;; n). Property (21) guarantees both the existence of limits ni = lim plj>, which are independent of i, and the existence of an ergodic distribution, i.e. one with ni > 0. The distribution (n 1 , •.• , nN) is also a stationary distribution. Let us now show that the set (n 1 , .•. , nN) is the only stationary distribution. In fact, let (ft 1, •.• , ftN) be another stationary distribution. Then

and since P~1--+ ni we have iii

= L (ft~ · n) = ni. ~

These problems will be investigated in detail in Chapter VIII for Markov chains with countably many states as well as with finitely many states.

119

§12. Markov Chains. Ergodic Theorem. Strong Markov Property

We note that a stationary probability distribution (even unique) may exist for a nonergodic chain. In fact, if

then [p>2n

=

(0 1)

1 0 '

[p>2n+ 1

=

(1 0)

0 1 '

and consequently the limits lim pjj> do not exist. At the same time, the system j

= 1, 2,

reduces to

of which the unique solution satisfying n 1 + n 2 = 1 is(!,!). We also notice that for this example the system (24) has the form no

=

noPoo

+ n1P1o·

from which, by the condition n 0 = n 1 = 1, we find that the unique stationary distribution (n 0 , n 1) coincides with the one obtained above: no=

1- Pu ' 2 - Poo - Pu

nl

=

1- Poo

2- Poo-

P11

.

We now consider some corollaries of the ergodic theorem. Let A be a set of states, A ~ X and IA(x)

1, XEA, = { 0, x ¢:A.

Consider

which is the fraction of the time spent by the particle in the set A. Since E[JA((k)i(o = i] = P((kEAI(o = i) = LPI,l(=p\kl(A)), jeA

120

I. Elementary Probability Theory

we have

1

E[vA(n)l~o = i] = - 1

n

+

and in particular

E[vw(n)l~o = i] = -

n

1-

L pjkl(A) k=o n

±

+ 1 k=o

pjj>.

It is known from analysis (see also Lemma 1 in §3 of Chapter IV) that if an-+ a then (a 0 + · · · + an)/(n + 1)-+ a, n-+ oo. Hence if pj'l-+ ni, k-+ oo, then where

nA =

L ni.

jeA

For ergodic chains one can in fact prove more, namely that the following result holds for I A(~ 0 ), ••• , I A( ~n), ....

Law of Large Numbers. If ~ 0 , ~ 1 , ... form a .finite ergodic Markov chain, then

n-+ oo,

(26)

for every e > 0 and every initial distribution. Before we undertake the proof, let us notice that we cannot apply the results of §5 directly to IA(~ 0 ), ... , JA(~n), ... , since these variables are, in general, dependent. However, the proof can be carried through along the same lines as for independent variables if we again use Chebyshev's inequality, and apply the fact that for an ergodic chain with finitely many states there is a number p, 0 < p < 1, such that (27)

Let us consider states i and j (which might be the same) and show that, fore> 0,

n-+ oo. By Chebyshev's inequality,

Hence we have only to show that

n-+ oo.

(28)

121

§12. Markov Chains. Ergodic Theorem. Strong Markov Property

A simple calculation shows that

=

1 (n

+

n

n

'\'

'\' m!~· I) L., l} ' k=o l=o L.,

1)2

where

s = min(k, l) and t = lk- 11. By (27),

P!'!> l}

= n:.J + e!'!>

l}'

Therefore lml~· 1 >1:::; C1[p•

+ P + Pk + /], 1

where C 1 is a constant. Consequently

1

n

n

--,-----____,.,...,.. '\' '\' m
<

- (n

C

n

1

'\'

n

'\' [

s

+ 1)2 k~o 1~0 p 4C 1 2(n + 1) 2 + 1) 1 - p

+

t

p

+

k+

p

1]

p

8C 1

(n

+ 1)(1

- p)

--+

O

,

n --+ oo.

Then (28) follows from this, and we obtain (26) in an obvious way. 5. In §9 we gave, for a random walk S 0 , S 1, ... generated by a Bernoulli scheme, recurrent equations for the probability and the expectation of the exit time at either boundary. We now derive similar equations for Markov chains. Let~= (~ 0 , ••• , ~")be a Markov chain with transition matrix IIPiill and phase space X = {0, ± 1, ... , ± N}. Let A and B be two integers, - N :::; A :::; 0 :::; B :::; N, and x EX. Let f!lk+ 1 be the set of paths {x 0 , x 1 , ..• , xk), xi E X, that leave the interval (A, B) for the first time at the upper end, i.e. leave (A, B) by going into the set (B, B + 1, ... , N). , For A :::; x :::; B, put f3k(x) = P{(~o .... , ~k) E fflk+ll~o = x}.

In order to find these probabilities (for the first exit of the Markov chain from (A, B) through the upper boundary) we use the method that was applied in the deduction of the backward equations.

122

I. Elementary Probability Theory

We have {3k(x) = P{(eo, ... ' ek) E .?Jk+ 11 eo = X} =

L Pxy. P{(eo, ... ' ek) E .?Jk+lleo =X, el = y

y},

where, as is easily seen by using the Markov property and the homogeneity of the chain,

P{(eo, ... ' ek)

.?Jk+lleo =X, el = y} = P{(x, y, e2, ... ' ek) E .?Jk+lleo =X, el = P{(y, e2, ... , ek) E BBkle1 = y}

E

= y}

= P{(y, eb ... ' ek- dE .?lkl eo = y} = Pk-l(y). Therefore y

for A < x < B and 1 :::;; k :::;; n. Moreover, it is clear that Pix)= 1,

X

= B, B + 1, ... ' N,

and X= -N, ... ,A.

In a similar way we can find equations for ocix), the probabilities for first exit from (A, B) through the lower boundary. Let rk = min{O :::;; l :::;; k: ¢(A, B)}, where rk = k if the set { ·} = 0. Then the same method, applied to mix)= E(rkl eo = x), leads to the following recurrent equations:

e,

mk(x) = 1 +

L mk-l (Y)Pxy y

(here 1 :::;; k :::;; n, A < x < B). We define

x ¢(A, B). It is clear that if the transition matrix is given by (11) the equations for ock(x), {3k(x) and mk(x) become the corresponding equations from §9, where they were obtained by essentially the same method that was used here. These equations have the most interesting applications in the limiting case when the walk continues for an unbounded length of time. Just as in §9, the corresponding equations can be obtained by a formal limiting process (k --+

CX) ).

By way of example, we consider the Markov chain with states {0, 1, ... , B} and transition probabilities

Poo

= 1,

PBB =

1,

123

§12. Markov Chains. Ergodic Theorem. Strong Markov Property

and

Pi > 0, ~ = ~ + 1, pij = { ri, J = z, qi > 0, j = i - 1, for 1 :-:;; i :-:;; B - 1, where Pi + qi + ri = 1. For this chain, the corresponding graph is

a~---·~e 0

q1

1

2

P1

qB-1 B- 1 PB-1

B

It is clear that states 0 and B are absorbing, whereas for every other state i the particle stays there with probability ri, moves one step to the right with probability Pi• and to the left with probability qi. Let us find a(x) = limk-oo ak(x), the limit of the probability that a particle starting at the point x arrives at state zero before reaching state B. Taking limits ask--+ oo in the equations for aix), we find that

when 0 < j < B, with the boundary conditions

a(O)

=

1,

a(B) = 0.

Since rj- 1 - qj- pj, we have pj(a(j + 1) - a(j)) = qj(a(j) - a(j - 1)) and consequently a(j

+ 1) -

a(j) = pj(a(1) - 1),

where Po= 1.

But a(j

+ 1) -

j

1=

L (a(i + 1) -

a(i)).

i=O

Therefore a(j

+ 1) -

j

1 = (a(1) - 1) ·

L Pi·

i=l

124

I. Elementary Probability Theory

Ifj = B - 1, we have r:~.(j

+ 1) =

0, and therefore

r:~.(B) =

1

r:J.(l)=1=-'\'B-1

L..i= 1 Pi

'

whence an d

'\'B-1

L..i=i Pi ' L..i=1 Pi

( r:I.J

0)

='\'B-1

j = 1,

0

0

0'

B.

(This should be compared with the results of §9.) Now let m(x) = limk mk(x), the limiting value of the average time taken to arrive at one of the states 0 or B. Then m(O) = m(B) = 0, m(x)

= 1 + L m(y)Pxy y

and consequently for the example that we are considering,

m(j) = 1 + qimU - 1)

+ rimU) + pimU + 1)

for j = 1, 2, ... , B - 1. To find mU) we put

M(j) = m(j) - m(j- 1),

j = 0, 1,. 0. 'B.

Then

piM(j + 1)

= qiM(j)

- 1,

j = 1, ... , B- 1,

and consequently we find that

M(j

+ 1) =

piM(1)- Ri,

where q1 .. qj 0

P1 · · Pi ' 0

Therefore j-1

m(i) = mU)- m(O) =

L M(i + 1)

i=O

j-1

=

L

i=O

(pim(l)- Ri) = m(l)

j-1

j-1

i=O

i=O

L Pi- L Ri.

It remains only to determine m(1). But m(B) = 0, and therefore '\'B-l R m(1) = Ik=o1 i, i=O Pi

and for 1 < j ::;; B,

( 0)

mJ =

j-1

'\'~-1R.

j-1

'\' R L..• = 0 I '\' i~o Pi. Lf;-o1 Pi - i:--o io

125

§12. Markov Chains. Ergodic Theorem. Strong Markov Property

(This should be compared with the results in §9 for the case ri = 0, Pi = p,

qi

= q.)

6. In this subsection we consider a stronger version of the Markov property (8), namely that it remains valid if time k is replaced by a random time (see also Theorem 2). The significance of this, the strong Markov property, can be illustrated in particular by the example of the derivation of the recurrent relations (38), which play an important role in the classification of the states of Markov chains (Chapter VIII). Let ~ = (~ 1 , . . . , ~n) be a homogeneous Markov chain with transition matrix IIPijll; let ~~ = (~f)oo;;k,;;n be a system of decompositions, ~f = ~ ~o •...• ~k. Let ~f denote the algebra ri(~i) generated by the decomposition ~f.

B

We first put the Markov property (8) into a somewhat different form. Let Let us show that then

E ~f.

= ak+llB n (~k = ak)} = P{~n =an, ... , ~k+l =

Pgn =an,···, ~k+l

ak+ll~k

= ak}

(29)

(assuming that P{B n (~k = ak)} > 0). In fact, B can be represented in the form

B=

where

I* {~~ .... , ~k = at},

I* extends over some set (a~, ... ' an Consequently P{~n =an, .. ·, ~k+l P{(~n

= ak+llB n

(~k

= ak)}

=an, ... , ~k = ak) n B} P{(~k = ak) n B}

I* P{(~n

= aw .. , ~k = ak) n (~ 0 = a6, ... , ~k =a%)} P{(~k = ak) n B}

(30)

But, by the Markov property, P{(~n

=an, ... , ~k = ak) n (~o = a~, ... , ~k =at)}

=

{

P{~n =an, ... , ~k+l = ak+ll~o =a~,···, ~k =at} x P{~ 0 =a~, ... , ~k =at} ifak =at, 0 if ak i= at,

=

{

P{~n =an, ... , ~k+l = ak+ll~k = ak}P{~o =a~, ... , ~k =at} if ak =at, 0 if ak i= at,

={

pgn =an,··., ~k+1 0

if ak =at, if ak i= at.

= ak+ll~k = ak}P{(~k = ak) n B}

126

I. Elementary Probability Theory

Therefore the sum I* in (30) is equal to P{ ~n = an, ... , ~k+ 1 = ak+ d ~k = adP{(~k = ak) n B},

This establishes (29). Let r be a stopping time (with respect to the system D~ = (DDo 5 k 5 n; see Definition 2 in §11).

Definition. We say that a set Bin the algebra~~ belongs to the system of sets ~; if, for each k, 0 ~ k ~ n, B n {r

= k}

(31)

E ~f.

It is easily verified that the collection of such sets B forms an algebra (called the algebra of events observed at timer).

Theorem 2. Let ~ = (~ 0 , ••• , ~n) be a homogeneous Markov chain with transition matrix IIPull, r a stopping time (with respect to 2fi~), BE~~ and A = {w: r + l ~ n}. Then if P{A n B n (~, = a0 )} > 0, we have P{~r+t

and if P{A n

(~,

= =

a1,

••• ,

P{~r+t

= a 1 IA n B n (~, = a 0 )} = a1, ••• , ~r+l = a 1 IA n (~, = a 0 )}, ~r+l

(32)

= a0 )} > 0 then

For the sake of simplicity, we give the proof only for the case l = 1. Since B n (r = k) E ~L we have, according to (29), P{~r+l =

a 1, An B n (~, = a0 )}

L

k5n-l

P{~k+l

= a1, ~k =

a0 , r

= k, B}

k5n-l k5n-l

=

Paoal.

L

k5n-l

P{~k = ao, r = k, B} =

Paoal.

P{A n B n (~,

= ao)},

which simultaneously establishes (32) and (33) (for (33) we have to take B= Q).

Remark. When l

= 1, the strong Markov property (32), (33) is evidently equivalent to the property that

(34)

127

§12. Markov Chains. Ergodic Theorem. Strong Markov Property

for every C s;; X, where

Pao(C) =

L Paoa1·

a1EC

In turn, (34) can be restated as follows: on the set A = {T

~

n- 1}, (35)

which is a form of the strong Markov property that is commonly used in the general theory of homogeneous Markov processes. 7. Let ~ = (~ 0 , .••• , matrix IIPiill, and let

~n)

be a homogeneous Markov chain with transition

!!7>

= P{ ~k = i, ~~ =F i, 1 ~ l ~ k - 11 ~ 0 = i}

(36)

f!J>

= P{~k =j, ~~ #j,

1 ~ l ~ k- 11~ 0 = i}

(37)

and

for i =F j be respectively the probability of first return to state i at time k and the probability of first arrival at state j at time k. Let us show that n

P I}\~)

= i..J "'

j\~>p<.~-k)

k=1

I}

}} ·

'

where

p)~> =

1.

(38)

The intuitive meaning of the formula is clear: to go from state i to state j inn steps, it is necessary to reach statej for the first time ink steps (1 ~ k ~ n) and then to go from state j to state j in n - k steps. We now give a rigorous derivation. Let j be given and ' = min{l ~ k ~ n: ~k = j}, assuming that'! = n

+ 1 if {·}

=

0. Then f!J> = P{T =

kl~o = i} and

P!~j> = P{~n =jl~o = i}

L

P{~n = j, '! = kl~o = i}

L

P{~r+n-k=j,T=kl~o=i},

15k5n

15k5n

(39)

where the last equation follows because ~t+n-k = ~n on the set {'! = k}. Moreover, the set {' = k} = {' = k, ~r = j} for every k, 1 ~ k ~ n. Therefore if P{~ 0 = i,' = k} > 0, it follows from Theorem 2 that P{~r+n-k =jl~o = i, '! = k} = P{~r+n-k =jl~o = i, '! = k, ~t =j}

= P{~t+n-k = j I~t = j} = ptf-k>

128

I. Elementary Probability Theory

and by (37)

P!j> = =

n

L P{~t+n-k = jl~o = i,

7:

k= 1

= k}P{r = kl~o = i}

n

~ p(~-k>:r(~)

L,

k=l

JJ

I] '

which establishes (38). 8.PROBLEMS

= (~ 0 , ... , ~.)be a Markov chain with values in X and f = f(x) (x EX) a function. Will the sequence (f(~ 0 ), ... ,f(~.)) form a Markov chain? Will the "reversed" sequence

1. Let ~

(~., ~.- 1> • • · ' ~o)

form a Markov chain? 2. Let IJll = IIP;)I, 1 :::; i,j:::; r, be a stochastic matrix and A. an eigenvalue of the matrix, i.e. a root of the characteristic equation det IIIJll - A.E II = 0. Show that A.0 = 1 is an eigenvalue and that all the other eigenvalues have moduli not exceeding 1. If all the eigenvalues ..1. 1, ••• , A., are distinct, then p~~l admits the representation p~~l = ni

+ a;;{1)A.~ + · ·· + a;k)A.~,

where ni, a;;(!), ... , a;;{r) can be expressed in terms of the elements of IJll. (It follows from this algebraic approach to the study of Markov chains that, in particular, when I..1. 1 1< 1, ... , IA., I < 1, the limit lim Pl~l exists for every j and is independent of i.) 3.

Let~ = (~ 0 , ••. , ~.)be

tion matrix

IJll

=

a homogeneous Markov chain with state space X and transi-

IIPxyll. Let

T
E[
(=

~
Let the nonnegative function