Topics In Analysis

58.16 The Multivariate Normal Distribution

Definition 58.16.1 A random vector, X, with values in ℝ^p has a multivariate normal distribution written as X ∼N_p

if for all Borel E ⊆ ℝ^p,

\int 1 -1 * -1 λX (E ) = p XE (x)---p∕2------1∕2e 2 (x-m) Σ (x-m )dx ℝ (2π) det(Σ )

for μ a given vector and Σ a given positive definite symmetric matrix.

Theorem 58.16.2 For X ∼ N_p

,m = E

and

( *) Σ = E (X - m)(X - m ) .

Proof: Let R be an orthogonal transformation such that

RΣR * = D = diag (σ21,\cdot\cdot\cdot,σ2p).

Changing the variable by x − m = R^∗y,

\int -1 * -1 ( 1 ) E (X ) \equiv xe 2 (x-m) Σ (x-m )dx ----p∕2------1∕2 ℝp ((2π) det(Σ ) ) \int * - 1y*D -1y 1 = p (R y +m )e 2 dy ----p∕2\prodp---- ℝ\int ( (2π) ) i=1σi - 12y*D-1y ------1------ = m ℝp e dy (2π)p∕2\prodp σ = m i=1 i

by Fubini’s theorem and the easy to establish formula

\int \sqrt-1-- e- y22σ2dy = 1. 2πσ ℝ

Next let M ≡ E

. Thus, changing the variable as above by x − m = R^∗y

\int ( ) * -21(x-m )*Σ-1(x- m) -------1------- M = ℝp (x - m )(x - m ) e dx (2π)p∕2det(Σ)1∕2 \int ( ) = R * yy*e- 12y*D -1ydy ------1\prod----- R ℝp (2π)p∕2 pi=1σi

Therefore,

( ) * \int - 1y*D-1y ------1------ (RM R )ij = ℝp yiyje 2 dy (2π)p∕2\prodp σ = 0, i=1 i

so; RMR^∗ is a diagonal matrix.

\int 1 * -1 ( 1 ) (RM R *)ii = y2ie-2y D ydy ----p∕2\prodp----- . ℝp (2π) i=1σi

Using Fubini’s theorem and the easy to establish equations,

\int 2 \int 2 \sqrt-1-- e- y2σ2dy = 1,\sqrt-1-- y2e- y2σ2dy = σ2, 2πσ ℝ 2π σ ℝ

it follows

_ii = σ_i². Hence RMR^∗ = D and so M = R^∗DR = Σ. ■

Theorem 58.16.3 Suppose X₁ ∼ N_p

, X₂ ∼ N_p

and the two random vectors are independent. Then

X1 + X2 \sim Np (m1 +m2, Σ1 + Σ2). (58.16.31)

(58.16.31)

Also, if X ∼ N_p

then −X ∼ N_p

. Furthermore, if X ∼ N_p

then

( it\cdotX ) it\cdotm - 12t*Σt E e = e e (58.16.32)

(58.16.32)

Also if a is a constant and X ∼ N_p

then aX ∼ N_p

Proof: Consider E

for X ∼ N_p

\int E (eit\cdotX) \equiv -------1------- eit\cdotxe- 12(x- m)*Σ-1(x-m)dx. (2π )p∕2(detΣ)1∕2 ℝp

Let R be an orthogonal transformation such that

RΣR * = D = diag (σ21,\cdot\cdot\cdot,σ2p).

Then let R

= y. Then

\int E (eit\cdotX) = -----1------- eit\cdot(R*y+m )e- 12y*D -1ydy. (2π)p∕2\prodpi=1 σi ℝp

Therefore

( ) 1 \int 1* -1 E eit\cdotX = ----p∕2-\prodp---- p eis\cdot(y+Rm )e- 2y D ydy (2π) i=1σi ℝ

where s =Rt. This equals

it\cdotm \prodp( \int isiyi- 2σ12y2i ) ---1-- e ℝe e i dyi \sqrt2-πσi i=1

p (\int ) = eit\cdotm \prod eisiσiue- 12u2du \sqrt1-- i=1 ℝ 2π

p \int it\cdotm \prod - 12s2iσ2i-1- - 12(u-isiσi)2 = e e \sqrt2-π- ℝe du i=1

it\cdotm - 12\sumpi=1s2iσ2i it\cdotm - 12t*Σt = e e = e e

This proves 58.16.32.

Since X₁ and X₂ are independent, e^it⋅X₁ and e^it⋅X₂ are also independent. Hence

( it\cdotX1+X2) ( it\cdotX1) ( it\cdotX2) E e = E e E e .

Thus,

E (eit\cdotX1+X2) = E (eit\cdotX1)E (eit\cdotX2) it\cdotm1 - 1t*Σ1tit\cdotm2 - 1t*Σ2t = e e 2 1e* e 2 = eit\cdot(m1+m2)e-2t (Σ1+Σ2)t

which is the characteristic function of a random vector distributed as

Np (m1 +m2, Σ1 + Σ2).

Now it follows that X₁ + X₂ ∼ N_p

by Theorem 58.8.4. This proves 58.16.31.

The assertion about −X is also easy to see because

( ) ( ) E eit\cdot(-X) = E ei(-t)\cdotX \int = -------1------- ei(-t)\cdotxe- 12(x-m)*Σ-1(x-m )dx (2π)p∕2(detΣ)1∕2 ℝp 1 \int it\cdotx - 1(x+m )*Σ-1(x+m) = ---p∕2------1∕2- p e e 2 dx (2π) (detΣ) ℝ

which is the characteristic function of a random variable which is N

. Theorem 58.8.4 again implies −X ∼ N

. Finally consider the last claim. You apply what is known about X with t replaced with at and then massage things. This gives the characteristic function for aX is given by

( 1 ) E (exp (it\cdotaX)) = exp(it\cdotam )exp -2 t*Σa2t

which is the characteristic function of a normal random vector having mean am and covariance a²Σ. This proves the theorem.

Following [?] a random vector has a generalized normal distribution if its characteristic function is given as

it\cdotm - 1t*Σt e e 2 (58.16.33)

(58.16.33)

where Σ is symmetric and has nonnegative eigenvalues. For a random real valued variable, m is scalar and so is Σ so the characteristic function of such a generalized normally distributed random variable is

itμ- 1t2σ2 e e 2 (58.16.34)

(58.16.34)

These generalized normal distributions do not require Σ to be invertible, only that the eigenvalues be nonnegative. In one dimension this would correspond the characteristic function of a dirac measure having point mass 1 at μ. In higher dimensions, it could be a mixture of such things with more familiar things. I won’t try very hard to distinguish between generalized normal distributions and normal distributions in which the covariance matrix has all positive eigenvalues.

Here are some other interesting results about normal distributions found in [?]. The next theorem has to do with the question whether a random vector is normally distributed in the above generalized sense.

Theorem 58.16.4 Let X =

where each X_i is a real valued random variable. Then X is normally distributed in the above generalized sense if and only if every linear combination, ∑ _j=1^pa_iX_i is normally distributed. In this case the mean of X is

m = (E (X ),\cdot\cdot\cdot,E(X )) 1 p

and the covariance matrix for X is

Σ = E ((X - m )(X - m )*). jk j j k k

Proof: Suppose first X is normally distributed. Then its characteristic function is of the form

ϕ (t) = E (eit\cdotX) = eit\cdotme- 12t*Σt. X

Then letting a =

( \sump ) ( ) 1 * 2 E eit j=1aiXi = E eita\cdotX = eita\cdotme-2a Σat

which is the characteristic function of a normally distributed random variable with mean a ⋅ m and variance σ² = a^∗Σa. This proves half of the theorem. If X is normally distributed, then every linear combination is normally distributed.

Next suppose ∑ _j=1^pa_jX_j = a ⋅ X is normally distributed with mean μ and variance σ² so that its characteristic function is given in 58.16.34. I will now relate μ and σ² to various quantities involving the X_j. Letting m_j = E

,m =

^∗

( ) p p ( p p ) 2 μ = \sum ajE (Xj) = \sum ajmj,σ2 = E |( ( \sum ajXj - \sum ajmj) |) j=1 j=1 j=1 j=1 ( ( ) ) \sump 2 \sum = E|( ( aj(Xj - mj )) |) = ajakE ((Xj - mj )(Xk - mk )) j=1 j,k

It follows the mean of the normally distributed random variable, a ⋅ X is

\sum μ = ajmj = a\cdotm j

and its variance is

σ2 = a*E ((X - m)(X - m )*)a

Therefore,

E (eita\cdotX ) = eitμe- 12t2σ2

ita\cdotm - 1t2a*E((X-m )(X-m )*)a = e e 2 .

Then letting s = ta this shows

E (eis\cdotX) = eis\cdotme- 12s*E((X -m)(X -m)*)s is\cdotm - 1s*Σs = e e 2

which is the characteristic function of a normally distributed random variable with m given above and Σ given by

Σjk = E((Xj - mj)(Xk - mk)).

By assumption, a is completely arbitrary and so it follows that s is also. Hence, X is normally distributed as claimed. ■

Corollary 58.16.5 Let X =

,Y =

where each X_i,Y _i is a real valued random variable. Suppose also that for every a ∈ ℝ^p, a ⋅ X and a ⋅ Y are both normally distributed with the same mean and variance. Then X and Y are both multivariate normal random vectors with the same mean and variance.

Proof: In the Proof of Theorem 58.16.4 the proof implies that the characteristic functions of a ⋅ X and a ⋅ Y are both of the form

eitme- 12σ2t2.

Then as in the proof of that theorem, it must be the case that

\sump m = ajmj j=1

where E

= m_i = E

and

2 * ( *) σ = a E ((X - m )(X - m ) )a = a *E (Y - m )(Y - m)* a

and this last equation must hold for every a. Therefore,

E ((X - m )(X - m )*) = E ((Y - m )(Y - m )*) \equiv Σ

and so the characteristic function of both X and Y is e^is⋅me⁻

s^∗Σs as in the proof of Theorem 58.16.4. ■

Theorem 58.16.6 Suppose X =

is normally distributed with mean m and covariance Σ. Then if X₁ is uncorrelated with any of the X_i, meaning

E ((X1 - m1)(Xj - mj)) = 0 for j > 1,

then X₁ and