11.6. APPROXIMATIONS 201

Corollary 11.6.9 |Ax−b| ≤ |Ax̂−b| for all x̂ ∈ Rn if and only if 0 = (Ax−b,Az) forevery z ∈ Rn.

Proof:This is the content of the above theorem because 0 = (Ax−b,Az) for everyz ∈ Rn if and only if AT Ax= ATb. ■

The corollary says that the vector Ax−b is perpendicular to the subspace Im(A) is thesame as saying that Ax is as close as possible to b. Note that this orthogonality condition0=(Ax−b,Az) is equivalent to saying 0=(Ax−b,Ax−Az)= (b−Ax,Az−Ax). Hereis a picture which illustrates the conclusion of this important theorem.

•

• •

A(Fn)≡ Im(A)

Ay

b

Ax

Next consider the problem of projection onto a subspace. Letting V be a subspace ofRn, and letting b ∈Rn, how do we find x ∈V such that |b−x| ≤ |b−y| for every y ∈V ?

The subspace has a basis, {v1, · · · ,vm}, m≤ n. Let

A =(

v1 · · · vm 0 · · · 0)

Thus V is the column space of A, the span of the columns of A which is also Im(A). Thusthe question is to find Ay which is closer to b than any Az. Isn’t this just what was solvedabove? Then the closest point to b in V will be x ≡ Ay. From the above explanation, ymust satisfy

AT Ay = ATb so(b−

xAy,

uAz)= 0

for all z ∈Rn. In other words, you need x to be the point in V which satisfies (b−x,u) = 0for all u in V . Note that since x is in V , a generic point of V is of the form u−x. Thus itmakes no difference whether we write (b−x,u) = 0 for all u in V or (b−x,u−x) = 0for all u in V . The following picture illustrates what was just shown.

•

• •

A(Fn)≡ Im(A)≡V

u

b

x

Theorem 11.6.10 Let V be a finite dimensional subspace of Rn and let b ∈Rn. Then thereexists a unique point of V which is closest to b out of all points of V . This point x ischaracterized by the equation

(b−x,z) = 0 (11.7)

for all z ∈V.