The particular solution of the least squares problem given in 14.30 is important enough that it motivates
the following definition.
Definition 14.16.1Let A be an m×n matrix. Then the Moore Penroseinverse of A, denoted by A^{+}isdefined as
( )
+ σ−1 0 ∗
A ≡ V 0 0 U .
Here
( )
U∗AV = σ 0
0 0
as above.
Thus A^{+}y is a solution to the minimization problem to find x which minimizes
Ax − y 
. In fact, one
can say more about this. In the following picture M_{y} denotes the set of least squares solutions x such that
A^{∗}Ax = A^{∗}y.
+
A (y)
M 
y x
 ker(A∗A )





Then A^{+}
(y)
is as given in the picture.
Proposition 14.16.2A^{+}y is the solution to theproblem of minimizing
Ax − y
for all x which hassmallest norm. Thus
 
AA+y − y ≤ Ax − y  for all x
and if x_{1}satisfies
Ax − y
1
≤
Ax − y
for allx,then
A+y 
≤
x 
1
.
Proof:Consider x satisfying 14.29, equivalently A^{∗}Ax =A^{∗}y,
( ) ( )
σ2 0 V∗x = σ 0 U∗y
0 0 0 0
which has smallest norm. This is equivalent to making
V∗x
as small as possible because V^{∗} is unitary and
so it preserves norms. For z a vector, denote by
(z)
_{k} the vector in F^{k} which consists of the first k entries of
z. Then if x is a solution to 14.29
( 2 ∗ ) ( ∗ )
σ (V x)k = σ (U y)k
0 0
and so
(V ∗x)
_{k} = σ^{−1}
(U∗y)
_{k}. Thus the first k entries of V^{∗}x are determined. In order to make
V ∗x
as
small as possible, the remaining n − k entries should equal zero. Therefore,
( ) ( ) ( )
∗ (V∗x)k σ− 1(U∗y)k σ− 1 0 ∗
V x = 0 = 0 = 0 0 U y
and so
( )
σ−1 0
x = V U ∗y ≡ A+y ■
0 0
Lemma 14.16.3The matrix A^{+}satisfies the following conditions.
+ + + + + +
AA A = A,A AA = A ,A A and AA are Hermitian. (14.31)
(14.31)
Proof:This is routine. Recall
( )
σ 0 ∗
A = U 0 0 V
and
( − 1 )
A+ = V σ 0 U ∗
0 0
so you just plug in and verify it works. ■
A much more interesting observation is that A^{+} is characterized as being the unique matrix which
satisfies 14.31. This is the content of the following Theorem. The conditions are sometimes called the
Penrose conditions.
Theorem 14.16.4Let A be an m × n matrix. Then a matrix A_{0}, is the Moore Penrose inverse of A ifand only if A_{0}satisfies
AA0A = A,A0AA0 = A0, A0A and AA0 are Hermitian. (14.32)
(14.32)
Proof: From the above lemma, the Moore Penrose inverse satisfies 14.32. Suppose then that A_{0} satisfies
14.32. It is necessary to verify that A_{0} = A^{+}. Recall that from the singular value decomposition, there exist
unitary matrices, U and V such that
( )
U ∗AV = Σ ≡ σ 0 ,A = U ΣV ∗.
0 0
Recall that
( )
+ σ− 1 0 ∗
A = V 0 0 U
Let
( )
P Q
A0 = V U ∗ (14.33)
R S
(14.33)
where P is r × r, the same size as the diagonal matrix composed of the singular values on the main
diagonal.
The theorem is significant because there is no mention of eigenvalues or eigenvectors in the
characterization of the Moore Penrose inverse given in 14.32. It also shows immediately that the Moore
Penrose inverse is a generalization of the usual inverse. See Problem 3.