The particular solution of the least squares problem given in 12.18 is important enough that it
motivates the following definition.
Definition 12.12.1Let A be an m×n matrix. Then the Moore Penroseinverse of A, denoted byA^{+}is defined as
( )
A+ ≡ V σ−1 0 U ∗.
0 0
Here
( )
∗ σ 0
U AV = 0 0
as above.
Thus A^{+}y is a solution to the minimization problem to find x which minimizes
Ax − y 
. In
fact, one can say more about this. In the following picture M_{y} denotes the set of least squares
solutions x such that A^{∗}Ax = A^{∗}y.
A+ (y)

My 
x ∗
  ker(A A)




Then A^{+}
(y)
is as given in the picture.
Proposition 12.12.2A^{+}y is the solution to theproblem of minimizing
Ax − y 
for all x whichhas smallest norm. Thus
 
AA+y − y ≤ Ax − y for all x
and if x_{1}satisfies
Ax − y
1
≤
Ax − y
for allx,then
A+y 
≤
x 
1
.
Proof:Consider x satisfying 12.17, equivalently A^{∗}Ax =A^{∗}y,
( ) ( )
σ2 0 V ∗x = σ 0 U ∗y
0 0 0 0
which has smallest norm. This is equivalent to making
V∗x 
as small as possible because V^{∗} is
unitary and so it preserves norms. For z a vector, denote by
(z)
_{k} the vector in F^{k} which consists
of the first k entries of z. Then if x is a solution to 12.17
( ) ( )
σ2(V∗x)k σ(U∗y)k
0 = 0
and so
(V ∗x)
_{k} = σ^{−1}
(U∗y)
_{k}. Thus the first k entries of V^{∗}x are determined. In order
to make
V∗x
as small as possible, the remaining n − k entries should equal zero.
Therefore,
( ) ( ) ( )
∗ (V ∗x)k σ−1(U ∗y )k σ−1 0 ∗
V x = 0 = 0 = 0 0 U y
and so
( σ− 1 0 )
x = V U ∗y ≡ A+y ■
0 0
Lemma 12.12.3The matrix A^{+}satisfies the following conditions.
+ + + + + +
AA A = A, A AA = A ,A A and AA are Hermitian. (12.19)
(12.19)
Proof:This is routine. Recall
( )
A = U σ 0 V∗
0 0
and
( )
+ σ−1 0 ∗
A = V 0 0 U
so you just plug in and verify it works. ■
A much more interesting observation is that A^{+} is characterized as being the unique matrix
which satisfies 12.19. This is the content of the following Theorem. The conditions are sometimes
called the Penrose conditions.
Theorem 12.12.4Let A be an m×n matrix. Then a matrix A_{0}, is the Moore Penrose inverseof A if and only if A_{0}satisfies
AA0A = A,A0AA0 = A0,A0A and AA0 are Hermitian. (12.20)
(12.20)
Proof: From the above lemma, the Moore Penrose inverse satisfies 12.20. Suppose then that
A_{0} satisfies 12.20. It is necessary to verify that A_{0} = A^{+}. Recall that from the singular value
decomposition, there exist unitary matrices, U and V such that
( )
∗ σ 0 ∗
U AV = Σ ≡ 0 0 ,A = UΣV .
Recall that
( )
+ σ−1 0 ∗
A = V 0 0 U
Let
( P Q )
A0 = V U∗ (12.21)
R S
(12.21)
where P is r × r, the same size as the diagonal matrix composed of the singular values on the
main diagonal.
The theorem is significant because there is no mention of eigenvalues or eigenvectors in the
characterization of the Moore Penrose inverse given in 12.20. It also shows immediately
that the Moore Penrose inverse is a generalization of the usual inverse. See Problem
3.