This is a real derivation of regression analysis in matrix form. An internet search for such a derivation brings up purported derivations which are not really derivations. They set up the material for a derivation and then simply give the final result.

Let Y be an n-dimensional row vector, a 1 by n matrix. This is the data for the dependent variable. The data for the m explanatory variables is given as an m by n matrix. Let B be an m dimensional row vector, a 1 by m matrix.

The deviations D between the dependent variables and the estimates based upon the coefficients B are given by

D = Y − BX

The sum S of the squared deviations can be expressed as

S(B) = D·D^T

The First Order Conditions

The best B is the one that minimizes S. They are called the Least Squares estimates. Here is how they are derived.

S(B) = (Y − BX)(Y − BX)^T = (Y − BX)(Y^T − X^TB)
= YY^T −YX^TB^T − BXY^T + BXX^TB^T

Now consider S(B+ΔB) which is

YY^T −YX^T(B+ΔB)^T − (B+ΔB)XY^T + (B+ΔB)XX^T(B+ΔB)^T

In the expression for ΔS=[S(B+ΔB) − S(B)] all of the terms not involving ΔB are eliminated. That leaves

ΔS = −YX^TΔB^T − ΔBXY^T + ΔBXX^TB^T + BXX^TΔB^T + ΔBXX^TΔB^T

Now let ΔB go to an infinitesimal dB. This eliminates the term ΔBXX^TΔB^T which would involve the products of infinitesimals. Thus

dS = −YX^TdB^T − dBXY^T + dBXX^TB^T + BXX^TdB^T
which can be rearranged to
dS = (BXX^T−YX^T)dB^T + dB(XX^TB^T− XY^T)

The term dB(XX^TB^T− XY^T) is just the transpose of (BXX^T−YX^T)dB^T and both are 1x1 matrices; i.e., scalars.

Thus

dS = 2dB(XX^TB^T− XY^T) = 2(BXX^T−YX^T)dB^T

This means that

(∂S/∂B) = 2(XX^TB^T− XY^T) = 2(BXX^T−YX^T)

For an extremum (∂S/∂B) must be the zero row vector 0 and hence

BXX^T = YX^T
and thus
B = YX^T(XX^T)⁻¹

Second Order Conditions

The second order conditions for an unconstained minimum are that the matrix of the second derivatives of the parameters is positive definite. Let the second derivatives of the regression coefficients B be denoted as

(∂²S/∂B²)

Since

(∂S/∂B) = 2(BXX^T−YX^T)
this means that
d(∂S/∂B) = 2dB(XX^T)
and hence
(∂²S/∂B²) = (XX^T)

The matrix (XX^T) by the nature of its construction is positive definite. Therefore the regression estimates

B = YX^T(XX^T)⁻¹

minimize the sum of the squared deviations between the actual dependent variables and their regression estimates.

Conclusions

The Least Squares estimates of the regression coefficients for the depemdent variables in terms of the independent explanatory variables X are given by

B = YX^T(XX^T)⁻¹