Matrix Differentiation Properties

Theorem

Let $x \in R^{n}$ , $A$ be a constant matrix, and $f, g$ be differentiable functions.

Linearity:

\frac{\partial}{\partial x} (a f + b g) = a \frac{\partial f}{\partial x} + b \frac{\partial g}{\partial x}

Product Rule (scalar):

\frac{\partial}{\partial x} (f g) = f \frac{\partial g}{\partial x} + g \frac{\partial f}{\partial x}

\frac{\partial}{\partial x} (a^{T} x) = a

\frac{\partial}{\partial x} (x^{T} A) = A

\frac{\partial}{\partial x} (x^{T} x) = 2 x

\frac{\partial}{\partial x} (x^{T} A x) = (A + A^{T}) x

If $A$ is symmetric:

\frac{\partial}{\partial x} (x^{T} A x) = 2 A x

For $f (g (x))$ :

\frac{\partial f}{\partial x} = \frac{\partial f}{\partial g} \frac{\partial g}{\partial x}

Example 1: Linear form

Let $a = [23]$ and $f (x) = a^{T} x = 2 x_{1} + 3 x_{2}$

\frac{\partial f}{\partial x} = a = [23]

Example 2: Quadratic form

Let $A = [2113]$ and $f (x) = x^{T} A x = 2 x_{1}^{2} + 2 x_{1} x_{2} + 3 x_{2}^{2}$

Since $A$ is symmetric:

\frac{\partial f}{\partial x} = 2 A x = 2 [2113] [x_{1} x_{2}] = [4 x_{1} + 2 x_{2} 2 x_{1} + 6 x_{2}]

Example 3: Chain rule

Let $g (x) = A x$ where $A = [1324]$ , and $f (g) = g^{T} g$

\frac{\partial f}{\partial x} = \frac{\partial f}{\partial g} \frac{\partial g}{\partial x} = (2 g^{T}) (A) = 2 (A x)^{T} A = 2 x^{T} A^{T} A

Evaluating at $x = [11]$ :

A^{T} A \frac{\partial f}{\partial x}_{x = [11]} = [1234] [1324] = [10141420] = 2 [11] [10141420] = 2 [2434]

As a column vector: $[4868]$