Matrix calculus often appears in discussions around machine learning due to its role in differentiating functions with vector or matrix inputs. However, this approach requires learning numerous complex rules, especially when dealing with row versus column vectors or higher-dimensional tensors. Instead of relying on matrix calculus, a straightforward alternative exists: perform differentiation by writing out the formulas element by element and applying ordinary differentiation.
This technique, rooted in how physicists handled tensors nearly a century ago, involves expressing matrix operations in terms of their individual components. By doing so, differentiation reduces to familiar single-variable calculus, which is generally simpler and more reliable.
For example, consider a neuron activation function defined as the elementwise maximum of zero and a linear transformation plus bias. Writing out the matrix multiplication explicitly and differentiating each element separately allows one to compute derivatives with respect to inputs, weights, or biases by setting the relevant derivative terms to one and others to zero. This approach yields clear, interpretable results without the need for specialized matrix calculus rules.
Compared to matrix calculus, which can be cumbersome and less intuitive, this index-based method works universally and often more efficiently. It avoids the pitfalls of memorizing various derivative rules for different matrix forms and provides a practical tool for those working with neural networks or other vector-valued functions.