Some Ponderings On Gradients
NOTE: THIS POST USES d AS PARTIAL DERIVATIVE NOTATION, NOT NORMAL DERIVATIVE. ALSO NOTE THAT PERFECTLY SQUARE GOT NOTHING TO DO WITH PERFECT SQUARES, ASIDE FROM MAYBE THAT THEY'RE SQUARES.
So I can't lie, despite having taken the course before in HS, I'm not doing the hottest in multi right now (probably due to more rigorous and advanced material + stacking other hard courses + lack of sleep + lack of time effectively studying), so in an attempt to "lock in" during this last half of the semester, I've found myself delving into a concept I've found particularly interesting yet unstudied very much by me: gradients.
Back in junior year of senior high, I remember we were told this: Given a surface in R^3, defined via f(x, y) = [some expression in terms of x and y], we have: gradient = (partial derivative of f w.r.t. x, partial derivative of f w.r.t. y), which also corresponds to the direction of greatest increase, with magnitude being the slope of such direction.
Which at the time seemed absolutely bonkers to me - how could you just claim such thing, derived simply from partial derivatives of two arbitrary orthonormal vectors in R^2, was of "greatest increase." Just seemed pretty unintuitive. But as I learned and pondered, I soon came to a conclusion - the crux of the intuition came from the tangent plane approximation to a given POINT on the plane.
Due to time constraints, that's about as far as I got back in HS. But now that it's winter break, I found time to revisit this interesting problem. So to my delight, I think I've made some good additional progress on really understanding the nature of gradients.
Let's start simple, back to the drawing board simple. Imagine an arbitrary surface, given by the problem setting before.
Then what the gradient vector is actually describing is "go df/dx units in the x direction, then go df/dy units in the y direction, and that is the gradient vector"; this follows simply from splitting components.
Let's assume we slapped a TANGENT PLANE onto the surface at that point. Then we're really running: By going df/dx units in the x direction, on this tangent plane, you're essentially running df/dx units on slope df/dx, which quite beautifully represents a (df/dx)^2 unit change in the z-axis. Why do I say it's beautiful? Because we're in R^3, (df/dx)^2 is GUARANTEED to always be nonnegative. And since we're on a tangent plane here, the opposite direction is thus GUARANTEED nonpositive. This is REALLY REALLY NEAT, because what this is telling us is that the x-component step we take via the gradient vector will always be the (non-negative) 1-dimensional-cross-section direction of max ascent (i.e., max ascent along that particular direction, which in this case is along-1-dimension unit x vector direction). Very cool.
So now I wanna focus on that (df/dx)^2 change in z here. This is actually super super neat as well. Think about it: you step df/dx units in this direction, and end up at that perfectly squared magnitude LOWER than what the plan ends up having its height (or z) at. This is REALLY REALLY COOL. If you just added df/dx + some positive epsilon value units instead, you wouldn't get the perfectly square effect, and you'd instead get like (df/dx + some epsilon value) * (df/dx) change (derived from the slope, i.e. partial derivative in x direction, being df/dx). Not as cool.
So now, one might throw in the towel, say "ok so if the two components are guaranteed to be non-negative, i.e. (nonstrictly) ascending, then the combined component vector would intuitively make sense to also be ascending, and I think we can basically make the intuitive jump here that this component mixing with correct magnitude proportions is gonna give you the direction of max ascent" and sorta just hand-wave the conclusion from here. Maybe even add in a touch of loose limiting behavior justification "if the tangent plane slope limits to infinity in the, say, x direction, then the steps in that direction for the gradient will also limit to infinity, and fixing y in place that makes the direction of the actual gradient limit to literally just the x axis direction in the direction of its ascent, and since we're limiting it to infinity, makes sense to have it as the direction of max ascent" (but note here, any arbitrary strict monotonically increasing relationship between slope and steps taken works here in this justification; the limiting behavior justification is too general to fit our specific slope and perfectly squared slope terms in this case). But of course, none of this hand wavy stuff is gonna get you a formal grounded proof, and is much less gonna give me some satisfaction.
So then we could just brute force derive it using calculus from here, yknow the typical setting up diffeqs and/or algebraically solving for zeros, but again, that doesn't seem very satisfying. I want a cool, geometric intuition to outline this proof, and I believe it is possible.
And here's a big part where the tangent plane helps: partial derivatives on a tangent plane are POSITION INVARIANT. You can visualize this intuitively by placing a point down on the plane, memorizing that state, then moving the point whatever way you want, and memorizing that state. Then it becomes clear: the two memorized states, aside from potentially being at differing positions, are COMPLETELY THE SAME. The same visualized information, including the same partial derivatives in EVERY DIRECTION, INCLUDING THE COMPONENT DIRECTIONS. Queue the guy with the photoshopped galaxy explosion going on in this brain. But ok, seriously, what this means is that component partial derivatives are constant regardless of where you move along the plane.
Now this observation is crucial here, because now we can say the following: since the gradient vector is basically just composed of moving df/dx units steps in the x direction, then df/dy units steps in the y direction (or vice versa, or interchanged, blah blah blah you get the point), you can LINEARLY APPLY THE z ABOVE DIFFERENCE ON THE TWO DIRECTION STEPS. What this means is that:
- First, we move df/dx units steps in the x direction. The tangent plane looms over us at a height difference of (df/dx)^2 units.
- Next, we move df/dx units steps in the y direction. If we just stepped without doing (1), then we'd be stepping from the gradient point, and then you'd have the height difference (df/dy)^2 units. But since we previously established partial derivatives are constant for points on the tangent plane, it DOESN'T ACTUALLY MATTER whether we do (1) or not, or whether we do (1) after (2) or blah blah blah ORDER DON'T MATTER AND THATS REALLY REALLY COOL YAY, SINCE WE WILL ALWAYS STILL GET THAT (df/dy)^2 units height difference. supa cool. So thus, by linearizing this height difference "function", we get that the gradient vector started at the particular arbitrary point itself lies (df/dy)^2 + (df/dy)^2 below the tangent plane (recall height in this case is z axis). Right, it took that much work just to get that plus sign in there. Whoo!
(Editor's (Me) Note: This linearly applied z above height sorta summing strategy really comes from the nature of (how we define vectors and how they can be combined) - in this specific instance, we call this vector addition (wow novel term, ikik)), and the pretty intuitive properties it involves, and so the entire thing is pretty intuitive, and easily geometrically visualizable, so I won't go into it further.
So now it remains to show that (df/dx)^2 + (df/dy)^2 is the biggest height difference you can get for any vector with length of that gradient vector, which we established was (df/dx, df/dy), and thus has length sqrt(df/dx squared + df/dy squared). And for this we can invoke yet another (but when you think about it, really just the same as the last one just worded in another perspective) tangent plane property: NORMALIZATION WITHOUT CHANGING THE ACTUAL SLOPE YAY.
So basically we just normalize the length down to the unit circle. And thus we conclude that we want to show that on this tangent plane, the biggest slope you're gonna get at the point in any R^2 direction is going to be sqrt((df/dx)^2 + (df/dy)^2) (we just normalized the vector we originally had), aka a unit step will give sqrt((df/dx)^2 + (df/dy)^2) units up. So how do we know that this is the max we're gonna get? Well, as always, we could set up equations and algebraically/calculusaically solve, but I'm looking for something more physically and geometrically intuitive.
Its also important to note that the fact that we chose x and y axis to form the basis is technically arbitrary here. Why? I mean I could rerun you down the proof and show you how there's no such basis dependencies, but I feel like it's already intuitive, and it would be a LOT of work otherwise. And besides, that's what I'm trying to chase anyways - intuition. But anyways, with this in mind, we could've chosen technically any R^2 orthogonal basis and said the gradient vector is formed via the perfectly squared stepping component-ization, and it should STILL HOLD THAT IT'S DIRECTION OF MAX ASCENT. Crazy, I know, but my intuition tells me it's simply due to the nature of any arbitrary tangent plane to be able to simply be rotated about an arbitrary axis (including any directly vertical axis in this R^3 hypothetical WE're imagining), and still remain an arbitrary ol' tangent plane. Very cool.
OK so I'm gonna go on a sorta tangent and introduce yet another object to play with: the properties of DIRECTIONAL DERIVATIVES.
You see, given our arbitrary 2 orthogonal basis vectors, which in this case are simply unit x and y, we can break down any unit vector into a linear combination of components of these orthogonal basis vectors. Applying the same component step formula (follows from that position invariance trick from before), we get that any point lies df/dx * comp_x + df/dy * comp_y below the z (if it's negative, then it's above) val of the tangent plane. Now as we just established, moving unit gradient direction will give sqrt((df/dx)^2 + (df/dy)^2) below the z. So then it remains that we should show that given bounds sqrt(comp_x^2 + comp_y^2) = 1, which since we're in R^2, biimplies comp_x^2 + comp_y^2 = 1, we should find such argmax of df/dx * comp_x + df/dy * comp_y, which should, if our proof holds, have comp_x = df/dx and comp_y = df/dy. (Note, any unit vector in any arbitrary direction at that point has the magnitude below the z as the directional derivative value, which explains why the directional derivative formula has you dot product the gradient, as it is composed of (df/dx, df/dy), and unit vector of the particular direction).
So now it becomes yet again evermore tempting to just algebraically/calculaically solve, as it is literally just framed perfectly as a boundary problem now, but NO. WE MUST GO ON. WE MUST FIND OUR GEOMETRIC AND PHYSICAL INTUITION.
So yes, done with directional derivatives exposition, now we're back to the main proof: so this is a pretty general thing we're tryna prove here - any orthogonal basis works after all... Surely, there isn't already a proof for it... or is there??? ENTER CAUCHY-SCHWARZ!!!!
Cauchy-Schwarz claims that the dot product of any two vectors will always be less than the product of their individual lengths.
Thus, by applying it, we get that the dot product of any unit vector and the gradient vector (aka the directional derivative of that unit vector) will always be less than or equal to the length the unit vector (1 unit) times the length of the gradient vector, which using Euclidean distance we find to be sqrt((df/dx)^2 + (df/dy)^2), and 1 times that will just equal sqrt((df/dx)^2 + (df/dy)^2). Thus, we've established an UPPER BOUND TO THE DIRECTIONAL DERIVATIVE MAGNITUDE, AND THAT EXACT UPPER BOUND HAPPENS TO BE THE EXACT DIRECTIONAL DERIVATIVE WHEN YOU STEP A UNIT DISTANCE IN THE DIRECTION OF THE GRADIENT. SO BOOM, WE'VE PROVEN IT MAXXES THE ASCENT YIPPEEEEEEEEEE!!!!!
The end. The direction of steepest increase, in unit vector form, is grad f / Euclidean norm of grad f. Overall, the only thing you really need to reap from the Cauchy-Schwarz inequality is the fact that the unit vector directional derivative is maximized when in parallel with the actual gradient, which comes from the fact that dot product (which in this case is just the directional derivative in the unit vectors direction, which we're trying to maximize to get that max ascent) is maximized when the two vectors are parallel. Matter of fact, the cauchy schwarz inequality actually turns into EQUALITY when the vectors are parallel, so you don't JUST get a maximization guarantee, although the maximization guarantee follows from this equality since the inequality represents an upper bound and thus equality sets the constraints to the upper bound, as shown above, just now we know it applies in general to two vectors. Cool proof.
(Note the directional derivative intuition proof thingy is pretty shaky - expect future blog posts on it, but the proof is still defo very much there).
So we done now? NAH, we still have to prove Cauchy Schwarz, PLUS that general equality rule, PLUS an additional thing about gradients when you take not just f, BUT THE GRAPH OF f INSTEAD (hint hint, level sets incoming)... stay tuned for future blog posts! (oh, and also possibly more blog posts on geometric interp from dot prod, directional derivs, and interpretations of the length of the gradient, since this current blog post covers mainly just the interpretation(s) on direction of gradient).
Actually, on second thought, here's a nice proof of Cauchy-Schwarz: https://www.ethanepperly.com/index.php/2024/12/10/my-favorite-proof-of-the-cauchy-schwarz-inequality/.
I worked out basically the same proof in matrix alg last semester, cool to see how it has geometric applications despite having this algebraic proof - should be a pretty fun proof to reproduce in a blog post too... ;).
P.S. I later on realized you don't even need Cauchy-Schwarz - you can use position invariant property of the partial derivatives on the tangent plane + geometric interp of dot product + some algebra to prove the exact same thing we used Cauchy-Schwarz to prove instead. I'll leave this Cauchy-Schwarz replacement as an exercise to the reader, but I think it should be pretty intuitive when you roughly visualize it.
Hint hint: When I said "By geometric interp of dot product," I mean for you to think about the fact that the largest component is made when u take the directions of both vectors to be the same.
Note that this proof has opened up some conjectures that would also be cool to prove: Given any continuous and differentiable function (could be a surface, could not) that goes from R^m to R^n for some arbitrary m <= n, we have that if the m orthogonal vectors in R^(n + m) of an arbitrary point input of the function are pointed in the max ascent along that particular dimensional “slice”, and theyre magnitude matches the partial derivative of that slice at that point, adding them vectors all together “combines the fastness of the slices” to make the overall gradient vector, i.e. the vector of greatest ascent.