Why the gradient is the direction of steepest ascent 

I was still struggling with the intuition of this and I think I have come up with another simple way to conceptualize the gradient and the characteristic of steepest ascent. Start by remembering that the gradient is composed of the partial derivatives of the function in question. If you think about each partial derivative as just a simple rise-over-run problem, then you can see clearly that each partial derivative is going to give you the amount of change to the output (rise) as the input (run) is increased. Let's consider Grant's 3-dimensional example, so we can say that inputs for the multivariable problem are x and y and the output is z. Because the slopes vary based on location on the x-y grid, we need to pick a starting point on the grid. Let's just say (1,1). It doesn't matter. Now let's look at the x-z 2-dimensional problem first. Let's say the partial derivative tells us that at point (1,1) for each 1 unit increase of x, z is increased by 4 units (i.e. the derivative = 4x). Since x can move in only 1 dimension, the only choice of direction we have is whether the change in x is positive or negative. Obviously, if we move x by -1, then z will decrease by 4 units. So, if we need to choose in which direction we move x to increase z, we know that it is in the positive direction. If we decrease x, z will decrease as well. Now, do the same thing for y and z and let's say that the partial derivative for y at (1,1) is 3y. This means that a 1 unit increase in y will result in a 3 unit increase in z. Again, if you want to increase z by moving y, increase y, don't decrease it. Now, let's put the two variables together. We now have a choice of directions. It's no longer sufficient to say that we need to increase x and we need to increase y. (Though that is half the battle). We need to also decide the relative value of increasing x versus increasing y. When we choose a direction we are making a tradeoff between the relative movements in each of the basis (x, y) directions. Let's say that we are allowed to move in any direction by 5 units (if you haven't noticed yet, I like to keep my Pythagorean theorem problems simple!). The question is - "in what direction can I move 5 units to maximize the increase in z?" This would correspond to the direction of steepest ascent. So, let's say we use all of our 5 units in the x direction. This corresponds to the vector [5,0]. Since the increase in z is 4x + 3y, the total increase in z will be 20 (5∙4) . If, on the other hand, we use all of our 5 units in the y direction [0, 5], the total increase in z will be 15 (5∙3). But the beauty of using vector geometry is that we can use our 5 units in a direction that will give us effectively a movement in the x direction of 4 and in the y direction of 3. We get 7 units of movement for the price of 5! Of course, that's the hypotenuse of our 4 by 3 right triangle. So, by following the vector [4, 3], z is increased by 4∙4 from the x direction movement and by 3∙3 from the y direction movement for a total increase of 25! I think you can see that any other direction will produce a smaller increase. And, of course, the vector [4, 3] is exactly our gradient! It's interesting to also think about when one of the components of the gradient is negative. Let's imagine that instead of 3y, our partial y derivative is -3y. This means that a positive movement in y produces a decrease in z. Remembering that we can only move y in 1 dimension, if we move y downward then of course z will increase. So you can see that the gradient vector [4, -3] will produce the same 25 unit increase in z for a 5 unit move IF we move y in the negative (downward) direction. Just follow the vector! Of course this is calculus, so these 3, 4, and 5 unit moves are very, very small. 😊

the gradient is not just a vector, it's a vector that loves to be dotted with other things. /best

Beautiful! Didn't think of it that way. Thanks a lot!

You can't climb a mountain f(x,y) in the fastest way possible by moving just in the x- or the y-direction (using partial derivatives). You most of the time have to go in a direction that's a combination of the x- and the y-direction (using directional derivatives)!

Thank you so much Grant! Simplicity is the most difficult thing to acheive.

For anyone confused, This is how I see it, if it helps at all: If you’re at a point (a, b) then the directional derivative at that point looking in the direction v (where ||v||=1), is given by ∇f(a, b) • v. When we say,”What is the direction of steepest ascent?”, what we are really asking is,”In what direction do I move in to produce the largest directional derivative?” In other words, we want to maximise ∇f(a, b) • v. Given that we are at a single point, we can then say that ∇f(a, b) is a constant since it is evaluated by only using the particular values a and b. v is the only variable here and so the only way to maximise ∇f(a, b) • v is to alter the vector v. We know that the dot product of 2 vectors is maximised when they are pointing in the same direction as each other (see proof of this at bottom). Using this and the fact that we are not varying ∇f(a, b), we can conclude that to maximise the directional derivative (given by ∇f(a, b) • v) we must vary the vector v so that it points in the same direction as ∇f(a, b). And that’s it - we’ve shown how when the vector v, is in the same direction as the gradient function ∇f(a, b), it’s output (the directional derivative) is maximised. Saying that v is in the same direction as ∇f(a, b) is to say that v = (1/k) × ∇f(a, b) where k is the magnitude of ∇f(a, b). This is because v is a unit vector as previously stated. These 2 vectors (v and ∇f(a, b)) are pointing in the same direction so the conclusion can be drawn that ∇f(a, b) also points in the direction of steepest ascent Proof to maximise dot product: Consider two vectors a and b. The angle between a and b is given by cosθ = (a • b) / (|a| × |b|) We can rearrange to say that a • b = |a||b|cosθ. To maximise the left hand side, which is what we want to prove, we must maximise cosθ. This achieves its maximum value at 1, which occurs when θ = 0. When θ = 0, we can visualise this by saying the vectors are parallel and overlapping each other. So we can conclude that a•b is maximised when a is parallel to, and overlapping b, Vice Versa.

My takeaway from this is to try reduce it to one dimension to understand what each element is doing to increase the "steepness" of a gradient. Say *f(x)=-x^3* then *df/dy=-3x^2*. Since the constant of this example derivative is negative, that means *x* needs to move in the negative direction of the number line to increase the output of *f(x)*. After you know what direction of the number line to go towards for each element, the magnitude you move for each element is proportional to the size of the element compared to the whole gradient vector. Anyways thanks, great explanation Grant :)

Learning this concept was no less than a sense of accomplishment itself! Grant is Grand! Cheers!

Lucky that I found these intuitive explanations.. it truly feels great when you understand what's actually going on when we use a formula

Thanks! Here is how I would very concisely explain it: The problem of finding the direction of steepest ascent is exactly the problem of maximizing the directional derivative. The directional derivative is a dot product. When you are trying to maximize a dot product, choose the direction to make it parallel to the other vector, Since in this case, the other vector is given to be gradient(f)(v), THAT IS the direction of steepest ascent. For me, the basic intuition comes from the dot product. The part that is not so obvious to me is that gradient(f) dot v is the "right" formula for the (definition of) the directional derivative.

I'd like to add my two cents on this, as I couldn't relate some things at the beginning but after some reflection, I figured them out: 1. The gradient is the direction of the steepest ascent because the gradient encompasses all of the possible d (change) for the function. This is how: - The ubiquitous 1 dimension input derivative - the ordinary regular derivative - means the change of the function (in its entirety that is, the maximum possible change); - The gradient "owns" the derivatives on all possible "angles". It suffices to have snapshots of the d only from all orthogonal directions (2 in our case). As a note, the dot product is known for measuring how much of a vector, another vector is (roughly speaking). So dotting the vector of the directional derivative with the gradient is merely how much of the gradient - the entire change - the said vector is. And of course, to get a maximum you want to dot two parallel vectors. 2. Question: is the gradient with partials of x and y, the only possible vector to have the steepest ascent direction? Well I thought why not make one by taking any 2 orthogonal vectors (on xy plane) and get the directional derivatives of these. The two resulted derivatives can be the components of another gradient. I feel I'm missing something here but I'll get to its bottom.

This helps so much, thank you!

Beautiful and elucidating

It's like my mind just got illuminated!! I've always underestimated the power of the Del operator. Not only is this operator showing you the slope of a scalar field in the direction of a vector, it also points at the direction of the unit vector with the max gradient, and also tells you the size of the slope of this unit vector with a max value. It's crazy how this new revelation changes your understanding of vector calculus. Thanks a lot 🙏🏽🙏🏽🙏🏽🙏🏽🙏🏽

This is so easy to follow! I love it.

Thanks a lot Grant! It was a great pleasure to see you here. I am a big fan of 3B1B.

Since I´ve heard the voice of 3B1B I knowed that this is going to be a great video

Amazing video. Matchs perfectly with your Linear Algebra series.

I always wonder how Grant have developed such a great understanding of maths, he does magic with maths. !!!

Thanks man . That was the best explaination ever . Simple and sweet . I was very confused but you saved me :) Thanks

Great video helps a lot thanks :)

that last explanation kinda blew my mind a bit, nice!

Excellent exposition ... thanks a lot !

The best explanation of a gradient on Internet!

@steve manus, I like the way you broke this concept down almost like a problem of Lagrange multiplier, where we are asked to find the optimal value of some function f(x,y) subject to the constraint of another function g(x,y) in two dimensions. Of course as you may know already or expect, the concept of the gradient is incorporated into the solution. It is typically the scenario that involves the balance between the independent variables so as to produce the maximum output for the function of such variables. Usually, the optimal value lies in between the extreme choices for the variables. Extreme X or extreme Y choices, as you noted, didn’t produce the maximum output for f(x,y). I was hoping that the video maker would stay away from the concept of directional derivative to explain the geometrical meaning of the gradient. In fact, I liked the mapping to a straight line explanation that he started with at the start of the video. I wished that he finished that up.

i liked it. I would request put some lecturers on riemann stieltjes integral and the difference between it and Riemann integral

most critical video in understanding the gradient's relation to the steepest ascent.

Derivation of del in direcion of del that is equal to magnitude of del gives a lot of intuition. Nice!

so good!

Honestly, thank you

Great video. My only quibble is with the notation for the directional derivative. You have ∇_v f. I have seen the directional derivative written as D_v f, and ∂f/∂v, which seem to make sense. But the use of ∇_v f seems non standard and a bit confusing. How do we interpret ∇_v f? The "gradient in the direction of unit vector v" does not make sense, since the gradient is independent of v and is fixed for all intents and purposes.

Think about the gradient in one dimension. It is the biggest roc since its the only roc in one direction. Think about the gradient in two dimensions. It combines the greatest roc if you could only go in the y and the greatest roc if you could only go in the x. To create a greatest roc overall.

SIMPLY AWESOME.

So if I'm understanding this correctly, the argument he makes after he draws in the gradient line is that the vector dotted with the gradient that gives the max value for the gradient is the vector that is parallel to the gradient itself. But doesn't this argument only work if we already take it as true that the gradient *is* actually already in the direction of max increase, so that a vector parallel to it is also in the direction of max increase? I still don't get why the gradient definition inherently points in the direction of max increase??

I’m still processing everything (I’m not going to ace an exam any time soon) but I’m excited that I’ve been able to follow this logic. Thank you!

Thank you

ThankYou!

How to extend this conclusion to complex space?

My question is if the function is differentiable shouldn't the change in function from all the direction should be same like when we do for complex analysis.

thank you very much for a video!!! :D cheers from Ukraine

Hello, can you please tell what is the programming material do you use to illustrate the functions?

Marvellous💯

Thanks

Why isn't this linked to the video on the website?

Thanks grant ❤

When Geant first told us about the gradient giving the steepest ascent, I instantly imagined a graph where you have +ve partial derivatives in x and y directions, but a -ve one in between them (i.e. vector 1,1 etc). This would make the gradient vector not be the steepest ascent, rather the pure x or y direction (whichever is maximum slope). But after this I realised there must be a concept of multivariable differentiablity because in this case there would be a sharp point at that location!

For me the easiest way is to think of a basic function f : R -> R The derivative of f at a point a tells you which direction to walk (left or right on the y axis) for the steepest ascent. This is the same thing for 2 dimensions

Thank you. It's been bothering me for a long time

You just explained the Cauchy-Schwarz equation in the end, didn't you?

Beauty man ! Beauty.

Best video!

I'm not nearly as advanced as you guys but I'm a little bit unsure about the logic here. If we let nabla_f= [a,b], then, there exists another vector "-nabla_f"= -[a,b] (direction of fastest descent) such that dot(-nabla_f,-nabla_f) = dot(nabla_f,nabla_f) = max(nabla_f,V), even though "-nabla_f" points in the exact opposite direction. Would it be possible that the condition "dot(nabla_f,nabla_f) = max(nabla_f,V)" is a necessary but NOT sufficient condition to prove that nabla_f is the direction of fastest descent?

Thanks much, but i am still not clear why gradient direction can let the function f have the steepest change. How gradient relates to the steepest output change of function f? Thanks very much.

Great

i am wondering if a vector V is doted with any vector A other than gradient vector , it still give the max value if V is parallel to A . so still the video doesn't prove that gradient is the steepest ascent ,,correct me if i am wrong

could you provide us a video explaining why a dot b = a x b * cos(a,b)? I understand it for geometric vectors, but it's unclear for me how this can be scaled to n-dimensional vectors.

I don't understand why the directional derivative gives the slope. Firstly, if I have the slope of a function = df/dx, then I can only multiply it with dx if I expect the resulting change df to match the function graph. Otherwise it matches only the slope line. For directional gradients, you mentioned the vector length should be 1 instead of infinitely small. That means that a gradient component df/dx multiplied with the corresponding directional vector x-component will result in a change that will align with the slope line, but not with the graph. Can somebody cast light into this issue?

I'm still confused about why it is. I can see from the dot product formula that since the Directional Derivative is greatest and a positive number(due to taking absolute value of the gradient) that the gradient must represent the steepest ascent. I'm having trouble imagining the ascent part. Let's say the partial derivatives in respect to both x and y were both a negative value at a point. Wouldn't the Directional Derivative end up as a positive value and be referencing ascent still?

Wish you were my Prof.!!!!!

at 2:10 i thought the same, the combination of derivatives gives you the steepest ascent and not the steepest descent. f(x,y)= x (x-1) = x^2 -x , this function has the min at x= 0.5 and max at infinity del f = ( 2x - 1 ) i + 0 j when x= -1 del f = -3 i , this is correct. -3 direction will lead you the max when x= 1 del f = 1 i, this is correct. +1 direction will lead you the max when x =0.5 del f=0 , does this fail?, because it gives no directions, it doesnt know if this is a max or min.

Can anyone explain to me at 6:55, if the projected vector could also have a value of larger than 1? I mean it depends on the direction of projection too right?

To those who are confused, the direction of the steepest descent is the direction in which the directional derivative is maximum. Directional derivative for any vector v = Gradient * vector v So we maximise (gradient * vector v) to maximize directional derivative

Why is the magnitude of the gradient vector said to be the RATE of maximum ascent? When I see "rate", I think slope. Why isn't the rate of ascent simply the partial of y divided by the partial of x.? Isn't this the slope of the gradient....i.e. change in y over the change in x? What am I missing? thanks

let x^2 + y^2 + z^2 = 1, so that z = f(x,y) z = ( 1 - x ^2 + y ^2 ) ^ (1/2) z < 0 = - ( 1 - x ^2 + y ^2 ) ^ (1/2) z > 0 for z=0, use anything to make it continuous 1. prove the max value of gradient at (x,y,z) = (0,0,1) when the initial point is from (x,y,z) = (0,0,-1) 2. find the gradient at (x,y,z) = (1, 0, 0) from the problem1 , when the gradient becomes infinity if you will, change my questions to make it happen gradiently possible to do gradient on a closed surface?

Slope =/= Gradient There can be only one gradient (vector) that's mapped by a given point (a,b). So, the gradient is the same regardless of the applied vector at a given point. However, there can be multiple slopes (scalar) at a given point. The slope depends on the applied vector. We can slice the graph with a plane in the same direction of the applied vector, and we can do this in infinite ways, all resulting in a different slope value. Think of it as climbing a hill sideways (arbitrary applied vector) instead of directly up (following the gradient).

Why length is less than 1 at 5:50 ?

Thank you!

Correct me if I'm wrong, but the "proof" here seems to be a circular argument. Consider this: 1) The directional derivative could also be = (another vector that is NOT the gradient) dot (direction vector), isn't it? 2) Then with the argument presented here, wouldn't MAX(direction derivative) = (another vector) * (direction vector)? So the question remains: how do we know that projecting along the "gradient vector" gives a larger value than projecting along "another non-gradient vector"?

I'm having trouble reconciling the two views of f(x,y). On one hand, you show them as mapping a 2D space to a 1D number line. On the other hand, you show them as mapping a 2D space to a 3D space, as when you show a 3D graph. But, it is not really 3D, is it? The height, f, is just an interpretation of the dependent variable, correct? You could show a 2D graph with color instead of height, right? To me, that makes the gradient easier to understand why it's on a plane below a 3D shape. Thanks.

TLDW: The directional derivative represents the rate of change of the function in that direction. If you try all possible directions centered at that point, it happens that the magnitude of the directional derivative is largest when taken in the direction of the gradient. Therefore, we can conclude that the gradient points in the direction of steepest ascent.

I never faced such a difficulty to understand something in maths like this 😂

Thank you 3blue1brown

How can the gradient which is a vector dotted with the vector v equal the normalized version of the gradient vector???

It seems like we are saying that the gradient is not always the direction of steepest ascent.? What if f=-(x^2)-(y^2)?

I'm 5 videos in and my brain is on fire

I think it's hilarious how when Grant does videos for KA he repeats out loud what he's writing on the screen like Sal does. Makes me laugh every time

Does another change w

Is this grant from 3b1b?

The way I've always seen it is every sum of derivatives will be a combo of the partials. Therefore, the purest and least inefficient path is the least scaled up linear combination of the bases so path with least or minimized resistance and drag is the combo of just the two partial derivatives or the gradient vector. I'm pretty sure you could prove this with the triangle inequality. Any sum of multiples of the bases vectors will have a longer hypotenuse than the sum of just the partial derivatives holding one side, like the height, constant on both. In other words, you'll waste energy traveling farther than necessary horizontally for the same movement vertically compared to the path of the pure partial derivatives. But you can't move faster than those, because you're limited by the physical shape of the surface you're on. You have no other choice, the constraints knock down other paths physically or hypothetically in the case of imagined scenarios.

Help!!! I think this guy just claimed that the direction of maximun change is in the direction of the gradient because it is in the direction of the gradient further confusing me.

i think for steepest descent you need to put "-" , which come from cos theta , where theta = pi, gives the minimum for the objective function. cmiiw

ts not frix or not, can telx anyx by anyx nmw. no such thing as howx telx

kekek you are a god

it is rather consequence than reason

Yeah but why does the gradient point in the direction of steepest ascent? @ 5:18

Kindly tell me that .... As we know that... Gradient is (n-1)d as compared to scalar function of (n)d. .. Keeping in mind this thing..... Does gradient at a point means A vector of global maximum locator in dimension less than one to scalar function.....?????? For example.... If phi=3d func. Then Obviously ... Del phi= 2d vector at a point perp. To level surface... Then Does del means a vector in 2d that locates maximum value of phi???????

Terribly sorry to Sal, but I'm just too fond of Grant's voice to watch any of Sal's videos.

Is this video part of some course?

I don't think the narrator ever really explained why the Gradient vector is ALWAYS in the direction of the steepest slope. As far as I could tell, he only explained how the directional unit vector interacts with the Gradient to reduce it or maintain it at its maximum. If every point on a 3D-surface has an infinite number of tangent lines, all with potentially different slopes, how can taking partial derivatives from just 2 directions (x and y) and combining them into a vector always point in the direction of maximum steepness?

I find this fact so confusing. If the gradient is the direction of steepest ascent, what is the direction of greatest net change? I always assumed the gradient points in the direction where the function changes the most.

Wow

I understand why the gradient has the steepest slope of all the directional derivatives, but why can't it be in the direction of steepest DESCENT? Shouldn't there be a case where a pure step in x + a pure step in y lowers the value of the function?

7:40 isn't he suppose to say "multiplying it" rather than "dividing"?

if possible real analysis

What if I want to find the least steepest??

at 7:41, you are dividing it down by "2" and not half, actually you get half of it.

I was initially confused because I was thinking of a situation where: along x axis and y axis the graph is kinda stable and mildly changing but in quadrant one there is a freaking big valley and ma poor little point is at the origin point😂 I was worried that no info about that valley is shown in gradient and ma point don't know where to go😂 then i realize i was outside the scope of this discussion

👌👌👌👌👌👌

As a hiker I don't want the steepest ascent!

I didn't understand this one.