3. Gradient Descent

1. Rolling Downhill

We cannot see the whole mountain. We can only feel the slope under our feet.

• Slope: The Derivative (Gradient).
• Step: The Learning Rate.

2. The Update Rule

We move against the gradient to go down. $$ w_{new} = w_{old} - (lr * gradient) $$

python
# The essence of generic learning
weights -= learning_rate * weights.grad