Question: what is the computational cost of running a single step of gradient descent and Newton's method? Not including the training set, how much memory is required?

\n",
"\n",
"Suppose that computing gradients of the examples takes $\\Theta(d)$ time and computing Hessians takes $\\Theta(d^2)$ time, and express your answer in terms $n$ and $d$.\n",
"\n",
"Gradient descent: $\n",
" w_{t+1} = w_t - \\alpha_t \\cdot \\nabla f(w_t) = w_t - \\alpha_t \\cdot \\frac{1}{n} \\sum_{i=1}^n \\nabla f_i(w)\n",
"$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"Time: $O(nd)$\n",
"\n",
"Memory: $O(d)$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"Newton's method: $\n",
" w_{t+1} = w_t - \\left( \\nabla^2 f(w_t) \\right)^{-1} \\nabla f(w_t).\n",
"$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"Time: $O(nd^2 + d^3) = O(nd + nd^2 + d^3 + d^2 + d)$\n",
"\n",
"Memory: $O(d^2)$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Why does gradient descent work?\n",
"\n",
"**...and what does it mean for it to work?**\n",
"\n",
"Intuitively, GD decreases the value of the objective at each iteration, as long as the learning rate is small enough and the value of the gradient is nonzero.\n",
"Eventually, this should result in gradient descent _coming close to a point where the gradient is zero_.\n",
"We can prove that gradient descent works under the assumption that the second derivative of the objective is bounded (this is sometimes called an $L$-smoothness bound).\n",
"* There are other assumptions we could use here also, but this is the simplest one.\n",
"\n",
"Suppose that for some constant $L > 0$, for all $x$ and $y$ in $\\R^d$,\n",
"\n",
"$$\n",
" \\| \\nabla f(x) - \\nabla f(y) \\| \\le L \\| x - y \\|. \n",
"$$\n",
"\n",
"Here, $\\| \\cdot \\|$ denotes the $\\ell_2$ norm: $\\|u\\| = \\sqrt{\\sum_{i=1}^d u_i^2}$.\n",
"\n",
"* This is equivalent to the condition that $\\| \\nabla^2 f(x) \\|_2 \\le L$ (where the norm here denotes the induced norm) if the function is twice-differentiable."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Starting from this condition, let's look at how the objective changes over time as we run gradient descent with a fixed step size.\n",
"\n",
"\\begin{align*}\n",
"f(w_{t+1})\n",
"&=\n",
"f(w_t - \\alpha \\nabla f(w_t)).\n",
"\\end{align*}"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Let's consider this expression as a function of $\\alpha$. Call this $\\rho(\\alpha) = f(w_t - \\alpha \\nabla f(w_t))$. Observe that\n",
"\n",
"$$\\rho'(\\alpha) = \\frac{\\partial}{\\partial \\alpha} f(w_t - \\alpha \\nabla f(w_t)) = -\\nabla f(w_t - \\alpha \\nabla f(w_t))^T \\nabla f(w_t).$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Now, by the **fundamental theorem of calculus**,\n",
"\n",
"$$\n",
" \\rho(\\alpha) - \\rho(0) = \\int_0^\\alpha \\rho'(\\eta) \\; d \\eta.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"If we substitute back in our definition of $\\rho$, we get\n",
"\n",
"$$f(w_{t+1}) = f(w_t - \\alpha \\nabla f(w_t)) = f(w_t) - \\int_0^{\\alpha} \\nabla f(w_t - \\eta \\nabla f(w_t))^T \\nabla f(w_t) \\; d \\eta.$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"We can now simplify this as follows. (Keep in mind: $\\| \\nabla f(x) - \\nabla f(y) \\| \\le L \\| x - y \\|$).\n",
"\n",
"\\begin{align*}\n",
"f(w_{t+1}) \n",
"&= \n",
"f(w_t) - \\int_0^{\\alpha} \\nabla f(w_t - \\eta \\nabla f(w_t))^T \\nabla f(w_t) \\; d \\eta.\n",
"\\\\&= \n",
"f(w_t) \n",
"- \\alpha \\nabla f(w_t)^T \\nabla f(w_t)\n",
"- \\int_0^{\\alpha} \\left( \\nabla f(w_t - \\eta \\nabla f(w_t)) - \\nabla f(w_t) \\right)^T \\nabla f(w_t) \\; d \\eta.\n",
"\\\\&= \n",
"f(w_t) \n",
"- \\alpha \\| \\nabla f(w_t) \\|^2\n",
"+ \\int_0^{\\alpha} \\left( \\nabla f(w_t) - \\nabla f(w_t - \\eta \\nabla f(w_t)) \\right)^T \\nabla f(w_t) \\; d \\eta.\n",
"\\\\&\\le\n",
"f(w_t) \n",
"- \\alpha \\| \\nabla f(w_t) \\|^2\n",
"+ \\int_0^{\\alpha} \\left\\| \\nabla f(w_t) - \\nabla f(w_t - \\eta \\nabla f(w_t)) \\right\\| \\cdot \\left| \\nabla f(w_t) \\right| \\; d \\eta.\n",
"\\end{align*}\n",
"Where the last line follows from $x^T y \\le \\| x \\| \\cdot \\| y \\|$. Now by our smoothness bound,\n",
"\\begin{align*}\n",
"f(w_{t+1}) \n",
"&\\le\n",
"f(w_t) \n",
"- \\alpha \\| \\nabla f(w_t) \\|^2\n",
"+ \\int_0^{\\alpha} L \\cdot \\left\\| (w_t) - (w_t - \\eta \\nabla f(w_t)) \\right\\| \\cdot \\left| \\nabla f(w_t) \\right| \\; d \\eta\n",
"\\\\&=\n",
"f(w_t) \n",
"- \\alpha \\| \\nabla f(w_t) \\|^2\n",
"+ L \\| \\nabla f(w_t) \\|^2 \\cdot \\int_0^{\\alpha} \\eta \\; d \\eta\n",
"\\\\&\\le\n",
"f(w_t) \n",
"- \\alpha \\| \\nabla f(w_t) \\|^2\n",
"+ \\frac{1}{2} \\alpha^2 L \\| \\nabla f(w_t) \\|^2\n",
"\\\\&\\le\n",
"f(w_t) \n",
"- \\alpha \\left(1 - \\frac{1}{2} \\alpha L \\right) \\cdot \\| \\nabla f(w_t) \\|^2.\n",
"\\end{align*}\n",
"\n",
"Note that there is a somewhat easier way of getting to this result using Taylor's theorem rather than FTC; this easier approach is the one I use in the notes. I'm using this more-long-winded way of getting there to make sure that everyone gets the intuition."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"So, we got to\n",
"\n",
"$$f(w_{t+1}) \\le\n",
"f(w_t) \n",
"- \\alpha \\left(1 - \\frac{1}{2} \\alpha L \\right) \\cdot \\| \\nabla f(w_t) \\|^2.$$\n",
"\n",
"If we choose our step size $\\alpha$ to be _small enough_ that $1 \\ge \\alpha L$, then this simplifies to\n",
"\n",
"$$f(w_{t+1}) \\le\n",
"f(w_t) \n",
"- \\frac{\\alpha}{2} \\cdot \\| \\nabla f(w_t) \\|^2.$$\n",
"\n",
"That is, **the objective is guaranteed to decrease at each iteration**! This matches our intuition for why gradient descent should work."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Now, if we sum this up across $T$ iterations of gradient descent, we get\n",
"\n",
"$$\n",
" \\frac{1}{2} \\alpha \\sum_{t = 0}^{T-1} \\norm{ \\nabla f(w_t) }^2\n",
" \\le\n",
" \\sum_{t = 0}^{T-1} \\left(\n",
" f(w_t)\n",
" -\n",
" f(w_{t+1})\n",
" \\right)\n",
" =\n",
" f(w_0) - f(w_T)\n",
" \\le\n",
" f(w_0) - f^*\n",
"$$\n",
"\n",
"where $f^*$ is the global minimum value of the loss function $f$.\n",
"From here, we can get\n",
"\n",
"$$\n",
" \\min_{t \\in \\{0, \\ldots, T\\}} \\norm{ \\nabla f(w_t) }^2\n",
" \\le\n",
" \\frac{1}{T} \\sum_{t = 0}^{T-1} \\norm{ \\nabla f(w_t) }^2\n",
" \\le\n",
" \\frac{2 (f(w_0) - f^*)}{\\alpha T}.\n",
"$$\n",
"\n",
"\n",
"This means that the smallest gradient we observe after $T$ iterations is getting smaller proportional to $1/T$.\n",
"So gradient descent converges...\n",
"* as long as we look at the smallest observed gradient\n",
"* and we care about finding a point where the gradient is small"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Question: does this ensure that gradient descent converges to the global optimum (i.e. the value of $w$ that minimizes $f$ over all $w \\in \\R^d$)?

"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"..."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### When can we ensure that gradient descent _does_ converge to the unique global optimum?\n",
"\n",
"Certainly, we can do this when there is only one global optimum.\n",
"\n",
"The simplest case of this is the case of **convex objectives**."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Convex Functions\n",
"\n",
"A function $f: \\R^d \\rightarrow \\R$ is convex if for all $x, y \\in \\R^d$, and all $\\eta \\in [0,1]$\n",
"\n",
"$$f(\\eta x + (1 - \\eta) y) \\le \\eta f(x) + (1 - \\eta) f(y).$$\n",
"\n",
"What this means graphically is that if we draw a line segment between any two points in the graph of the function, that line segment will lie above the function.\n",
"There are a bunch of equivalent conditions that work if the function is differentiable.\n",
"In terms of the gradient, for all $x, y \\in \\R^d$,\n",
"\n",
"$$(x - y)^T \\left( \\nabla f(x) - \\nabla f(y) \\right) \\ge 0,$$\n",
"\n",
"and in terms of the Hessian, for all $x \\in \\R^d$ and all $u \\in \\R^d$\n",
"\n",
"$$u^T \\nabla^2 f(x) u \\ge 0;$$\n",
"\n",
"this is equivalent to saying that the Hessian is positive semidefinite."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Question: What are some examples of hypotheses and loss functions that result in a convex objective??

"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"..."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"An even easier case than convexity: **strong convexity**.\n",
" \n",
"A function is strongly convex with parameter $\\mu > 0$ if for all $x, y \\in \\R^d$,\n",
"\n",
"$$f(y) \\ge f(x) + \\nabla f(x)^T (y - x) + \\frac{\\mu}{2} \\norm{x - y}^2$$\n",
"\n",
"or equivalently, for all $x \\in \\R^d$ and all $u \\in \\R^d$\n",
"\n",
"$$u^T \\nabla^2 f(x) u \\ge \\mu \\norm{u}^2.$$\n",
"\n",
"One (somewhat weaker) condition that is implied by strong convexity is\n",
"\n",
"$$\\norm{\\nabla f(x)}^2 \\ge 2 \\mu \\left( f(x) - f^* \\right);$$\n",
"\n",
"this is sometimes called the _Polyak-Lojasiewicz condition_.\n",
"* A useful note: you can make any convex objective strongly convex by adding $\\ell_2$ regularization."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Gradient descent on strongly convex objectives.\n",
"\n",
"As before, let's look at how the objective changes over time as we run gradient descent with a fixed step size.\n",
"* This is a standard approach when analyzing an iterative algorithm like gradient descent.\n",
"From our proof before, we had (as long as $1 \\ge \\alpha L$)\n",
"\n",
"$$f(w_t) \\le\n",
"f(w_t) \n",
"- \\frac{\\alpha}{2} \\cdot \\| \\nabla f(w_t) \\|^2.$$\n",
"\n",
"Now applying the Polyak-Lojasiewicz condition condition to this gives us\n",
"\n",
"$$ f(w_{t+1})\n",
" \\le\n",
" f(w_t)\n",
" -\n",
" \\alpha \\mu \\left( f(w_t) - f^* \\right).$$\n",
" \n",
" \n",
"Subtracting the global optimum $f^*$ from both sides produces\n",
"\n",
"$$\n",
" f(w_{t+1})\n",
" - \n",
" f^*\n",
" \\le\n",
" f(w_t)\n",
" - \n",
" f^*\n",
" -\n",
" \\alpha \\mu \\left( f(w_t) - f^* \\right)\n",
" =\n",
" (1 - \\alpha \\mu) \\left( f(w_t) - f^* \\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"So we got that\n",
"\n",
"$$f(w_{t+1}) - f^* \\le (1 - \\alpha \\mu) \\left( f(w_t) - f^* \\right).$$\n",
"\n",
"But this means that\n",
"\n",
"\\begin{align*}\n",
"f(w_{t+2}) - f^* &\\le (1 - \\alpha \\mu) \\left( f(w_{t+1}) - f^* \\right)\n",
"\\\\&\\le (1 - \\alpha \\mu) \\left( (1 - \\alpha \\mu) \\left( f(w_t) - f^* \\right) \\right)\n",
"\\\\&\\le (1 - \\alpha \\mu)^2 \\left( f(w_t) - f^* \\right).\n",
"\\end{align*}"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Applying this inductively, after $K$ total steps\n",
"\n",
"$$f(w_K) - f^*\n",
" \\le\n",
" (1 - \\alpha \\mu)^K \\left( f(w_0) - f^* \\right)\n",
" \\le\n",
" \\exp(-\\alpha \\mu K) \\cdot \\left( f(w_0) - f^* \\right).$$\n",
"\n",
"This shows that, for strongly convex functions, **gradient descent with a constant step size converges exponentially quickly to the optimum**.\n",
"This is sometimes called _convergence at a linear rate_."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"If we use the largest step size that satisfies our earlier assumption that $1 \\ge \\alpha L$ (i.e. $\\alpha = 1/L$), then this rate becomes\n",
"\n",
"$$ f(w_K) - f^*\n",
" \\le\n",
" \\exp\\left(-\\frac{\\mu K}{L} \\right) \\cdot \\left( f(w_0) - f^* \\right).$$\n",
" \n",
"Equivalently, in order to ensure that $f(w_K) - f^*$ for some target error $\\epsilon > 0$, it suffices to set the number of iterations $K$ large enough that\n",
"\n",
"$$K\n",
" \\ge\n",
" \\frac{L}{\\mu}\n",
" \\cdot\n",
" \\log\\left( \\frac{f(w_0) - f^*}{\\epsilon} \\right).$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"The main thing that affects how large $K$ needs to be here is the ratio $L / \\mu$, which is a function just of the problem itself and not of how we initialize or how accurate we want our solutions to be.\n",
"We call the ratio\n",
"\n",
"$$\\kappa = \\frac{L}{\\mu}$$\n",
"\n",
"the condition number of the problem.\n",
"The condition number encodes _how hard a strongly convex problem is to solve_.\n",
"\n",
"* Observe that this ration is invariant to scaling of the objective function $f$.\n",
"\n",
"When the condition number is very large, the problem can be very hard to solve, and take many iterations of gradient descent to get close to the optimum.\n",
"Even if the cost of running gradient descent is tractible as we scale, **if scaling a problem causes the condition number to blow up, this can cause issues**!"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"What affects the condition number? What can we do to make the condition number smaller?

\n",
"\n",
"DEMO.

"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"@webio": {
"lastCommId": null,
"lastKernelId": null
},
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
},
"rise": {
"scroll": true,
"transition": "none"
}
},
"nbformat": 4,
"nbformat_minor": 2
}