CS410, Summer 1998 Lecture 2 Outline Dan Grossman Goals: * Review formal definition and practical uses of asymptotic notation * Review induction and why you gotta know it * Recurrence relations, solving them * Note: This lecture has above average mathematical scariness. Fear not. Reading: CLR, Ch. 4 Asymptotic Notation: * Easy examples to get us warmed up: * n^2 + 5000 O(n^2) (just below n = 71 is where n^2=5000) * 2^n + n^100 O(2^n) (right at n = 1024 is where 2^n=n^100) * Formal example or two (using c and N) * Claim: 17n(n+ 3) is Theta(n^2) * for O, let c=24, n_o = 3 for Omega, let c=1, n_o = 1 Actual proof is just reversing of the steps used to determine these values. Other pairs of values work too. [Note: A more complete proof was given in lecture.] Induction: * Let's agree that it's sound: * if it's true for the base case (say 1) * and assuming it's true for n then it's true for n+1 then it's true for all n >= 1 * Mathy example: sum from i = 0 to d of z^i = (z^(d+1) - 1)/(z-1) Proof: By induction Our I.H. is: sum from i=0 to d of z^i = (z^(d+1) - 1)/(z-1) * base: d=0: z^0 = 1 = (z^1 -1)/(z^1 - 1) done * inductive: d>0 sum from i=0 to d of z^ i = (by math) z^d + sum from i=0 to (d-1) of z^i = (by I.H. with (d-1)) z^d + (z^d -1)/(z-1) = (by math) [z^d(z-1)+z^d-1]/(z-1) = (by math) [z^(d+1)-z^d+z^d]/(z-1) = (by math) done Recurrence Relations: * Motivation: Often easy to derive the relation from the code. Now we wish to bound the relation. * Basic approach: Use ad hoc methods to "see what it oughtta be", then use induction to make sure you're right. * Method one: Guess. * Eg. T(n) = T(n-1) + O(1) Claim: T(n) is O(n) Proof: By induction. I.H. is T(n) <= cn * base: T(1) is O(1) always (we don't bother to write it) need 1 <= c, so just pick a c >= 1 * inductive: T(n) = T(n-1) + O(1) (by definition) <= c(n-1) + k (by I.H. and defn of O(1)) <= cn (as long as c >= k) Note: By the definition of O, we must use the SAME c throughout the proof. * example on page 56 for strengthening inductive hypothesis. [See the text, no need for me to re-type it.] * Method two: Expand it out, then guess. (Use recursion tree) Eg. T(n) = T(n/2) + 2T(n/4) + 3n [Drawn sideways from what we did in lecture] 3n/16 ... 3n/4 3n/16 ... 3n/8 ... 3n/16 ... 3n 3n/4 3n/16 ... 3n/8 ... 3n/8 ... 3n/2 3n/8 ... 3n/4 ... What we are doing is expanding out the formula. The sum of all nodes in the tree is T(n). The root node is the time for T(n) not spent solving smaller problems. Each subtree is the time required to solve a smaller problem. So the three subtrees of the root represent the time required to solve T(n/2), T(n/4), and T(n/4). We repeat the process until we see a pattern. We notice two things: * The sum of the times at each level of the tree is 3n * The tree stops when the problem size is 1 (because T(1) is O(1)). On all branches, the problem size is going down by at least a factor of 2. So the depth of the tree is nowhere more than log_2 n. On all branches, the problem size is going down by at most a factor of 4. So the depth of the tree is nowhere less than log_4 n. So there are log_2 n levels, although some will be "incomplete". So we guess that there are log_2 n levels, each of which sums to less than or equal to 3n. Putting it together we can guess T(n) is O(nlog n) and proceed to try to prove it formally (which we won't do here, but it's not too hard). * Master method: A lot of algorithms fit this framework. Plug in and you're done -- no need to figure out the induction! Get some intuition for what's going on, though. T(n) = aT(n/b) + cn^k Look up / memorize: If a < b^k, then O(n^k) If a = b^k, then O(n^k log n) If a > b^k, then O(n^(log_b a)) Get intuition for what makes some factors bigger than others: As k gets big, the work done for big problem instances overshadows the work done for small problems As b gets small, the size of the big problems overshadows the size of the small problems Note: Book definition is every-so-slightly more general. Proof of the Master Theorem: Assume just for mathematical simplicity that n=b^d for some d. T(n) = cn^k + aT(n/b) (by definition) = cn^k + a(c(n/b)^k + a(c(n/b^2)^k + a(... + c(n/b^d)^k))) (by expansion) = cn^k[1 + a/b^k + (a/b^k)^2 + ... + (a/b^k)^d] (by rearrangement) = cn^k[sum from i=0 to d of (a/b^k)^i] = cn^k[sum from i=0 to d of z^i] (letting z = a/b^k) * case: a < b^k, that is z < 1 T(n) = cn^k[sum from i=0 to d of z^i] < cn^k[sum from i=0 to infinity of z^i] (by adding terms > 0) = cn^k(1/(1-z)) (by applying earlier proof in the limit of d going to infinity) is O(n^k) (because 1/(1-z)) is a constant * case: a = b^k, that is z = 1 T(n) = cn^k[sum from i=0 to d of z^i] = cn^k[sum from i=0 to d of 1] = cn^kd = cn^klog_b n (because n=b^d, so d = log_b n) is O(n^k log n) (see your hw for why the base doesn't matter) * case: a > b^k, that is z > 1 T(n) = cn^k[sum from i=0 to d of z^i] = cn^k(z^(d+1)-1)/(z-1) (by earlier proof) is O(n^kz^d) (because z-1 is a constant and z^(d+1)+1 is O(z^d)) Well, n^kz^d = (b^d)^k(a/b^k)^d (by plugging in for n and z) = a^d (by cancelling) = a^(log_b n) (by plugging in for d) = n^(log_b a) (by a logarithm identity) Done!