CS410, Summer 1998 Lecture 17 Outline Dan Grossman Goals: * Sorting * Selection Sort * Insertion Sort * Heap Sort * Merge Sort Reading: CLR, chapters 1 and 7. Sedgewick has a much more complete discussion (including many methods we won't discuss) in chapters 6, 8, and 9. We are going to learn a lot of ways to sort. The specification of sorting was given yesterday. For each method, we will be concerned with several attributes: * Is the method stable? * Is the method "list-friendly" or is the best way to sort lists using the method to convert the list to an array? * Is the method "in-place" or do we need more than O(1) additional space? * What is the best-case/worst-case/"average-case" running time? What input data exhibits the best-case and worst-case? * How do the constant factors hidden by big-O compare between methods? * How can we get constant factor improvements for the method? For conciseness, we will generally ignore that the keys have values attached to them. In reality, the values are there and we carry them around in the obvious way. Selection Sort For i from 1 to n, find the ith smallest thing in the array and swap it with whatever is currently at array[i]. After n iterations, the array is sorted. On the ith iteration, the first i-1 positions are correct, so it suffices to find the smallest element in the ith through nth positions. This method is stable as long as we only update are "smallest so far" when we find an element strictly less than the current smallest. The method is list-friendly. We need two pointers -- on for the start of the next iteration and one for finding the smallest thing in the rest of the list. Each only moves in one direction, so we can efficiently move them through the list. The method is in-place. The running time is always O(n^2). More precisely, iteration i takes time proportional to n-i. So the total is the sum of i from 1 to n of (n-i). This is the same as the sum of i from 1 to n of i, which we know from homework 1 is n(n+1)/2. This is basically the slowest method we'll learn, but it is simple. It also does fewer swaps than any other method -- only n of them. If our values were large and we were swapping the actual data instead of a pointer to it, then this could be a reason for selection sort. I'm not sure you will have such a situation very often, though. Insertion Sort After i iterations, the first i items are sorted. We add the next item by finding the correct place for it among the already sorted items. The code might look like: for (j=1; j < arr.length; j++) i = j-1; key = arr[j]; while(i >= 0 && arr[i] > key) { arr[i+1] = arr[i]; i--; } arr[i+1] = key; Basically, we are moving the jth forward, swapping it with the previous element until it is in the right place. But rather than swap with each element, we just move the elements forward and then put it in the right place. We did the exact same efficiency improvement with heapify when we studied heaps. The method is stable because the we use strict inequality for arr[i] < key. The method is list-friendly. As shown above, it would require a doubly-linked list. If we have the "find the right place pointer" go forward stopping if it gets to the jth position (rather than backwards stopping if it gets to the beginning), then a singly-linked list works fine. (I forgot to mention this in class.) Moving forward instead of backward, exactly flips the best-case and worst-case situations described below. The method is in-place. The best-case running time is O(n). This is when the list is already sorted. It is O(n) because the while loop is never executed, so each of the n iterations is O(1). The worst-case running time is O(n^2). This is when the list is reverse-sorted. On the ith iteration, the while loop executes i times, so we have a total running time proportional to the sum of i from 1 to n of i. That is, n(n+1)/2. If the array elements are originally in a "random" order, then we expect the ith iteration to execute the while loop i/2 times. The running time in this case is O(n^2). The algorithm never does more comparisons than selection sort. Its constant factors are quite low. You may have learned bubble sort in high school. It is inferior to insertion sort and is not worth discussing further. Heap Sort We actually already saw a way to sort when we studied heaps: Build a heap out of the array O(n) Keep deleting-min until the heap is empty n*O(log n) So we can sort in total time O(nlog n). In fact, when we call delete-min for the ith time, we can put the element in array index n-i. Hence, we can do everything with the original array. That is, the method is in-place. (Heaps work best when ignoring index 0. We can just insert this one element in O(n) time at the end.) This is not list-friendly -- we can not efficiently follow the implicit tree pointers of the heap using a list. This is not stable. Here is an example (where letters reveal the original order of elements with equal keys). 1 2a / \ / \ 2a 2b ===> 2c 2b / \ / 2c 2d 2d We needed 2a to be the new root, so our policy for resolving ties, must have been, "resolve in favor of the left child". But now we're stuck with this (our heap can't change the policy based on what it needs because it doesn't know the letters!). The next delete-min will then cause ===> 2c / \ 2d 2b So 2c will appear before 2b -- the sort is not stable. Merge Sort The idea for mergesort is to divide the input in half, recursively sort each half, then merge the results. That is: mergesort (arr, l, r) { if (l < r) { m = (l+r)/2; mergesort(arr, l, m); mergesort(arr, m+1, r); merge(arr, l, m, r); } } The recursion "bottoms out" when we sort a 1-element subarray. They're really easy to sort. :-) The merge operation can run in time O(n), where n is r-l, but it uses an auxiliary array: merge(arr, l, m, r) { i = m+1; k = -1; while (k < r-l) { k++; if (i > r || arr[l] <= arr[i]) { aux[k] = arr[l] l++; } else { aux[k] = arr[i] i++; } } copy aux back into arr indices l to r } The intuition is that we we walk down each sorted subarray and just compare two pointers to see which goes next in the final array. Then when we're done, we copy everything back to where we started. So the running time for mergesort on an array of size n is T(n) = 2T(n/2) + O(n) which is O(n log n) by the Master Theorem. The running time is basically the same for any inital ordering of the data. This method is stable, but it is not in-place. It is sort of list-friendly. The only hard part is finding the middle of the list. This will take O(n) at each iteration, so there is no change in the asymptotic complexity. We will study several improvements to this basice mergesort algorithm, only the first of which we will get to today... There is no reason the recursive calls to sort smaller sub-arrays have to use the same sorting method. All we care is that the smaller array is sorted. It turns out that for small n (say less than ten or so), insertion sort is faster than merge sort. (n^2 > nlog n, but for small n, the hidden constant factors make a difference). So we could change mergesort to be if (r-l <= 10) insertion_sort(arr, l, r) else merge_sort (arr, l, m) merge_sort (arr, m+1, r) merge (arr, l, m, r) assuming insertion_sort is appropriately modified to handle sub-arrays. Notice that this eliminates roughly 15/16ths of all the calls to merge sort at the expense of having to do roughly n/10 insertion sorts on small arrays. This is based on the observation that almost all of the recursive calls are on array of small size. (Draw the recursion tree to see why!) This is generally a win in practice.