Prelim 2  Tuesday, November 16 7:30-9:00pm, in WN 131/231 (Warren)
I will try to get us a single large room.  If I succeed, the info will be on the web and posted on the doors of WN 131/231.
Prelim coverage: through the evaluator lecture.  You will be asked to add a special form to the evaluator on the prelim (also, on the final!)
Review session will be Monday evening, or possibly Friday (sorry, scheduling problems are hard). Review time: (show of hands).  Room TBA on the web.
Practice prelim with solutions is on the web.  Please do it before looking at the solutions!!
PS #6 is now up.  The compiler part will be posted next week.
New topic area today.  
  Streams:
· Another way of thinking about state
For the last month or so we've been working with local state
· procedural abstraction and local state variables
    > a bank account with hidden balance variable
· Object-oriented programming
    > instance variables of objects
· Mutable data structures
    > queues and heaps and such
These all have the feel of ``real life'':
· The state of the moment is visible.
· Past states are just memories
· Future states are inaccessible.
For example, we'd simulate
(A) A physical object with some state
    >>> draw a spaceship <<<
as a
(B) A computational object with some state
    >>> (make-spaceship) <<<
Changes in (A) are reflected in (B) and vice versa.
 
Today, we'll look at a different view of state:
  * More like a book (or CD-ROM)
    - Watch the state in terms of its evolution across time.
    - The whole thing's there if we want to skim forward or backward.


This view of programming is rather like signal processing.
· Watch the signal / data flow through a processor/program.
          +-----------+   +----------+   +------+ 
{laser    |           |   |          |   |      | 
 light}-->| CD player |-->| Preamp   |-->| Amp  |-->sound      
          |           |   |          |   |      | 
          +-----------+   +----------+   +------+
             
You can think of each of those boxes as something that takes an entire 72-minute signal, and outputs a different 72-minute signal that's closer to music.  (Or you can look at what is there at each point in time.)
>>> Draw some curves at each of the arrows, some notes at the sound
We're going to do something similar, having information flowing through a collection of boxes.
We'll look at primitives (kinds of boxes)
· enumerate [generate a signal]
· map [turn one signal into another]
· filter [remove some signal values]
· accumulate [turn a signal into a scalar]
Idea:
· Stereos and sound-systems are great because they come in lots of boxes and you can hook them together in many many useful ways.
· We'll try to do the same thing with boxes for streams.
We've looked at some of these (e.g., map) for lists.  Streams will be similar.
· Initially, they'll just *be* lists
· But next time we'll let them be infinitely long.
    - We'll have a stream of all the integers.
    - That couldn't possibly fit as a list.
 
First, let's look at a pretty standard piece of Scheme code:
  * Sum the squares of the prime integers between 1 and N:
(define sum-prime-squares
  (lambda (n)
    (letrec 
       ((next (lambda (k)
          (cond
              ((> k n) 0)
              ((prime? k) (+ (square k) (next (+ 1 k))))
              (else (next (+ 1 k))))))
      (next 1))))
There are four things going on here:
  1. ENUMERATE      the numbers 1 ... n
  2. FILTER	    out the prime numbers from that list
  3. MAP	    square on each of the selected numbers
  4. ACCUMULATE     the result, using +, starting from 0.
Note that filter/map/accumulate are already defined on lists. We'll could make them generics to handle streams, or we could call the stream ones enumerates, filters, maps, accumulates.
     +-----+     +-----+     +-----+      +-----+      
n -->|ENUM | --> |FILT | --> |MAP  | -->  |ACCUM| --> sum
     +-----+     +-----+     +-----+      +-----+         
 This pattern is pretty hard to see from the code, though:
· Everything's going on at once.
We're going to use STREAMS to capture this picture.
FOR NOW -- and this is *WRONG* but a good start --
  think of a stream as a list.
[Another of those 212 "white lies"...]
Here are the operations on STREAMS:
  (cons-stream thing stream)
  (empty-stream? stream)  -- true only of empty-stream
  (heads stream)
  (tails stream)
Contract:
  (heads (cons-stream thing stream)) ==> thing
  (tails (cons-stream thing stream)) ==> stream
Note that so far it looks just like lists.
Now, we can implement our procedure as a bunch of little boxes.  
Here's a stream made from scratch:
>>> Save this to the end of class <<<
 (define enumerate-interval
  (lambda (low high)
    (if (> low high)
        empty-stream
        (cons-stream low
          (enumerate-interval (+ low 1) high)))))
We'd like our program to look like that chain of black boxes. We'll write it:
(define sum-prime-squares
  (lambda (n)
    (accumulates + 0
                (maps square
                     (filters prime?
                            (enumerate-interval 1 n))))))
A black box is then going to be a Scheme function
· One input, viz. the stream from the left.
· One output, the stream going right.
    - For all but the last box, it's going to be a stream.


Now we need to define each of the boxes.  They will be just like their list equivalents, except they use <stream>, heads, tails and cons-stream instead of <list>, head, tail and cons.
>>> Save filter until end of class <<<
(define filters
  (lambda (pred stream)
    (cond ((empty-stream? stream) stream)
          ((pred (heads stream))
           (cons-stream (heads stream)
                        (filters pred (tails stream))))
          (else (filters pred (tails stream))))))
So (filters odd? s) returns a stream of all the odd numbers in stream s (if any)
(define maps
  (lambda (f stream)
    (if (empty-stream? stream) 
        stream
        (cons-stream 
            (heads stream) 
            (maps pred (tails stream))))))
(map square s) returns a stream of all the elements of s squared
 (define accumulates
  (lambda (combiner init stream)
    (if (empty-stream? stream)
        init
        (combiner (heads stream)
                  (accumulates combiner init 
                               (tails stream))))))
Now sum-prime-squares works as diagrammed in the streams diagram.
 
So just looks like we are re-writing a bunch of code that we already knew how to write another way.  Have this different view of ``boxes'' processing the data. Why are we bothering with this?
  1.  It's a very powerful metaphor.
· 60% of scientific FORTRAN code fits into this view.
· the famous and popular Unix system gets major mileage out of such a view (pipes)
  2. Once you've got a decent library of stream functions, you can throw together fancy programs quite fast.
· That's Unix's major win.
  3. It will enable us to talk about computations over infinite data
     structures:
· "the integers", say. (advert. for next lecture)
 
But... implementing streams as lists can be massively inefficient.
Suppose I ask,
  "What is the second prime between 10,000 and 93,000,000?"
>> Use this at end of lecture <<
(heads (tails (filters prime? (enumerate-interval 10000 93000000))))
We end up having to
  1. create a list of 92,990,000 integers,
  2. Check them ALL for primality, 
  3. Pick the second one!
That's a pretty impressive waste of work. 
How do we do better?
We use a common and very powerful idea:
  BE LAZY! -- but be lazy in a particular way
Specifically, 
  At selected points in the code, we deliver a promise to do something
  rather than actually doing it.
Maybe nobody will actually collect on it!  Then we don't have to do
the work!
 
The difference between streams and lists is just this:
· With a stream, the tail is *not* evaluated when you *MAKE* the stream
· It is only evaluated when you *USE* it.
- tails evaluates the tail tail
- cons-stream doesn't
Contrast this with a list.  Same contract, but evaluation happens at different time, 
  (heads (cons-stream thing stream)) ==> thing  
[stream not evaluated here]
  (tails (cons-stream thing stream)) ==> stream  
The tail is a promise to evaluate the tail when asked to, not an actual object.
 
We do this by inventing the special form DELAY:
  -- It doesn't follow the usual evaluation rules  (SPECIAL FORM)
  -- We can add it to Dylan easily using macros
(delay stuff) 
  --> delivers a promise to evaluate stuff when it has to.
(force promise)
  --> collects on the promise
(define x (delay (/ 1 0)))
· Not an error,
(force x)
· gets the division by zero error.
Note that this is a lot like functions (lambda of no arguments)
(define xx (lambda () (/ 1 0)))
(xx)
We could define delay this way as a macro, because as we saw macros allow the definition of "equivalent forms"
(delay x) is equivalent to (lambda () x)
(defmacro (delay x)
  (list 'lambda '() x)))
(defmacro (delay x)
   `(lambda () ,x))
[NOTE: this definition of delay as a macro is just here for completeness, you are not responsible for knowing how to define macros].
 
Then force would be defined as a regular function:
(define force
  (lambda (promise)
    (promise)))
 
Now that we have delay and force, back to streams.  cons-stream is a special form too, because the second argument isn't evaluated.
(cons-stream thing stream) is equivalent to (cons thing (delay stream))
Which can be defined using the following macro:
(defmacro (cons-stream t s)
  (list 'cons t (list 'delay s))))
Then,
(define heads 
  (lambda (s) (head s)))
(define tails
  (lambda (s) (force (tail s))))
And 
(define <stream> <list>)
This DELAYED EVALUATION gives a demand-driven computation
  -- Do things when you need to 
    +-----+      +-----+      +-----+      
--> |     | -->  |     | -->  |     | -->  
    +-----+      +-----+      +-----+      
It's more like pulling a string through a bunch of holes, than pouring
water through them
  - you can get as much as you want
  - but you don't get any more.
 
>>> Point to that 93000000 example <<<
(heads (tails (filters prime? (enumerate-interval 10000 93000000))))
The stream (enumerate-interval 10000 93000000) is a cons cell:
  ( 10000 . {promise to (enumerate-interval 10001 93000000)})
Well, 10000 isn't prime, so filter will ask for the next one in the stream.  
>>> Point to filter
and (tails (10000 . {promise to (enumerate-interval 10001 93000000)} ) forces the tail, which does the next enumerate-interval computation
 
Note for section: streams have an asymmetry, namely the head is always forced.  So for example 
(filter (lambda (x) (> x 1000000)) (enumerate-interval 1 10000000000)) runs for a long time.

There's one last inefficiency possible:
· If we're implementing delay as a function of no arguments (as above), what if you need the same element multiple times?
  
    Each time it is computed anew.
    Expensive!  
We MEMOIZE it the first time we compute it --- 
  save result, use it later
 
STREAMS:
· Like lists, but they *delay* their tails
    - Only evaluate them when necessary
· Delay - promise to compute something later
    - When (force x)'ed.
· Stream operations:
     maps
     filters
     etc.