CS 5220 Slides and Notes

The CS&E Picture

Applications Everywhere!

Climate modeling
CAD tools (computers, buildings, airplanes, ...)
Computational biology
Computational finance
Machine learning and statistical models
Game physics and movie special effects
Medical imaging
...

Parallel Computing Essentials

Need for speed and for memory
Many processors working simultaneously on same problem
- vs concurrency (about logical structure vs performance)
- or distributed systems (coupled but distinct problems, clients and servers are often at different locations)

Why Parallel Computing?

Scientific computing went parallel long ago:

Want an answer that is right enough, fast enough
Either of those might imply a lot of work!
... and we like to ask for more as machines get bigger
... and we have a lot of data, too

Why Parallel Computing?

Today: Hard to get non-parallel hardware!

How many cores are in your laptop?
How many in NVidia’s latest accelerator?
Biggest single-node EC2 instance?

Organizational Basics

Cores packaged together on CPUs
- Cores have instruction-level parallelism (e.g. vector units)
Memory of various types (memory hierarchy)
Accelerators have similar pieces, organized differently
CPUs and accelerators packaged together in nodes
Nodes often connected in racks
Networks (aka interconnect or fabric) connecting the pieces

How Fast Can We Go?

Speed records for Linpack benchmark

https://www.top500.org

Speed measured in flop/s (floating point ops / second):

Giga ( $10^9$ ) – a single core
Tera ( $10^{12}$ ) – a big machine
Peta ( $10^{15}$ ) – current top 10 machines
Exa ( $10^{18}$ ) – favorite of funding agencies

What do these machines look like?

How Fast Can We Go?

An alternate benchmark: Graph 500

Data-intensive graph processing benchmark
Metric is traversed edges per second (TEPS)
How do the top machines for Linpack and Graph 500 compare?

What do these machines look like?

What HW and How Fast?

Some high-end machines look like high-end clusters
- Except custom networks.
Achievable performance is
- $\ll$ peak performance
- Application-dependent
Hard to achieve peak on more modest platforms, too!

Parallel Performance in Practice

So how fast can I make my computation?

Peak $>$ Linpack $>$ Gordon Bell $>$ Typical
Measuring performance of real applications is hard
- Even figure of merit may be unclear (flops, TEPS, ...?)
- Typically a few bottlenecks slow things down
- And figuring out why they slow down can be tricky!
And we really care about time-to-solution
- Sophisticated methods get answer in fewer flops
- ... but may look bad in benchmarks (lower flop rates!)

Example: Reduction

How can we speed up summing an array of length $n$ with $p \leq n$ processors?

Theory: $n/p + O(\log(p))$ time with reduction tree
Is this realistic?

Quantifying Parallel Performance

Starting point: good serial performance
Strong scaling: compare parallel to serial time on the same problem instance as a function of number of processors ( $p$ )

$\begin{aligned} \mbox{Speedup} &= \frac{\mbox{Serial time}}{\mbox{Parallel time}} \\ \mbox{Efficiency} &= \frac{\mbox{Speedup}}{p} \end{aligned}$

Barriers

Ideally, speedup = $p$ . Usually, speedup $< p$ .

Barriers to perfect speedup:

Serial work (Amdahl’s law)
Parallel overheads (communication, synchronization)

Amdahl’s Law

$\begin{aligned} p = & \mbox{ number of processors} \\ s = & \mbox{ fraction of work that is serial} \\ t_s = & \mbox{ serial time} \\ t_p = & \mbox{ parallel time} \geq s t_s + (1-s) t_s / p \end{aligned}$

Amdahl’s law: $\mbox{Speedup} = \frac{t_s}{t_p} = \frac{1}{s + (1-s) / p} > \frac{1}{s}$

So $1\%$ serial work $\implies$ max speedup < $100 \times$ , regardless of $p$ .

A Little Experiment

Let’s try a simple parallel attendance count:

Parallel computation: Rightmost person in each row counts number in row.
Synchronization: Raise your hand when you have a count
Communication: When all hands are raised, each row representative adds their count to a tally and says the sum (going front to back).

(Somebody please time this.)

A Toy Analysis

Parameters: $\begin{aligned} n = & \mbox{ number of students} \\ r = & \mbox{ number of rows} \\ t_c = & \mbox{ time to count one student} \\ t_t = & \mbox{ time to say tally} \\ t_s \approx & ~n t_c \\ t_p \approx & ~n t_c / r + r t_t \end{aligned}$

How much could I possibly speed up?

Modeling Speedup

Student count:

viewof nstudents = Inputs.range(
  [10, 200], 
  {value: 100, step: 1}
);

nstudents = 100

Plot.plot({
  marks: [
    Plot.lineY(transpose(data), {x: "rows", y: "speedup"})
  ]
})

(Parameters: $t_c = 0.3$ , $t_t = 1$ .)

Modeling Speedup

Mostly-tight bound: $\mathrm{speedup} < \frac{1}{2} \sqrt{\frac{n t_c}{t_t}}$

Poor speed-up occurs because:

The problem size $n$ is small
The communication cost is relatively large
The serial computation cost is relatively large

Some of the usual suspects for parallel performance problems!

Weak scaling?

Things would look better if I allowed both $n$ and $r$ to grow — that would be a weak scaling study.

This probably does not make sense for a classroom setting…

Summary: Parallel Performance

Today:

We’re approaching machines with peak exaflop rates
But codes rarely get peak performance
Better comparison: tuned serial performance
Common measures: speedup and efficiency
Strong scaling: study speedup with increasing $p$
Weak scaling: increase both $p$ and $n$
Serial overheads and communication costs kill speedup
Simple analytical models help us understand scaling

And in case you arrived late

http://www.cs.cornell.edu/courses/cs5220/2024fa/

... and please enroll and submit HW0!

CS 5220

Logistics

CS 5220

Enrollment

Prerequisites

Objectives: Performance sense

Objectives: Learn about HPC

Objectives: Numerical SWE

Lecture Plan: Basics

Lecture Plan: Technology

Lecture Plan: Patterns

Coursework: Lecture (10%)

Coursework: Homework (15%)

Homework 0

Coursework: Group projects (45%)

Coursework: Final project (30%)

Palate Cleanser

Hello, world!

The Good Stuff

The CS&E Picture

Applications Everywhere!

Parallel Computing Essentials

Why Parallel Computing?

Why Parallel Computing?

Organizational Basics

How Fast Can We Go?

How Fast Can We Go?

What HW and How Fast?

Parallel Performance in Practice

Example: Reduction

Quantifying Parallel Performance

Barriers

Amdahl’s Law

A Little Experiment

A Toy Analysis

Modeling Speedup

Modeling Speedup

Weak scaling?

Summary: Parallel Performance

And in case you arrived late