CS 5220

Basic questions

• How much does a message cost?
  • Latency: time to get between processors
  • Bandwidth: data transferred per unit time
  • How does contention affect communication?
• This is a combined hardware-software question!
• Goal: understand just enough to model roughly

Conventional wisdom

• Roughly constant latency (?)
  • Wormhole routing (or cut-through) flattens latencies vs store-forward at hardware level
  • Software stack dominates HW latency!
  • Latencies not same between networks (in box vs across)
  • May also have store-forward at library level
• Avoid topology-specific optimization
  • Want code that runs on next year’s machine, too!
  • Bundle topology awareness in vendor MPI libraries?
  • Sometimes specify a software topology

$\alpha$-$\beta$ model

Crudest model: $t_{\mathrm{comm}} = \alpha + \beta M$

• $t_{\mathrm{comm}} =$ communication time
• $\alpha =$ latency
• $\beta =$ inverse bandwidth
• $M =$ message size

Works pretty well for basic guidance!

Typically $\alpha \gg \beta \gg t_{\mathrm{flop}}$. More money on network, lower $\alpha$.

LogP model

Like $\alpha$-$\beta$, but includes CPU time on send/recv:

• Latency: the usual
• Overhead: CPU time to send/recv
• Gap: min time between send/recv
• P: number of processors

Assumes small messages (gap $\sim$ bw for fixed message size).

Communication costs

Some basic goals:

• Prefer larger to smaller messages (avoid latency)
• Avoid communication when possible
  • Great speedup for Monte Carlo and other embarrassingly parallel codes!
• Overlap communication with computation
  • Models tell you how much computation is needed to mask communication costs.

Intel MPI on Totient

• Two 6-core chips per nodes, eight nodes
• Heterogeneous network:
  • Ring between cores
  • Bus between chips
  • Gigabit ethernet between nodes
• Test ping-pong
  • Between cores on same chip
  • Between chips on same node
  • Between nodes

Approximate $\alpha$-$\beta$ parameters (on node)

Approximate $\alpha$-$\beta$ parameters (cross-node)

Network model

• On-chip: $\alpha$-$\beta$ works well!
• Off-chip: Not so much
• But cross-node communication is clearly expensive!

Moral

Not all links are created equal!

• Might handle with mixed paradigm
  • OpenMP on node, MPI across
  • Have to worry about thread-safety of MPI calls
• Can handle purely within MPI
• Can ignore the issue completely?

For today, we’ll take the last approach.