# CS 5220: Applications of Parallel Computers ## Matmul and tiling ## 08 Sep 2015
## A memory benchmark (membench) for array A of length L from 4KB to 8MB by 2x for stride s from 4 bytes to L/2 by 2x time the following loop for i = 0 to L by s load A[i]

## Membench in pictures

• Size = 64 bytes (16 ints)
• Strides of 4, 8, 16, 32 bytes

## Membench on totient CPU

• Vertical: 64B line size, 4K page size
• Horizontal: 64KB L1, 256KB L2, 15MB L3
• Diagonal: 8-way cache associativity, 512 entry L2 TLB

## Note on storage

• Two standard layouts:
• Column major (Fortran): A(i,j) at A+i+j*n
• Row-major (C): A(ij) at A+i*n+j
• I default to column major
• Also note: C has poor language matrix support
## Matrix multiply How fast can naive matrix multiply run? #define A(i,j) AA[i+j*n] #define B(i,j) BB[i+j*n] #define C(i,j) CC[i+j*n] memset(C, 0, n*n*sizeof(double)); for (int i = 0; i < n; ++i) for (int j = 0; j < n; ++j) for (int k = 0; k < n; ++k) C(i,j) += A(i,k) * B(k,j);

## One row in naive

• Access $A$ and $C$ with stride $8n$ bytes
• Access all $8n^2$ bytes of $B$ before first re-use
• Poor arithmetic intensity

## Hmm...

• Compiler makes some difference
• Naive Fortran is faster than naive C
• Local instruction mix sets speed of light
• Access pattern determines how close we get to limit

## Engineering strategy

• Maybe odd sizes, strange layouts -- just go fast!
• May play with AVX intrinsics, compiler flags, etc
• Deserves its own timing rig
• Use blocking based on kernel to improve access pattern
## Simple model - Two types of memory (fast+slow) - $m$ = words read from slow memory - $t_m$ = slow memory op time - $f$ = number of flops - $t_f$ = time per flop - $q = f/m$ = average flops/slow access - Time: $$ft_f + mt_m = ft_f \left( 1 + \frac{t_m/t_f}{q} \right)$$ - Larger $q$ means better time
## How big can $q$ be? Level 1/2/3 Basic Linear Algebra Subroutines (BLAS) 1. Dot product: $n$ data, $2n$ flops 2. Matrix-vector multiply: $n^2$ data, $2n^2$ flops 3. Matrix-matrix multiply: $2n^2$ data, $2n^3$ flops We like to build on level 3 BLAS (like matrix multiplication)!
## Tuning matrix multiply - [Matmul assignment is up](https://github.com/cornell-cs5220-f15/matmul) - You will get email with group assignments - Goal is single core performance *analysis* and *tuning* - Deliverables - Report describing strategy and performance results - Pointer to a repository so we can run a competition
## Possible tactics - Manually tune some small kernels - Write an auto-tuner to sweep parameters - Try different compilers (and flags) - Try different layouts - Copy optimization - Study strategies from past/present classes!
## Warning - Tuning can be like video games! - Do spend the time to do a good job - Don't get so sucked in you neglect more important things