CS 5220

Applications of Parallel Computers

Mixed languages, libraries, frameworks

Prof David Bindel

Please click the play button below.

Grumpy Old Man Warning!

  • Lots of opinions ahead
  • I can be wrong!

I have said…

  • Use libraries! And frameworks! And tools!
  • Use high-level languages (Lua, Python)!
  • Lots of cool new tools! UPC++, SYCL, …
  • Languages evolve, prefer modern C/C++/Fortran/Python
  • You just need to tweak the Makefile.in files

Why Johnny Can’t Build

  • Libraries aren’t installed
  • How do I call Fortran from C++?
  • How do I build with Lua, Python, etc?
  • What about accelerators?
  • How do I link everything?


  • Tool pragmatics
  • Languages: big and little
  • Some major frameworks

What is a compiler?

C/Fortran build flow

  • Compiler maps source to assembly
    • Or to something else
  • Assembler maps to object files
  • Librarian produces
    • Static *archives (.a)
    • Dynamic libraries (.dylib) / shared objects (.so)
  • Linker combines objects, resolves symbols
  • Loader brings executables into memory

Compilation issues

  • Language issues
    • Old compiler? (common, alas)
    • Specific extensions?
  • Missing header files?

Assembler issues?

AARGH! (but not so common)

Static linker issues?

  • Problems with name mangling
    • Know: extern "C" and iso_c_binding
  • Can’t find libraries?
    • Esp language support libraries

Dynamic linking issues?

  • Dynamic libraries resolve at load time
    • Need to have them on search paths!
  • Classic problem: “DLL hell”
    • Versioning on libraries helps
    • Still can run into issues


  • Lowest: compile/link
  • Next: Makefiles
  • Next: CMake/autoconf/etc
  • Use package managers if possible!


  • configure; make; make install
  • Behind the scenes: shell scripts and M4
  • Finds libraries / platform specifics
  • Generates build system for you


  • Commercial tool (Kitware)
    • Pay for documentation
  • Configure and build all in one
  • Improvement on autotools?

Package managers

  • Common on single node
    • Linux: apt, yum
    • MacOS: brew, macports
    • Cross: conda
  • Spack, EasyBuild
    • Support environment modules
    • Allow multiple build toolchains

A step back

The mountain of abstraction

Consider class trajectory:

  • Started very low-level
  • Up to general ideas/kernels
  • Up to parallel concepts, applications
  • Nirvana: performance “just happens”?

Low-level frameworks and languages

  • OpenMP and MPI
  • Intel TBB
  • Global arays
  • Newer(?) parallel languages, extensions


One thing (or a few) done fast:

  • BLAS (MKL, OpenBLAS, ATLAS, etc)
  • LAPACK and successors
  • FFTW
  • Sparse direct solvers

Key challenge: linking (esp across languages)

Framework libraries

  • Many in PDE land
    • PETSc, SLEPc, TAO, etc
    • Trilinos
    • Overture
    • deal.ii
  • More complicated interfaces
  • Effectively defines embedded solver lang

Framework libraries

Renaissance thanks to ML!

  • TensorFlow, PyTorch, JAX, etc
  • Often define embedded DSLs
    • Build expression in Python, etc
    • Separate compile/optimize/execute

Scripting languages and PSEs

  • MATLAB, Octave, R, Python, Julia
  • “High productivity” vs “high performance”?
  • Not necessarily slow! Extensions, libs, JIT
  • Performance strategies transfer
  • Bottlenecks may not be where you expect

Domain specific languages

Great opportunities from limited scope!

  • Classic example: SQL
  • PDE domain: finite element compilers
    • Dolfin framework
    • Sundance
  • Embedded languages/specializers
  • Modern: SimIt, GraphIt, Halide, etc

Simulation codes

  • Typical pattern
    • Cusom lang/preproc for problem input
    • Scripting language to describe analysis
    • User-defined elements/modules in compiled lang
  • Great for some classes of problems
  • Can be torturned into covering other types!

Role of scripting languages

  • Often interpreted
  • Great for “high-level”
  • Python is the default (or Lua?)
  • Interface with C/Fortran for perf

Issues in mixing languages

  • How do languages communicate?
  • How are extensions compiled/linked?
  • What support libraries are needed?
  • Who owns main loop?
  • Who owns program objects?
  • How are exceptions handled?

Cross-language calls?

  • Via pipes (pre-/post-processors)
  • Inter-process communication
  • Remote procedure call
  • Linked together (with wrapper gen?)

Mixed language: tech issues

Primary pain points:

  • Cross-language communication
  • Building and deployment
  • Debugging and run-time issues

Mixed language: social issues

  • Availability of docs
  • Availability of tools/libraries
  • Active user base
  • Open-ness of code base?
  • Longevity

Best advice

  • Conservative re general-purpose lang
    • (Modern) C++ and Fortran
    • Otherwise, look for C-compatible calling
    • Python on top
  • More adventurous for DSLs

Also like Julia (yes, I’m inconsistent)

Concluding thought

I don’t know what the programming languge of the year 2000 will look like, but I know it will be called FORTRAN. – C.A.R. Hoare