Cornell Theory Center

• The Challenge
• The Programming Cycle
• Kinds of Tools Used
• Creating HPF Source
• Compiling
• Debugging
• Performance Tuning
• Summary

• HPF is a Relatively New Language
• Vendor Tool Groups Remote are from Compiler Groups
• Little Experience in the Field
• Different Kinds of Tools are Needed than for Message Passing

These are described next, but first let us consider the HPF programming life cycle.

• Generally you start with existing source
• Generally you start with the serial code
• Sometimes it really old and moldy F77
• You rarely convert MPI to HPF

• Generally you know where your major loop is
  • If it runs over your data, you want to parallelize this
  • Key to get rid of inhibitors
  • Replace IF with WHEN
  • Use F90 intrinsic functions
• Get Rid of Data Dependences
• What Large Array is Stored into by the Loop?

• Find the Loop[s] (Serial Profiling, e.g. Forge)
• Analyze Data Dependences (FORGE, Pghpf)
• Random Number Generation (PRNG)
• Choose a Data Distribution
• Seed the Program (XHPF)

• kapf (Kuck & Associates)
• Forge Explorer (Applied Parallel Research)
• PAT (Obsolete Tool from Georgia Tech)

• Often easy to pick out one key distribution
• Tedious to hunt them all down
• xhpf (from Applied Parallel Research) is handy for this:
  1. First, insert one or more distribution and align directives
  2. Load up all of your source into Forge Explorer as a package
  3. Run your code through xhpf as follows:

  xhpf -p pkg -Auto=1 -BenignUnknown -ohpf=pkg_hpf.f
  

• The output has many more ALIGN and DISTRIBUTE directives added!

Plus if you use -plist to save the parallelization report, you'll get more information about parallel loop inhibitors.

A handy way to get independent streams is to associate a different stream with each index of the loop you wish to parallelize. We have a locally developed tool at Cornell to do just that:

          real*8 function prng_next (i)
          integer i

Available from ftp.tc.cornell.edu as pub/utilities/prng.tar.Z.

• Right now, the more HPF compilers you can run, the better
• Different reports, different error messages
• Different features
• Different possibilities for instrumentation
• All ultimately produce and MPI program

Example: Different report formats

1. 1-way: test accuracy of program cleanup
   • answers should match the serial
   • execution time should be within 5%

2. 2-way: test parallelism
   • answers should still match
   • execution time should probably be horrible
   • ... and get worse when you go 3-way. (Don't worry).

3. You should hope the answers are right, because
   most parallel debuggers are clueless about HPF

• Analyze variables in INDEPENDENT loops again
• Use Print statements
• Use a parallel debugger
  • pdbx, a generic linemode parallel debugger
  • TotalView with PGHPF (wave of the future)
• User markers containing information

Example: an HPF debugger

• profile (statement,loop, routine level) - any useful concurrency?
• compiler report - did the loops really parallelize?
• trace - serial bottlenecks?
• monitor - I/O a problem?

• vampir trace shows load imbalance
• same info in a vt trace
• xprofiler shows statement level hot spots
• pgprof profiler shows program hotspots, and communication to computation ratio
• FORGE parallel profiler shows communication to computation ration
• summary statistics from mp2sddf and pablo
• ntv is simple to use and gives a useful statistical overview

• Excessive speed-down: usually due to excessive communication
  • Check profile or trace
  • Reorganize or replicate your data to reduce communication
  • If it is I/O, parallelize it (i.e. stripe it)
  • Check if loops are replicated but working with distributed data
• Flat performance (no speedup)
  • Check to see if each machine is executing the whole program
  • Trace can show one processor executing on its data while other processors wait
  • Check compiler generated code to see if loop is serialized

In summary, you will probably be using trace performance tools more than debugging tools to get your HPF codes to work well. Why this is the case is shown in a diagram I like to show, which compares the MPI programming experience with the HPF one:

As with all parallel codes, your first attempt is almost certain to get slowdown. And there comes a point beyond which it doesn't pay to optimize more. That is shown in the two curves here. Unlike MPI, where your effort will be to getting the right answer, (or even to get it running), with HPF your effort will be to get acceptable performance. Ultimate HPF performance is a little lower than MPI, the speedup knee is a little sooner, but you get there with the same effort (or less) as with MPI.



---