Optimising OpenFOAM: How to get a free performance boost using compiler flags

A few months ago, I managed to meander my mind into the land of software compilation and the various flags one can use to help the compiler optimise your program.  This lead me to wonder if there was any way I could speed up OpenFOAM for “free” by adding a few of these flags and came up with the following conclusions:

• OpenFOAM compiled from source runs faster than OpenFOAM installed using a .deb package (although compiling takes longer and is much less convenient!)
• Adding -march=native to the compiler flags is a no-brainer and gives up to 20% faster execution!
• Adding -funroll-loops to the compiler flags gives you a little bit extra on top of this.
• Using -mfpmath=sse can give some marginal gains, whereas either -mfpmath=both or -mfpmath=sse,387 reduce the performance.

There are certain caveats with these findings: firstly and most obviously these are GCC flags; secondly the findings are true for the three machines tested but your mileage may vary.  So lets dig a bit further into the details, here is a graph of my results:

Each horizontal bar is a different part of my OpenFOAM benchmark (serial meshing, serial running, parallel meshing and parallel running) and/or different machine (Xeon, i7 and FX8370) and the results are noticeably variable, but generally show a net gain.

At this stage you may be thinking “excellent, but how do I change the compilation flags?”  This is something that will be system specific, but I put mine in the files c, c++, cOpt and c++Opt in the $WM_PROJECT_DIR/wmake/rules/linux64Gcc/ directory, with the flags starting -m following the -m64 flag and the flags starting -f following the -O3 flag.  So there you have it, a way of knocking a few % off your solution time, which can add up to quite a big difference when you have limited computing resources.