exaFOAM, Exploitation of Exascale Systems for Open-Source Computational Fluid Dynamics by Mainstream Industry, was a EuroHPC project from April 2021 to March 2024. A European consortium of industry, universities and HPC centres, coordinated by ESI Group (now Keysight Technologies), set out to remove the bottlenecks that keep OpenFOAM® from scaling on massively parallel HPC systems, all in open source.
Highlights
- Coherent mesh and file format for parallel I/O, developed by Wikki together with HLRS. It received the HPC Innovation Excellence Award 2023.
- Pre- and post-processing of the 489-million-cell combustor case 31 times cheaper than with OpenFOAM’s collated format; the mesh decomposition alone is 60 times faster.
- Combustor Grand Challenge: time to solution 30 times shorter than the best result before the project, with scaling up to 131,072 cores.
- A record run on 4,096 nodes (524,288 CPU cores) of the Hawk supercomputer at HLRS.
- 30 public benchmarks in the repository of the OpenFOAM HPC Technical Committee, from a few thousand to over two billion cells.
- A prototype of fine-grained dynamic load balancing inside OpenFOAM.
The coherent I/O format
OpenFOAM’s file formats were designed for a few hundred processors. A mesh is stored already decomposed for a fixed number of processors, either as one file per processor and field or collated into large files that are written by a serial tool. For the 489-million-cell combustor, decomposing the mesh into 32,768 partitions took about 19 hours, and 65,536 partitions were impossible within the 24-hour job limit.
In the coherent format, the mesh is stored independently of the decomposition. Cells, faces and points are ordered so that each partition reads one contiguous section of each list, and the boundaries between processors are created when the mesh is read. The same mesh can therefore be read by any number of MPI ranks, and results no longer need to be reconstructed. The data are written in parallel through the ADIOS2 library, so a case consists of a handful of files instead of hundreds of thousands.
Over a typical workflow of decomposing, renumbering and reconstructing the combustor case on LUMI, the cost fell from 7,843 to 252 core-hours. The coherent format is available as a preview release for OpenFOAM v2412; its latest development state is on GitLab. Wikki provides tutorials, and a training session from the 18th OpenFOAM Workshop introduces the format. Its integration into the official release is being discussed with the OpenFOAM developers.
Combustor Grand Challenge
Wikki set up one of the project’s Grand Challenge cases: a large-eddy simulation of a combustor built at the Institute of Combustion Technology of the German Aerospace Center (DLR). It uses a recirculation-stabilised jet flame (FLOX), a technology developed for industrial furnaces with great potential for gas turbines. The geometry is scripted in FreeCAD, and meshes with 3, 24, 189 and 489 million cells were created with snappyHexMesh. The 24-million-cell case was validated against the experiment.
Profiling by BSC revealed an expensive MPI communication in the van Driest damping of the turbulence model at every time step. An algorithmic fix by ESI removed it and led to superlinear speed-up. The coherent format then made the decompositions for 65,536 and 131,072 cores possible.
Benchmarks for the community
Wikki led the work on the project’s benchmark suite and created four micro-benchmarks: a 3D lid-driven cavity, the ERCOFTAC conical diffuser in a RANS and an LES version, and the Pitz & Daily combustor. Each micro-benchmark shows the characteristic HPC issues of a large case at a size where code development is still practical.
All 30 benchmarks of the project, from external aerodynamics to reactive flows, are published in the repository of the OpenFOAM HPC Technical Committee, together with instructions, key performance indicators and guidelines for contributions. Wikki redesigned the repository’s landing page to give an overview of all cases. Restart files for selected benchmarks are available on DaRUS, the data repository of the University of Stuttgart.
Dynamic load balancing
Wikki also developed a fine-grained dynamic load balancing for OpenFOAM. Matrix operations are split into chunks that idle MPI ranks can take over from busy ones on the same node, using a thread pool, a scheduler and non-blocking global reductions. The prototype reduces MPI idle time and can be introduced operation by operation.
Publications
- R. G. Weiß, S. Lesnik, F. C. C. Galeazzo, A. Ruopp, H. Rusche: Coherent mesh representation for parallel I/O of unstructured polyhedral meshes. The Journal of Supercomputing 80, 16112–16132, 2024. doi:10.1007/s11227-024-06051-7
- F. C. C. Galeazzo, M. Garcia-Gasulla, E. Boella, J. Pocurull, S. Lesnik, H. Rusche, S. Bnà, M. Cerminara, F. Brogi, F. Marchetti, D. Gregori, R. G. Weiß, A. Ruopp: Performance comparison of CFD microbenchmarks on diverse HPC architectures. Computers 13(5), 115, 2024. doi:10.3390/computers13050115
- F. C. C. Galeazzo, R. G. Weiß, S. Lesnik, H. Rusche, A. Ruopp: Understanding superlinear speedup in current HPC architectures. IOP Conference Series: Materials Science and Engineering 1312, 012009, 2024. doi:10.1088/1757-899X/1312/1/012009
Funding
exaFOAM has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 956416. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and France, Germany, Spain, Italy, Croatia, Greece and Portugal.