AMCG, Asynchronous Modified Conjugate Gradient Method, was an innovation study within the EuroHPC project Inno4scale, carried out by Wikki GmbH and the Alpines team of INRIA Paris and Sorbonne University. It developed conjugate gradient solvers that need neither global sums nor synchronised communication between processors.
Highlights
- New variants of the conjugate gradient (CG) method that avoid global synchronisation, for symmetric and asymmetric matrices.
- An asynchronous preconditioned CG (aPCG) in OpenFOAM® with one Krylov space per MPI rank, MPI one-sided communication and an asynchronous global termination.
- Enlarged communication-avoiding CG (EnlargedCACG) tested in FreeFEM; a GMRES variant for asymmetric matrices.
- On the pressure equation of the 24-million-cell DLR combustor from the exaFOAM project: 3.1 times faster at 1.5 to 1.6 times lower cost than the standard solver, on both LUMI and MareNostrum 5.
- Tested on the EuroHPC systems LUMI and MareNostrum 5.
The challenge
Most simulation codes spend a large part of their run time in linear solvers, and at scale they are limited by the cost of communication: global sums and the exchange of data across partition boundaries. In the exaFOAM project, profiling of the 489-million-cell DLR combustor case prepared by Wikki showed a parallel efficiency of only 25–30 % of the solvers at more than 32,000 cores. A large part of the loss came from global reductions, where every process waits for all the others.
The effect is largest for “light-weight” problems such as the pressure equation in transient CFD: each solution needs only a few iterations, so the synchronisation in every iteration dominates.
The solution
The classic CG method advances one Krylov space for the global problem, which needs global sums in every iteration. AMCG instead treats each partition as its own problem and advances one Krylov space per MPI rank. The coupling to the neighbouring partitions enters as an additional source term that is updated whenever new boundary data arrive.
The asynchronous PCG uses the Polak–Ribière–Polyak search direction and runs without synchronisation from start to finish. Boundary data are exchanged with MPI one-sided communication (MPI_Put, MPI_Get), so each rank keeps computing while the data move in the background. The global residual is only checked when it changes significantly, and the solver terminates through an asynchronous hierarchical count-down instead of a global sum.
Performance
The aPCG solver was tested on the pressure equation of the DLR combustor case from the exaFOAM project (large-eddy simulation of a jet flame) with 3 and 24 million cells, on LUMI-C (AMD EPYC, 128 cores per node) and MareNostrum 5 (Intel Sapphire Rapids, 112 cores per node).
LUMI
The standard PCG is most cost-effective at 1,024 cores (12.8 ms per pressure solution), aPCG at 2,048 cores (4.1 ms). Comparing these two points, aPCG solves the equation 3.1 times faster at 1.6 times lower cost.
MareNostrum 5
The standard PCG is most cost-effective at 896 cores (14.4 ms per pressure solution), aPCG at 1,792 cores (4.7 ms): again 3.1 times faster at 1.5 times lower cost. The standard PCG stops scaling at 3,584 cores (6.9 ms), where aPCG needs 3.1 ms, 2.2 times less.
Applications
The algorithm is designed for problems where the classic PCG needs only a few iterations. Such matrices arise in every transient CFD code that solves a pressure equation, and in many other simulations that solve similar systems at scale. Because the ranks no longer wait for each other, ranks that have converged locally could help slower partitions (fine-grained load balancing) or go idle to save energy.
Partners
- Wikki GmbH, Wernigerode: coordinator; derivation, implementation and testing in OpenFOAM (Dr. Henrik Rusche, Ranjith Khumar Shanmugasundaram).
- INRIA Paris, Alpines team (joint team with Laboratoire Jacques-Louis Lions, Sorbonne University): enlarged Krylov methods and implementation in FreeFEM (Dr. Frédéric Nataf).
Dissemination
- Inno4scale mini-symposium on linear algebra at scale (online, March 2025), organised by Wikki and INRIA with the innovation studies AceAMG, aCG, CBM4scale, exaSIMPLE, ISOLV-BSE and MG4ML.
Funding
AMCG was carried out as an innovation study within the Inno4scale project, which has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 101118139. The JU receives support from the European Union’s Horizon Europe Programme.