SCALE-TRACK was an innovation study within the EuroHPC project Inno4scale, carried out from February 2024 to March 2025 by Wikki GmbH and the Leibniz Institute for Tropospheric Research (TROPOS).
Highlights
- A highly parallel, efficient, asynchronous multi-GPU algorithm for Lagrangian particle tracking.
- Implemented in Julia with an interface to OpenFOAM, adaptable to other CFD solvers for the Eulerian phase.
- Three solvers based on OpenFOAM solvers: two with the SCALE-TRACK algorithm, one for incompressible turbulent flows and one for compressible turbulent flows with heat transfer and water vapour transport. The third uses OpenFOAM’s own particle tracking and serves as the baseline for comparison.
- A collection of test cases to validate that SCALE-TRACK works correctly.
- Very good weak scaling on MareNostrum 5 (NVIDIA H100 GPUs) and strong scaling up to 256 GPUs on LUMI (AMD MI250X GPUs), see Performance.
- A test simulation of an 81 m³ cloud chamber with up to 256 billion cloud droplets.
The challenge
Euler-Lagrange simulations are limited by the number of particles that can be processed with reasonable effort, in both computing time and hardware. The reason is that the Lagrangian part of the simulation is hard to parallelise efficiently: CPUs are limited by the number of parallel threads, GPUs mainly by their memory. Although powerful HPC systems are available, a literature survey found no simulation with more than one billion particles.
For many applications, such as large cloud chambers or real clouds, handling substantially more particles would give a much more complete picture of the underlying mechanisms and the complex interactions in these systems.
The solution
SCALE-TRACK processes particles in chunks using a structure-of-arrays layout, and computes the Eulerian and the Lagrangian phase asynchronously: the flow on CPUs, the particles on GPUs. This greatly reduces waiting times and synchronisation barriers. A compact representation of the particle properties drastically reduces the memory needed on the GPUs. The Lagrangian phase is decomposed independently of the Eulerian one, so each phase is split according to its own work.
The Lagrangian algorithm is written in Julia and makes extensive use of its built-in support for NVIDIA and AMD GPUs and for parallelism with coroutines and threads. The Eulerian phase is computed with an OpenFOAM-based solver.
Performance
Strong scaling on the GPU partition of LUMI (nodes with one 64-core AMD EPYC 7A53 CPU and four AMD MI250X GPUs), for a cloud chamber with 24×10⁶ cells:
- With 8×10⁹ particles, the Lagrangian phase reaches a parallel efficiency of 1.03 at 256 GPUs.
- With 8×10⁸ particles, the efficiency is 0.94 at 64 GPUs and 0.82 at 256 GPUs: efficiency is governed by the number of particles per GPU rather than by the total number.
- The particles can also be tracked on CPU cores. On 12 to 96 nodes with two 64-core AMD EPYC 7763 CPUs each, the parallel efficiency of the Lagrangian phase stays between 0.95 and 1.00, and the Eulerian phase is hidden behind the tracking.
Application: clouds
As the application with billions of particles we chose cloud chamber simulations. Here, each particle is a single cloud droplet that grows and shrinks depending on the conditions it experiences. Such simulations are used to design cloud chambers and to design and interpret experiments in them. They can also provide parameterisations for large-scale cloud simulations and for weather and climate models.
Benefits
SCALE-TRACK is useful wherever a dispersed phase consists of a large number of individual gaseous, liquid or solid particles. Until now, the number of droplets that Euler-Lagrange simulations could reasonably handle was in the order of one billion. With SCALE-TRACK we demonstrated excellent scalability up to 256 billion droplets. This enables high-fidelity simulations of cloud chambers and of parts of real clouds, and could ultimately improve weather and climate models through better parameterisations.
In industry, applications include fuel injection in internal combustion engines, where a large number of droplets is sprayed, and spray drying in the chemical, pharmaceutical and food industries. All of these processes can be studied with SCALE-TRACK, giving new insights and showing where they can be improved, so that they become more efficient and use fewer resources.
Partners
- Wikki GmbH, Wernigerode: HPC expertise, algorithm development and programming (Dr. Henrik Rusche, Dr. Sergey Lesnik).
- Leibniz Institute for Tropospheric Research (TROPOS), Leipzig: test and validation cases and the proof of concept (Dr. Silvio Schmalfuß).
Open source and publication
SCALE-TRACK is free software under the GPL-3.0-or-later licence. The solvers, the Julia implementation, tutorial cases and the validation suite are available on GitHub.
S. Schmalfuß, S. Lesnik, H. Rusche, D. Niedermeier: SCALE-TRACK: Asynchronous Euler-Lagrange particle tracking on heterogeneous computing architecture. Journal of Computational Physics 565, 115335, 2026. doi:10.1016/j.jcp.2026.115335 (preprint: arXiv:2603.26691). The graphical abstract and Figs. 1, 2 and 6 of the paper are used on this page under the CC BY 4.0 licence.
Funding
SCALE-TRACK was carried out as an innovation study within the Inno4scale project, which has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 101118139.