waLBerla – widely applicable Lattice Boltzmann (LB) from Erlangen
Technical Description
waLBerla is a modern open source multi-physics simulation software framework with a focus on CFD applications. The main unique feature of waLBerla is its uncompromising focus on large-scale simulations and scalability. It supports the massive parallelism of current peta- and future exascale supercomputers with a framework of carefully designed distributed data structures. Automated testing ensures the correctness of the functionality across a wide range of target hardware and software environments such that it is well-suited for robust further developments. Its current version also features adaptive techniques. Moreover, it includes functionalities for load balancing, for checkpoint-restart, and even automatic resilience techniques that may become essential on future extreme-scale systems. waLBerla contains efficient, hardware specific compute kernels to achieve optimal performance on most common supercomputing CPU and GPU architectures.
waLBerla Explainer
If the above technical description was not what you’re looking for, check out the explainer answers to some common questions below.
What can waLBerla simulate?
Unlike all the other CEEC codes, waLBerla does not directly solve the Navier-Stokes equations. Instead, it uses the lattice Boltzmann method to produce results equivalent to solving the Navier-Stokes equations. Generally speaking, this means that instead of calculating the velocity of the fluid within a grid (building block) of the simulation, the lattice Boltzmann method calculates the velocity distribution of fluid molecules in that grid. Even though the results get you to the same place in the end, using a different method has implications for how the code works and how it can split the work into smaller pieces to delegate across processors.
waLBerla also can do coupled simulations of fluid and particle flow, similar to the Nek codes and Alya. It can simulate how fluid flows around particles, how the fluid moves the particles, and how the particles move the fluid in turn.
In terms of flow speed and compressibility, waLBerla can simulate weakly compressible fluid flow. This is similar to Nek and Neko in that weakly compressible flows do have low Mach numbers. However, weakly compressible flows also have low Reynolds numbers, indicating a low potential for turbulence, as well as low Froude and Rossby numbers that describe the ratio of momentum force to gravitational force and the ratio of inertial force to Coriolis force, respectively. Weakly compressible flows are most often seen in geophysical applications (atmospheric and oceanic flows), which is why CEEC will be using waLBerla for simulating erosion around off-shore wind turbine foundations.
What are the trade-offs of using waLBerla?
One of the benefits of using waLBerla is its portability to different hardware architectures. However, the way it accomplishes this is different from some other codes. Unlike Neko that uses a Navier-Stokes based numerical method and has kernels for each hardware built in, waLBerla is able to use many different versions of the lattice Boltzmann method. Hence, it would need a kernel for each lattice Boltzmann method and hardware combination. For this reason, waLBerla uses automated code generation to write the optimized kernel for each method and hardware combination as needed instead of storing a huge number of superfluous kernels. This provides the user with the most optimized kernel for their specific simulation and HPC system combination. The drawback of this approach is that the software pipeline becomes more complex both for waLBerla users and developers.
What could waLBerla do better?
It could improve in terms of usability. Especially compared to commercial software with a graphical user interface, waLBerla is more complicated for the user and requires more knowledge of HPC systems and the command line interface, which isn’t standard for industrial engineers, for example.
waLBerla also will need performance improvements to be able to run on the largest European HPC systems.
What will CEEC aim to improve about waLBerla?
CEEC will focus on making performance improvements to waLBerla, specifically for the coupled fluid and particle flow simulations needed for the off-shore wind turbine foundation lighthouse case.
Why is this improvement a challenge?
As with most of the work being done in CEEC, the challenge is the complexity. In the case of waLBerla, each hardware and lattice Boltzmann method combination will need to be analyzed for efficiency and performance bottlenecks, a variety of possible fixes will need to be written and tested, and then all the best improvements will need to be worked into both the simulation methods and the automatic code generation process that builds each custom kernel.
This process will need to be done for both the fluid flow and for the particle flow simulation modules, which have very different algorithmic structures and thus very different hardware needs.
Lastly, the best performance optimizations for each module will then also need to be tested when coupled (functionally running two interdependent but distinct simulations), which introduces a whole new layer of software communication and possible bottlenecks that will need to be analyzed and then iteratively corrected and tested.
What real-world benefits will an improved waLBerla help achieve?
The most direct real-world benefit of an improved waLBerla come from the off-shore wind turbine foundation lighthouse case. Suction installation of off-shore wind turbine foundations have both environmental and practical benefits in terms of speed, reversibility, and no pylon driving noise. Making suction installation more reliable will make off-shore wind energy easier for both humans on land and all life off-shore.
Beyond the lighthouse case, improved waLBerla will be more capable of helping solve bigger and more realistic coupled fluid and particle flow problems with any number of energy, sustainability, and practicality implications, just like all other CEEC codes.