Nek5000/NekRS
Technical Description
Nek5000 and NekRS are highly-efficient and scalable open source incompressible and low Mach flow solvers employing the spectral element method (SEM), a high-order weighted residual technique for spatial discretization that can accurately represent complex geometries. Globally, the SEM is based on decomposition of the domain into E smaller subdomains (elements), which are assumed to be curvilinear hexahedra (bricks) that conform to the domain boundaries. Locally, functions within each element are expanded as Nth order polynomials cast in tensor-product form, which allow differential operators on N3 grid points per element to be evaluated with only O(N4) work and O(N3) storage. The principal advantage of the SEM is that convergence is exponential in N, yielding minimal numerical dispersion and dissipation. Significantly fewer grid points per wavelength are required in order to accurately propagate a signal (or turbulent structure) over extended times in high Reynolds number simulations.
Nek Explainer
If the above technical description was not what you’re looking for, check out the explainer answers to some common questions below.
What can the Nek codes simulate?
Nek5000 and NekRS can do both incompressible and low Mach number compressible flow simulations. This means that they can do simulations where the density of the fluid doesn’t change as it comes in contact with an object or when the density changes due to heat release as is the case for example in combustion. This can be true if the fluid itself is incompressible like a liquid or if the fluid is moving slowly enough that collision with an object doesn’t change its density. This is where the Mach number is important. For flows where an object is moving through gas at speeds lower than about 250 miles per hour (about one third of the speed of sound), the change in density is so small that it can be ignored.
Nek can also simulate coupled fluid and particle flow meaning that it can simulate how a fluid changes the position of particles and how those particles in turn change the fluid. This is the kind of simulation that will be performed in the off-shore wind turbine lighthouse case using a different code, waLBerla.
Outside of what Nek can simulate, it can also use different approaches to simulate flow. It can run direct numerical simulations (DNS) of the fluid flow or use turbulence modelling, like large-eddy simulations (LES), depending on what is most accurate and efficient for a given simulation.
Lastly, Nek can run on many types of supercomputer hardware. Nek5000 is decades old and runs on central processing units (CPUs). NekRS, which is much younger, runs on systems that have both CPUs and graphical processing units (GPUs) and focuses mostly on GPUs.
Overall, this makes the Nek codes Nek5000 and NekRS unusually versatile and capable of even doing things well outside of CEEC activities such as reactive flow (combustion). For CEEC, Nek5000/NekRS will be used to simulate atmospheric boundary layer flows.
What are the trade-offs of using Nek codes?
Despite all the versatility mentioned above, there are still some things that Nek can’t do. As you might have guessed already, Nek isn’t useful for high-speed compressive flows, i.e. at higher Mach numbers. This means, for example, that we couldn’t use Nek to run our first lighthouse case concerning shock buffet for planes traveling near the speed of sound (Mach 1). For this, we’ll be using FLEXI.
However, for the right kinds of simulations, Nek still has advantages over other codes even commercial codes. Because Nek is a high-fidelity (accuracy) high-order code, you can get much more accurate results with fewer data points than with a low-order method as is the case in many commercial codes. So, if you know how accurate you need to be, you can run a simulation with fewer degrees of freedom with Nek, or conversely, you can get more accurate results from the same degrees of freedom. Since the degrees of freedom and number of points generally correlate with how much work the computers need to do in terms of time and energy used, Nek can either save you time and energy while maintaining accuracy or can give you much more accurate results for the same time and energy needed by a low-order code.
Nek also has proven efficiency at scale meaning that it can efficiently divide the computational work among millions of processors in order to complete simulations faster on larger machines. Basically, it is really good at delegating. If you give it more processors, it gets work done faster, and this has been tested on systems as large as Lumi in Finland.
What Could Nek codes do better?
HPC systems are getting bigger and new algorithms and models are being written every day to take better advantage of them. Nek can be improved by adding a wider variety of models to its library. Each new model allows us to tune the mathematics we’re using to the specific type of simulation we’re running in order to strike the best possible balance between accuracy and efficiency. Better models and increased efficiency in turn let Nek tackle more realistic problems.
What will CEEC aim to improve about the Nek codes?
CEEC work will focus on improving LES and wall modelling in Nek in order to maximize the accuracy and efficiency of simulating atmospheric boundary layer fluid flows. These improvements are what will enable the atmospheric boundary layer lighthouse case.
The second focus of CEEC work will be preparing NekRS for new types of GPUs, which is called porting. The first European Exascale machine, Jupiter in Julich, will use a new generation of Nvidia GPUs that will require some changes from NekRS in order to run as efficiently as possible. NekRS also needs to be ported to Intel GPUs which are some of the newest and fastest GPUs being increasingly used.
Why are these improvements a challenge?
Porting codes to new hardware is always a challenge because of how complex the codes are. The new hardware also is always demanding more from codes than its predecessors. As an example, new GPUs might expect at least four million points of data per GPU to maintain a reasonable parallel efficiency of say 80%, where older GPUs could do the same with two or three million. It is important that the simulation can be broken down into even smaller parts in order to make efficient use of new hardware.
As anyone who has ever tried to micromanage a large team can attest, breaking tasks down into the smallest possible steps without wasting time coordinating all the team members is a difficult and delicate balance. Part of porting codes to new hardware means trying many different “management” plans for running the simulation and testing them until we find a plan that lets us use at least eighty percent of the new hardware’s potential computational power.
Optimizing LES and wall models entails a similar process of testing the model with the Nek code base and ensuring that it both runs efficiently and accurately enough to provide useful results.
Although the process of porting and optimizing codes or models isn’t new, and despite all the previous experience brought to bear by our partners, the exact steps for finding the best possible solution are always discovered through iterative testing. Furthermore, testing the code on a small part of a supercomputer is very different from testing it on the entire supercomputer. Something that’s a minor problem when using a fraction of the total number of processors necessarily has the potential to be a major problem when applied to all the processors of a supercomputer. Since using an entire supercomputer to run a test is a huge use of time and energy, conducting tests as quickly and effectively as possible is always the challenge.
What real-world benefits will improved Nek codes help achieve?
Having more accurate LES and wall modelling and more efficient code will allow us to simulate more realistic problems. These simulations can then be used to create more accurate low-order models for other codes to use. The simulation data can also be used to validate experimental data or data produced while optimizing other simulation codes. So, in addition to what we learn from more realistic simulations themselves, the data from those simulations can be used to validate data produced by other experiments and codes, thereby advancing our overall scientific and societal understanding of turbulence and fluid dynamics.