Session: 3.2 - Computational Turbulent Combustion

Paper Number: 158539

158539 - Performance Modeling and Scaling of PETSc Based Direct Numerical Simulations for Hybrid Rocket Boundary Layers 

Abstract: 

This study presents a parallel scaling analysis of Direct Numerical Simulations for a reacting boundary layer. The DNS is based on a PETSc-based finite volume formulation and the application is a slab burner experiment.  The fuel is Polymethyl methacrylate (PMMA) with pure O₂ as the oxidizer.  A detailed kinetics mechanism is used to describe the pyrolysis and oxidation of MMA and consists of 113 species and 660 reactions. A ray-tracing based radiation solver, using a ray decomposition method, designed for distributed memory applications, is implemented to model radiation heat transfer. The simulations are validated using two-color pyrometry data, which provide experimental measurements of temperature and soot volume fractions for the slab burner. The simulation results closely match the experimental data, supporting the accuracy of the model. Parallel scalability is evaluated for 2D and 3D simulations with fully coupled flow, chemistry, and radiation heat transfer. Additionally, the overall scalability of the slab burner is evaluated with GPU accelerated chemical kinetics evaluations. The scalability of the flow solver is assessed separately for the detailed convective and diffusive flux calculators. Finite rate chemistry is evaluated with Zero-RK, a chemical kinetic calculator allowing for GPU support. Scalability is assessed for both simple, single-physics cases as well as combined physics for the slab burner configuration. Weak scaling studies confirm the chemistry module scales well, however, the flow process experiences some performance decline due to domain partitioning and memory bandwidth limitations. Strong scaling tests conducted on up to 15,000 computational ranks reveal robust performance scalability for up to 200 flow cells per rank. The analysis highlights challenges in the scalability of large reactive CFD simulations, particularly in flow processes, due to computational constraints such as increased memory footprint and bandwidth limitations from partitioning. For large 3D simulations, chemistry dominates the computational cost, accounting for up to 40% of the total time, while flow processes contribute approximately 35%. Radiative gain evaluations are performed at reduced frequencies, due to the slower evolution of temperature and soot fields, which keeps the computational contribution of the radiation solver below 10%. GPU-accelerated chemistry shows significant promise for the simulations. For certain simulations, the chemistry cost can be reduced to about 10% of the total cost for GPU accelerated chemistry evaluations for workloads exceeding 30,000 cells per GPU. However, diminishing returns are observed for smaller workloads due to communication overhead between CPUs and GPUs. This study offers realizable run configurations for 2D, 3D, and GPU-accelerated cases, aiming for efficient resource utilization and reduced overall time to solution. The findings provide valuable insights into optimizing the performance of reactive flow solvers like ABLATE, particularly for hybrid rocket combustion.

Presenting Author: Kolos Retfalvi University at Buffalo

Presenting Author Biography: