High performance parallel computing of flows in complex geometries

Laurent Y.M. Gicquel; N. Gourdain; J.-F. Boussuge; H. Deniau; G. Staffelbach; P. Wolf; Thierry Poinsot

doi:10.1016/j.crme.2010.11.006

High performance parallel computing of flows in complex geometries

Laurent Y.M. Gicquel ¹ ; N. Gourdain ¹; J.-F. Boussuge ¹; H. Deniau ¹; G. Staffelbach ¹; P. Wolf ¹; Thierry Poinsot ²

¹ CERFACS, 42, avenue Georges-Coriolis, 31057 Toulouse cedex , France
² IMFT, 1, allée du Professeur Camille-Soula, 31400 Toulouse, France

Comptes Rendus. Mécanique, High Performance Computing, Volume 339 (2011) no. 2-3, pp. 104-124.

Abstract
Résumé

Efficient numerical tools taking advantage of the ever increasing power of high-performance computers, become key elements in the fields of energy supply and transportation, not only from a purely scientific point of view, but also at the design stage in industry. Indeed, flow phenomena that occur in or around the industrial applications such as gas turbines or aircraft are still not mastered. In fact, most Computational Fluid Dynamics (CFD) predictions produced today focus on reduced or simplified versions of the real systems and are usually solved with a steady state assumption. This article shows how recent developments of CFD codes and parallel computer architectures can help overcoming this barrier. With this new environment, new scientific and technological challenges can be addressed provided that thousands of computing cores are efficiently used in parallel. Strategies of modern flow solvers are discussed with particular emphases on mesh-partitioning, load balancing and communication. These concepts are used in two CFD codes developed by CERFACS: a multi-block structured code dedicated to aircrafts and turbo-machinery as well as an unstructured code for gas turbine flow predictions. Leading edge computations obtained with these high-end massively parallel CFD codes are illustrated and discussed in the context of aircrafts, turbo-machinery and gas turbine applications. Finally, future developments of CFD and high-end computers are proposed to provide leading edge tools and end applications with strong industrial implications at the design stage of the next generation of aircraft and gas turbines.

L'utilisation et l'accès à des codes de calcul numérique tirant avantage de la puissance croissante des calculateurs hautes-performances, est devenu un élément clef dans les domaines de la production d'énergie et des transports. Ces outils sont non seulement critiques d'un point de vue scientifique mais aussi pour les concepteurs de l'industrie. En effet, les écoulements autour des produits industriels de l'énergie ou de l'aéronautique sont si compliqués qu'ils nécessitent souvent l'utilisation de modèles simplifiés avec l'hypothèse de stationnarité. Ce document présente comment les dévelopements récents des codes de calcul et des calculateurs massivement parallèles peuvent aider à repousser ces limites. Dans cet environement particulier, de nouveaux challenges technologiques et scientifiques peuvent être abordés en utilisant efficacement des milliers de coeurs de calcul en parallèle. Le parallèlisme de ces solveurs modernes est décrit avec un regard particulier sur le découpage de maillage, d'équilibrage de charge et de communication. Deux exemples issus des travaux du CERFACS sont utilisés pour illustrer ces concepts : un code de calcul multi-blocs structuré dédié à la simulation des écoulements avions et turbo-machines ainsi qu'un code non structuré ciblant les écoulements dans les turbines à gaz. Pour finir, des axes de recherche et développements de ces codes et calculateurs sont énnoncées afin d'identifier les utilisations futures possibles et en accord avec les besoins industriels apparaissant lors de la définition des futures concepts d'avions et turbines à gaz.

Published online: 2011-01-08

DOI: 10.1016/j.crme.2010.11.006

Keywords: Computer science, Parallel computing, Computational Fluid Dynamics
Mots-clés : Informatique, Calcul parallèle, Dynamique des fluides numérique

Author's affiliations:

Laurent Y.M. Gicquel ¹; N. Gourdain ¹; J.-F. Boussuge ¹; H. Deniau ¹; G. Staffelbach ¹; P. Wolf ¹; Thierry Poinsot ²

¹ CERFACS, 42, avenue Georges-Coriolis, 31057 Toulouse cedex , France
² IMFT, 1, allée du Professeur Camille-Soula, 31400 Toulouse, France

@article{CRMECA_2011__339_2-3_104_0,
     author = {Laurent Y.M. Gicquel and N. Gourdain and J.-F. Boussuge and H. Deniau and G. Staffelbach and P. Wolf and Thierry Poinsot},
     title = {High performance parallel computing of flows in complex geometries},
     journal = {Comptes Rendus. M\'ecanique},
     pages = {104--124},
     publisher = {Elsevier},
     volume = {339},
     number = {2-3},
     year = {2011},
     doi = {10.1016/j.crme.2010.11.006},
     language = {en},
}

TY  - JOUR
AU  - Laurent Y.M. Gicquel
AU  - N. Gourdain
AU  - J.-F. Boussuge
AU  - H. Deniau
AU  - G. Staffelbach
AU  - P. Wolf
AU  - Thierry Poinsot
TI  - High performance parallel computing of flows in complex geometries
JO  - Comptes Rendus. Mécanique
PY  - 2011
SP  - 104
EP  - 124
VL  - 339
IS  - 2-3
PB  - Elsevier
DO  - 10.1016/j.crme.2010.11.006
LA  - en
ID  - CRMECA_2011__339_2-3_104_0
ER  -

%0 Journal Article
%A Laurent Y.M. Gicquel
%A N. Gourdain
%A J.-F. Boussuge
%A H. Deniau
%A G. Staffelbach
%A P. Wolf
%A Thierry Poinsot
%T High performance parallel computing of flows in complex geometries
%J Comptes Rendus. Mécanique
%D 2011
%P 104-124
%V 339
%N 2-3
%I Elsevier
%R 10.1016/j.crme.2010.11.006
%G en
%F CRMECA_2011__339_2-3_104_0

Laurent Y.M. Gicquel; N. Gourdain; J.-F. Boussuge; H. Deniau; G. Staffelbach; P. Wolf; Thierry Poinsot. High performance parallel computing of flows in complex geometries. Comptes Rendus. Mécanique, High Performance Computing, Volume 339 (2011) no. 2-3, pp. 104-124. doi : 10.1016/j.crme.2010.11.006. https://comptes-rendus.academie-sciences.fr/mecanique/articles/10.1016/j.crme.2010.11.006/

References
Cited by

[1] D.E. Keyes, D.K. Kaushik, B.F. Smith, Prospects for CFD on petaflops systems, Technical Report TRT-97-73, Institute for Computer Applications in Science and, Engineering, 1997.

[2] N. Gourdain; L.Y.M. Gicquel; M. Montagnac; O. Vermorel; M. Gazaix; G. Staffelbach; M. Garcia; J.-F. Boussuge; T. Poinsot High performance parallel computing of flows in complex geometries – Part 1: Methods, Comput. Sci. Discov., Volume 2 ( November 2009 ), p. 015003 (26 pp.)

[3] N. Gourdain; L.Y.M. Gicquel; G. Staffelbach; O. Vermorel; F. Duchaine; J.-F. Boussuge; T. Poinsot High performance parallel computing of flows in complex geometries – Part 2: Applications, Comput. Sci. Discov., Volume 2 ( November 2009 ), p. 015004 (28 pp.)

[4] M.J. Flynn Some computer organizations and their effectiveness, IEEE Trans. Comput., Volume C-21 (1972) no. 9, pp. 948-960

[5] L. Cambier, M. Gazaix, elsA: an efficient object-oriented solution to CFD complexity, in: 40th AIAA Aerospace Science Meeting and Exhibit, 2002.

[6] L. Cambier, J.P. Veuillot, Status of the elsA CFD software for flow simulation and multidisciplinary applications, in: 46th AIAA Aerospace Science Meeting and Exhibit, AIAA paper 664, 2008.

[7] L. Davidson; S.H. Peng Hybrid LES-RANS modeling: a one-equation SGS model combined with a $k – ω$ model for predicting recirculating flows, Internat. J. Numer. Methods Fluids, Volume 43 (2003) no. 9, pp. 1003-1018

[8] J. Clique, R. Houdeville, D. Arnal, Application of laminar-turbulent transition criteria in Navier–Stokes computations, in: 46th AIAA Aerospace Science Meeting and Exhibit, Reno, USA, 2007.

[9] S. Deck Numerical simulation of transonic buffet over a supercritical airfoil, AIAA J., Volume 43 (2005), pp. 1556-1566

[10] P.R. Spalart, W.-H. Jou, M. Stretlets, S.R. Allmaras, Comments on the feasibility of LES for wings and on the hybrid RANS/LES approach, advances in DNS/LES, in: Proceedings of the First AFSOR International Conference on DNS/LES, 1997.

[11] J. Smagorinsky General circulation experiments with the primitive equations: 1. The basic experiment, Mon. Weather Rev., Volume 91 (1963), pp. 99-164

[12] G. Fillola, M.-C. Le Pape, M. Montagnac, Numerical simulations around wing control surfaces, in: 24th ICAS Meeting, Yokohama, Japan, 2004.

[13] R. Meakin The chimera method of simulation for unsteady three-dimensional viscous flows, Comput. Fluid Dynam. Rev. (1995), pp. 70-86

[14] P.L. Roe Approximate Riemann solvers, parameter vectors and difference schemes, J. Comput. Phys., Volume 43 (1981), pp. 357-372

[15] M.S. Liou A sequel to AUSM: AUSM+, J. Comput. Phys., Volume 129 (1996), pp. 364-382

[16] S. Yoon, A. Jameson, An LU-SSOR scheme for the Euler and Navier–Stokes equations, in: AIAA 25th Aerospace Sciences Meeting,AIAA paper 0600, 1987.

[17] A. Jameson, Time dependent calculations using multigrid, with applications to unsteady flows past airfoils and wings, in: AIAA Computational Fluid Dynamics Conference, 1991.

[18] M. García, Interpolates parallele de solutions, Projet ANR CAMPAS, Livrable L3 CR/CFD/07/149, CERFACS, Toulouse, France, 2007.

[19] P. Moin; S.V. Apte Large-eddy simulation of realistic gas turbine combustors, Am. Inst. Aeronaut. Astronaut. J., Volume 44 (2006) no. 4, pp. 698-708

[20] P.D. Lax; B. Wendroff Systems of conservation laws, Comm. Pure Appl. Math., Volume 13 (1960), pp. 217-237

[21] J. Donea Taylor–Galerkin method for convective transport problems, Internat. J. Numer. Methods Fluids, Volume 20 (1984) no. 1, pp. 101-119

[22] O. Colin; M. Rudgyard Development of high-order Taylor–Galerkin schemes for unsteady calculations, J. Comput. Phys., Volume 162 (2000) no. 2, pp. 338-371

[23] C.W. Hirt; A.A. Amsden; J.L. Cook An arbitrary Lagrangian–Eulerian computing method for all flow speeds, J. Comput. Phys., Volume 131 (1974) no. 4, pp. 371-385

[24] G. Karypis; V. Kumar A fast and high quality multilevel scheme for partitioning irregular graphs, SIAM J. Sci. Comput., Volume 20 (1998) no. 1, pp. 359-392

[25] M.J. Berger; S.H. Bokhari A partitioning strategy for nonuniform problems on multiprocessors, IEEE Trans. Comput., Volume C-36 (1987) no. 5, pp. 570-580

[26] A. Ytterström A tool partitioning structured multiblock meshes for parallel computational mechanics, Int. J. High Perform. Comput. Appl., Volume 11 (1997), pp. 336-343

[27] R.D. William Performance of dynamic load balancing algorithms for unstructured mesh calculations, Concurrency, Pract. Exp., Volume 3 (1991), pp. 451-481

[28] H.D. Simon Partitioning of unstructured problems for parallel processing, Comput. Systems Engrg., Volume 2 (1991) no. 3, pp. 135-148

[29] J.G. Lewis The Gibbs–Poole–Stockmeyer and Gibbs–King algorithms for reordering sparse matrices, ACM Trans. Math. Software, Volume 8 (1982) no. 2, pp. 190-194

[30] G. Karypis, V. Kumar, Multilevel algorithms for multi-constraint graph partitioning, Technical Report 98-019, University of Minnesota, Department of Computer Science/Army HPC Research Center, 1998.

[31] B. Hendrickson, R. Leland, A multilevel algorithm for partitioning graphs, Technical Report SAND93-1301, Sandia National Laboratories, Albuquerque, NM, 1993.

[32] J.-M. Senoner; M. García; S. Mendez; G. Staffelbach; O. Vermorel; T. Poinsot Growth of rounding errors and repetitivity of Large–Eddy simulations, AIAA J., Volume 46 (2008) no. 7, pp. 1773-1781

[33] M. García, Analysis of precision differences observed for the AVBP code, Technical Report TR/CFD/03/84, CERFACS, Toulouse, France, 2003.

[34] C. Rossow, N. Kroll, High performance computing serves aerospace engineering: Opportunities for next generation product development, in: 46th AIAA Aerospace Science Meeting and Exhibit, Reno, USA, 2008.

[35] J. Delbove, Unsteady simulations for flutter prediction, in: Proceedings of the Third International Conference on CFD – ICCFD3, 2006, pp. 205–210.

[36] F. Sicot; G. Puigt; M. Montagnac Block–Jacobi implicit algorithms for the time spectral method, AIAA J., Volume 46 (2008), pp. 3080-3089

[37] A. Gopinath, E. van der Weide, J.J. Alonso, A. Jameson, K. Ekici, K.C. Hall, Three-dimensional unsteady multi-stage turbomachinery simulations using the harmonic balance technique, in: 45th AIAA Aerospace Science Meeting and Exhibit, Reno, USA, 2007.

[38] V. Brunei, S. Deck, Zonal-detached eddy simulation of transonic buffet on a civil aircraft type configuration, in: 46th AIAA Aerospace Science Meeting and Exhibit, Reno, USA, 2008.

[39] P.R. Spalart, S.R. Allmaras, A one equation turbulence model for aerodynamic flows, in: 30th AIAA Aerospace Science Meeting and Exhibit, AIAA paper 92-0439, Reno, USA, 1992.

[40] A.F. Pouangué; H. Deniau; F. Sicot; P. Sagaut Curvilinear finite-volume schemes using high-order compact interpolation, J. Comput. Phys., Volume 229 (2010), pp. 5090-5122

[41] M.D. Hathaway, G. Herrick, J. Chen, R. Webster, Time accurate unsteady simulation of the stall inception process in the compression system of a US army helicopter gas turbine engine, in: DoD Users Group Conference, 2004, pp. 182–193.

[42] N. Gourdain; S. Burguburu; F. Leboeuf; G.-J. Michon Simulation of rotating stall in a whole stage of an axial compressor, J. Comput. Fluids, Volume 39 (2010) no. 9, pp. 1644-1655

[43] E. van der Weide, G. Kalitzin, J. Schluter, J.J. Alonso, Unsteady turbomachinery computations using massively parallel platforms, in: 44th AIAA Aerospace Sciences Meeting and Exhibit, 2006.

[44] N. Gourdain, X. Ottavy, A. Vouillarmet, Experimental and numerical investigation of unsteady flows in a high speed three stages compressor, in: 8th European Turbomachinery Conference, 2008.

[45] N. Gourdain; M. Montagnac; F. Wlassow; M. Gazaix High performance computing to simulate large scale industrial flows in multistage compressors, Int. J. High Perform. Comput. Appl., Volume 24 (2010) no. 4, pp. 429-443

[46] C. Sieverding; H. Richard; J.-M. Desse Turbine blade trailing edge flow characteristics at high subsonic outlet Mach number, Trans. ASME, Volume 125 (2003), pp. 298-309

[47] C. Sieverding; D. Ottolia; C. Bagnera; A. Comadoro; J.-F. de Brouckaert; J.-M. Desse Unsteady turbine blade wake characteristics, J. Turbomach., Volume 126 (2004), pp. 551-559

[48] T. Leonard, F. Duchaine, N. Gourdain, L.Y.M. Gicquel, Steady, unsteady Reynolds averaged Navier–Stokes and large eddy simulations of a turbine blade at high subsonic outlet Mach number, in: ASME Turbo Expo, 2010.

[49] T. Arts, M. Lambert de Rouvroit, A.W. Rutherford, Aero-thermal investigation of a highly loaded transonic linear turbine guide vane cascade, Technical Note 174, Von Karman Institute, 1990.

[50] T. Poinsot; D. Veynante Theoretical and Numerical Combustion, R.T. Edwards, 2005

[51] T. Poinsot; S. Lele Boundary conditions for direct simulations of compressible viscous flows, J. Comput. Phys., Volume 101 (1992) no. 1, pp. 104-129

[52] A. Roux; L.Y.M. Gicquel; S. Reichstadt; N. Bertier; G. Staffelbach; F. Vuillot; T. Poinsot Analysis of unsteady reacting flows and impact of chemistry description in large eddy simulations of side-dump ramjet combustors, Combust. & Flame, Volume 157 (2010), pp. 176-191

[53] A. Roux; S. Reichstadt; N. Bertier; L.Y.M. Gicquel; F. Vuillot; T. Poinsot Comparison of numerical methods and combustion models for LES of a ramjet, C. R. Mecanique, Volume 337 (2009) no. 6–7, pp. 352-361

[54] G. Boudier; N. Lamarque; G. Staffelbach; L.Y.M. Gicquel; T. Poinsot Thermo-acoustic stability of a helicopter gas turbine combustor using large-eddy simulations, Int. J. Aeroacoust., Volume 8 (2009) no. 1, pp. 69-94

[55] P. Wolf; G. Staffelbach; A. Roux; L. Gicquel; T. Poinsot; V. Moureau Massively parallel LES of azimuthal thermo-acoustic instabilities in annular gas turbines, C. R. Mecanique, Volume 337 (2009) no. 6–7, pp. 385-394

[56] G. Boudier; L.Y.M. Gicquel; T. Poinsot; D. Bissières; C. Bérat Effect of mesh resolution on large eddy simulation of reacting flows in complex geometry combustors, Combust. Flame, Volume 155 (2008) no. 1–2, pp. 196-214

[57] A. Roux; L.Y.M. Gicquel; Y. Sommerer; T.J. Poinsot Large eddy simulation of mean and oscillating flow in a side-dump ramjet combustor, Combust. Flame, Volume 152 (2007) no. 1–2, pp. 154-176

[58] C. Prière; L.Y.M. Gicquel; P. Gajan; A. Strzelecki; T. Poinsot; C. Bérat Experimental and numerical studies of dilution systems for low emission combustors, Am. Inst. Aeronaut. Astronaut. J., Volume 43 (2005) no. 8, pp. 1753-1766

[59] C. Prière; L.Y.M. Gicquel; A. Kaufmann; W. Krebs; T. Poinsot LES of mixing enhancement: LES predictions of mixing enhancement for jets in cross-flows, J. Turb., Volume 5 (2004), pp. 1-30

[60] G. Staffelbach; L.Y.M. Gicquel; G. Boudier; T. Poinsot Large eddy simulation of self-excited azimuthal modes in annular combustors, Proc. Combust. Inst., Volume 32 (2009), pp. 2909-2916

[61] B. Schuermans, C. Paschereit, P. Monkiewitz, Non-linear combustion instabilities in annular gas-turbine combustors, AIAA paper 2006-0549, 2006.

Cited by Sources:

Comments - Policy