Fluid–solid coupling on a cluster of GPU graphics cards for seismic wave propagation

Dimitri Komatitsch

doi:10.1016/j.crme.2010.11.007

Fluid–solid coupling on a cluster of GPU graphics cards for seismic wave propagation

Dimitri Komatitsch ^1,²

¹ Université de Pau et des Pays de l'Adour, CNRS & INRIA Magique-3D, Laboratoire de Modélisation et d'Imagerie en Géosciences UMR 5212, Avenue de l'Université, 64013 Pau cedex, France
² Institut universitaire de France, 103 boulevard Saint-Michel, 75005 Paris, France

Comptes Rendus. Mécanique, High Performance Computing, Volume 339 (2011) no. 2-3, pp. 125-135.

Abstract
Résumé

We develop a hybrid multiGPUs and CPUs version of an algorithm to model seismic wave propagation based on the spectral-element method in the case of models of the Earth containing both fluid and solid layers. Thanks to the overlapping of communications between processing nodes on the computer with calculation by means of non-blocking message passing, we obtain excellent weak scalability of this finite-element code on a cluster of 192 GPUs and speedup factors of more than one order of magnitude compared to the same code run on a cluster of traditional CPUs. This enables us to show a new geophysical phenomenon concerning wave propagation of diffracted shear waves in a layer called D″ located at the base of the Earth's mantle, namely that in this layer the transverse and radial components of these waves can undergo a relative shift even in an isotropic Earth model, whereas this observation in real seismological data was interpreted until now as an indication of the presence of anisotropy in this layer.

Nous développons une version hybride multiGPUs et CPUs d'un algorithme de modélisation de la propagation des ondes sismiques fondé sur la méthode des éléments spectraux dans le cas de modèles de terre présentant à la fois des couches fluides et des couches solides. Grâce au recouvrement des communications entre noeuds informatiques par du calcul au moyen de passage de messages non bloquants nous obtenons un excellent passage à l'échelle faible ( « weak scalability ») de ce code d'éléments finis d'ordre élevé sur un réseau ( « cluster ») de 192 GPUs et des facteurs d'accélération de plus d'un ordre de grandeur par rapport au même code exécuté sur un cluster de CPUs classique. Ceci nous permet de démontrer un phénomène géophysique nouveau concernant la propagation des ondes de cisaillement diffractées dans une couche appelée D″ située à la base du manteau terrestre, à savoir que dans cette couche les composantes transverse et radiale de ces ondes peuvent subir un décalage relatif y compris dans un modèle de terre isotrope, alors que cette observation dans des données sismologiques réelles était jusqu'à présent interprétée comme un signe de la présence d'anisotropie dans cette couche.

Published online: 2010-12-30

DOI: 10.1016/j.crme.2010.11.007

Keywords: Computer science, Numerical modeling, Finite elements, Seismic waves
Mots-clés : Informatique, Modélisation numérique, Éléments finis, Ondes sismiques

Author's affiliations:

Dimitri Komatitsch ^{1, 2}

¹ Université de Pau et des Pays de l'Adour, CNRS & INRIA Magique-3D, Laboratoire de Modélisation et d'Imagerie en Géosciences UMR 5212, Avenue de l'Université, 64013 Pau cedex, France
² Institut universitaire de France, 103 boulevard Saint-Michel, 75005 Paris, France

@article{CRMECA_2011__339_2-3_125_0,
     author = {Dimitri Komatitsch},
     title = {Fluid{\textendash}solid coupling on a cluster of {GPU} graphics cards for seismic wave propagation},
     journal = {Comptes Rendus. M\'ecanique},
     pages = {125--135},
     publisher = {Elsevier},
     volume = {339},
     number = {2-3},
     year = {2011},
     doi = {10.1016/j.crme.2010.11.007},
     language = {en},
}

TY  - JOUR
AU  - Dimitri Komatitsch
TI  - Fluid–solid coupling on a cluster of GPU graphics cards for seismic wave propagation
JO  - Comptes Rendus. Mécanique
PY  - 2011
SP  - 125
EP  - 135
VL  - 339
IS  - 2-3
PB  - Elsevier
DO  - 10.1016/j.crme.2010.11.007
LA  - en
ID  - CRMECA_2011__339_2-3_125_0
ER  -

%0 Journal Article
%A Dimitri Komatitsch
%T Fluid–solid coupling on a cluster of GPU graphics cards for seismic wave propagation
%J Comptes Rendus. Mécanique
%D 2011
%P 125-135
%V 339
%N 2-3
%I Elsevier
%R 10.1016/j.crme.2010.11.007
%G en
%F CRMECA_2011__339_2-3_125_0

Dimitri Komatitsch. Fluid–solid coupling on a cluster of GPU graphics cards for seismic wave propagation. Comptes Rendus. Mécanique, High Performance Computing, Volume 339 (2011) no. 2-3, pp. 125-135. doi : 10.1016/j.crme.2010.11.007. https://comptes-rendus.academie-sciences.fr/mecanique/articles/10.1016/j.crme.2010.11.007/

References
Cited by

[1] M.E. Wysession; T. Lay; J. Revenaugh; Q. Williams; E.J. Garnero; R. Jeanloz; L.H. Kellogg The D″ discontinuity and its implications (M. Gurnis; M.E. Wysession; E. Knittle; B.A. Buffett, eds.), The Core–Mantle Boundary Region, American Geophysical Union, Washington, DC, USA, 1998, pp. 273-298

[2] B. Romanowicz Using seismic waves to image Earth's internal structure, Nature, Volume 451 (2008), pp. 266-268

[3] M. Panning; B. Romanowicz Inferences on flow at the base of Earth's mantle based on seismic anisotropy, Science, Volume 303 (2004), pp. 351-353

[4] Paulius Micikevicius, 3D finite-difference computation on GPUs using CUDA, in: GPGPU-2: Proceedings of the 2nd Workshop on General Purpose Processing on Graphics Processing Units, Washington, DC, USA, March 2009, pp. 79–84.

[5] Rached Abdelkhalek; Henri Calandra; Olivier Coulaud; Jean Roman; Guillaume Latu Fast seismic modeling and reverse time migration on a GPU cluster http://hal.inria.fr/docs/00/40/39/33/PDF/hpcs.pdf in: Waleed W. Smari, John P. McIntire (Eds.), High Performance Computing & Simulation 2009, Leipzig, Germany, June 2009, pp. 36–44

[6] Dimitri Komatitsch; David Michéa; Gordon Erlebacher Porting a high-order finite-element earthquake modeling application to NVIDIA graphics cards using CUDA, J. Parallel Distributed Comput., Volume 69 (2009) no. 5, pp. 451-460

[7] Dimitri Komatitsch; Gordon Erlebacher; Dominik Göddeke; David Michéa High-order finite-element seismic wave propagation modeling with MPI on a large GPU cluster, J. Comput. Phys., Volume 229 (2010) no. 20, pp. 7692-7714

[8] David Michéa; Dimitri Komatitsch Accelerating a 3D finite-difference wave propagation code using GPU graphics cards, Geophys. J. Int., Volume 182 (2010) no. 1, pp. 389-402

[9] Zhe Fan, Feng Qiu, Arie E. Kaufman, Suzanne Yoakum-Stover, GPU cluster for high performance computing, in: SC '04: Proceedings of the 2004 ACM/IEEE Conference on Supercomputing, November 2004, p. 47.

[10] Dominik Göddeke; Robert Strzodka; Jamaludin Mohd-Yusof; Patrick McCormick; Sven H.M. Buijssen; Matthias Grajewski; Stefan Turek Exploring weak scalability for FEM calculations on a GPU-enhanced cluster, Parallel Comput., Volume 33 (2007) no. 10–11, pp. 685-699

[11] James C. Phillips, John E. Stone, Klaus Schulten, Adapting a message-driven parallel application to GPU-accelerated clusters, in: SC '08: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, November 2008, pp. 1–9.

[12] Everett H. Phillips, Yao Zhang, Roger L. Davis, John D. Owens, Rapid aerodynamic performance prediction on a cluster of graphics processing units, in: Proceedings of the 47th AIAA Aerospace Sciences Meeting, January 2009, pp. 1–11.

[13] R. Vai; J.M. Castillo-Covarrubias; F.J. Sánchez-Sesma; D. Komatitsch; J.P. Vilotte Elastic wave propagation in an irregularly layered medium, Soil Dyn. Earthquake Eng., Volume 18 (1999) no. 1, pp. 11-18

[14] Jeroen Tromp; Dimitri Komatitsch; Qinya Liu Spectral-element and adjoint methods in seismology, Commun. Comput. Phys., Volume 3 (2008) no. 1, pp. 1-32

[15] P. Moczo; J. Robertsson; L. Eisner The finite-difference time-domain method for modeling of seismic wave propagation (Ru-Shan Wu; Valérie Maupin, eds.), Advances in Wave Propagation in Heterogeneous Media, Advances in Geophysics, vol. 48, Elsevier–Academic Press, London, UK, 2007, pp. 421-516 (Chapter 8)

[16] B. Lombard; J. Piraux; C. Gélis; J. Virieux Free and smooth boundaries in 2-D finite-difference schemes for transient elastic waves, Geophys. J. Int., Volume 172 (2008) no. 1, pp. 252-261

[17] Kasper van Wijk; Dimitri Komatitsch; John A. Scales; Jeroen Tromp Analysis of strong scattering at the micro-scale, J. Acoust. Soc. Am., Volume 115 (2004) no. 3, pp. 1006-1011

[18] S. Chevrot; N. Favier; D. Komatitsch Shear wave splitting in three-dimensional anisotropic media, Geophys. J. Int., Volume 159 (2004) no. 2, pp. 711-720

[19] J.M. Carcione; P.J. Wang A Chebyshev collocation method for the wave equation in generalized coordinates, Comp. Fluid Dyn. J., Volume 2 (1993), pp. 269-290

[20] D. Komatitsch; F. Coutel; P. Mora Tensorial formulation of the wave equation for modelling curved interfaces, Geophys. J. Int., Volume 127 (1996) no. 1, pp. 156-168

[21] C. Bernardi; Y. Maday; A.T. Patera A new nonconforming approach to domain decomposition: the Mortar element method (H. Brezis; J.L. Lions, eds.), Nonlinear Partial Differential Equations and Their Applications, Séminaires du Collège de France, Pitman, Paris, 1994, pp. 13-51

[22] D.A. Kopriva; S.L. Woodruff; M.Y. Hussaini Computation of electromagnetic scattering with a non-conforming discontinuous spectral element method, Int. J. Numer. Meth. Eng., Volume 53 (2002) no. 1, pp. 105-122

[23] E. Chaljub; Y. Capdeville; J.P. Vilotte Solving elastodynamics in a fluid–solid heterogeneous sphere: a parallel spectral-element approximation on non-conforming grids, J. Comput. Phys., Volume 187 (2003) no. 2, pp. 457-491

[24] D.A. Kopriva Metric identities and the discontinuous spectral element method on curvilinear meshes, J. Sci. Comput., Volume 26 (2006) no. 3, pp. 301-327

[25] Lucas C. Wilcox; Georg Stadler; Carsten Burstedde; Omar Ghattas A high-order discontinuous Galerkin method for wave propagation through coupled elastic-acoustic media, J. Comput. Phys., Volume 229 (2010) no. 24, pp. 9373-9396

[26] W.H. Reed, T.R. Hill, Triangular mesh methods for the neutron transport equation, Technical Report LA-UR-73-479, Los Alamos Scientific Laboratory, Los Alamos, USA, 1973.

[27] Richard S. Falk; Gerard R. Richter Explicit finite element methods for symmetric hyperbolic equations, SIAM J. Numer. Anal., Volume 36 (1999) no. 3, pp. 935-952

[28] Fang Q. Hu; M.Y. Hussaini; Patrick Rasetarinera An analysis of the discontinuous Galerkin method for wave propagation problems, J. Comput. Phys., Volume 151 (1999) no. 2, pp. 921-946

[29] B. Rivière; M.F. Wheeler Discontinuous finite element methods for acoustic and elastic wave problems, Contemp. Math., Volume 329 (2003), pp. 271-282

[30] Peter Monk; Gerard R. Richter A discontinuous Galerkin method for linear symmetric hyperbolic systems in inhomogeneous media, J. Sci. Comput., Volume 22–23 (2005) no. 1–3, pp. 443-477

[31] Marcus J. Grote; Anna Schneebeli; Dominik Schötzau Discontinuous Galerkin finite element method for the wave equation, SIAM J. Numer. Anal., Volume 44 (2006) no. 6, pp. 2408-2431

[32] Marc Bernacki; Stéphane Lanteri; Serge Piperno Time-domain parallel simulation of heterogeneous wave propagation on unstructured grids using explicit, nondiffusive, discontinuous Galerkin methods, J. Comput. Acoust., Volume 14 (2006) no. 1, pp. 57-81

[33] M. Dumbser; M. Käser An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes—II. The three-dimensional isotropic case, Geophys. J. Int., Volume 167 (2006) no. 1, pp. 319-336

[34] S.P. Oliveira; G. Seriani Effect of element distortion on the numerical dispersion of spectral element methods, Commun. Comput. Phys., Volume 9 (2011) no. 4, pp. 937-958

[35] L. Brillouin Tensors in Mechanics and Elasticity, Academic Press, New York, USA, 1964

[36] Roland Martin; Dimitri Komatitsch; Abdelaâziz Ezziani An unsplit convolutional perfectly matched layer improved at grazing incidence for seismic wave equation in poroelastic media, Geophysics, Volume 73 (2008) no. 4, p. T51-T61

[37] Roland Martin; Dimitri Komatitsch; Stephen D. Gedney A variational formulation of a stabilized unsplit convolutional perfectly matched layer for the isotropic or anisotropic seismic wave equation, Comput. Model. Eng. Sci., Volume 37 (2008) no. 3, pp. 274-304

[38] G. Seriani; E. Priolo A spectral element method for acoustic wave simulation in heterogeneous media, Finite Elem. Anal. Des., Volume 16 (1994), pp. 337-348

[39] Jonás D. De Basabe; Mrinal K. Sen Grid dispersion and stability criteria of some common finite-element methods for acoustic and elastic wave equations, Geophysics, Volume 72 (2007) no. 6, p. T81-T95

[40] Laura Carrington, Dimitri Komatitsch, Michael Laurenzano, Mustafa Tikir, David Michéa, Nicolas Le Goff, Allan Snavely, Jeroen Tromp, High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE on 62 thousand processor cores, in: Proceedings of the ACM/IEEE Supercomputing SC '2008 Conference, 2008, pp. 1–11 (Article #60, Gordon Bell Prize finalist article).

[41] Thomas J.R. Hughes The Finite Element Method, Linear Static and Dynamic Finite Element Analysis, Prentice–Hall International, Englewood Cliffs, New Jersey, USA, 1987

[42] Tarje Nissen-Meyer; Alexandre Fournier; F.A. Dahlen A 2-D spectral-element method for computing spherical-earth seismograms – II. Waves in solid–fluid media, Geophys. J. Int., Volume 174 (2008), pp. 873-888

[43] D. Komatitsch; L.P. Vinnik; S. Chevrot SHdiff/SVdiff splitting in an isotropic Earth, J. Geophys. Res., Volume 115 (2010) no. B7, p. B07312

[44] NVIDIA Corporation, NVIDIA CUDA Programming Guide version 2.3, Santa Clara, California, USA, July 2009, 139 pp.

[45] K.T. Danielson; R.R. Namburu Nonlinear dynamic finite element analysis on parallel computers using Fortran90 and MPI, Adv. Eng. Software, Volume 29 (1998) no. 3–6, pp. 179-186

[46] R. Dolbeau, S. Bihan, F. Bodin, HMPP: A hybrid multi-core parallel programming environment, in: Proceedings of the Workshop on General Purpose Processing on Graphics Processing Units (GPGPU '2007), Boston, Massachusetts, USA, October 2007, pp. 1–5.

Cited by Sources:

Comments - Policy