Features

Numerical schemes and Physics

Here are the features of the fluid solvers implemented in RAMSES-GPU with MPI and GPU (CUDA) parallelisation :

  • finite volume methods for compressible flows (HYDRO and MHD), 3D cartesian grids
  • Directional splitting Godunov-based, 2nd order for Hydrodynamics
  • Unsplit Godunov-based, 2nd order for Hydrodynamics; Riemann solvers are almost exact (involves Newtow-Raphson iterations), HLL, HLLC
  • Unsplit Godunov-based, 2nd order for Hydrodynamics + Constraint Transport for magnetic field update (see reference [Fromang2006]); Riemann solvers for MHD are HLL, LLF, HLLD
  • Dissipative terms (viscosity, resistivity)
  • forcing source term for driven turbulence simulation
  • online structure functions computations for turbulence analysis
  • Parallel IO: parallel HDF5 with XDMF, parallel-NetCDF, pVTK file format
  • Static Gravity (e.g. Rayleigh-Taylor study)
  • Self-gravity using a FFT-based Poisson solver (partially implemented as of March 2015)

Everything runs on GPU (including the numerical scheme computations, boundary condition update at each time step) without requiring transferring large data back on CPU memory. We only require to download data from GPU memory to CPU when dumping data into files.

All the schemes are available in single and double precision.

About Numerical Solutions for Hyperbolic Systems of Conservation Laws, here are some good starting online materials amr_lecture1.pdf and amr_lecture2.pdf by R. Teyssier.

Some online material about grid MHD numerical methods and Riemann solvers: GridMHD by P. Hennebelle.

Border conditions

There are 5 types of borders conditions implemented (CPU/GPU):

  • reflexive
  • absorbant
  • periodic
  • shear (MHD only, using the shearing box approximation): used in the Magneto-Rotational Instability studies
  • z_stratified (MHD only, used in stratified, shearing box problems)

Memory footprint

TODO

Output file formats

  • PNETCDF can be used in MPI parallel run (multi CPU or multi GPU). PnetCDF often gives better IO performance than HDF5 for very large file on Lustre file system.

  • HDF5 can be used for both serial (mono-CPU / mono-GPU) run or for parallel MPI run where collective IO will be used (every MPI process writes in the same file)

    • XDMF When HDF5 ouput is enabled, a xdmf wrapper file is written. This allow the output to be read directly by Paraview or VisIt.

Tip

Hdf5 data can be efficiently used for debug, see tool h5diff (usefull for a quick comparison between CPU and GPU generated data). h5diff can print the location and values where there are differences between two hdf5 files having the same structure.

  • VTK can be used for both serial (mono-CPU / mono-GPU) run or for parallel MPI run.

    • VTK internal format is image data (.vti extension) especially usefull for uniform grid
    • When doing parallel MPI run, the VTK file format is pvti (partioned image data), that is each MPI process writes its own sub-domain data using VTI format, and rank 0 process write a pvti (XML) file used to gather all pieces in Paraview.

    Note

    VTK output can be used wether or not the VTK library is installed on the build system. If VTK library is found at build time, it will be used; if not found, we use our own routine to dump data in the VTI format (binary

    Note

    Our own VTK output routine only supports Ascii and Raw Binary format (no base64 encoding which is available is building with the VTK library).

  • PNG for 2D data, serial run, only available through library GraphicsMagick++

  • XSM for 2D/3D data, serial run. This is a custom binary format.

  • NRRD for 2D/3D data, serial run. This output can be used for visualization with the WebGL toolkit xtk

Note that if you want to perform a restart run, this feature is only available with HDF5 outputs.

GUI

A GLUT widget is available (for 2D CPU/GPU) simulations.

Other numerical schemes

The following features are only available for 2D simulations and no-MPI parallelization:

  • Lax-Liu positive scheme (adapted from original Fortran code found in [LaxLiu1995], Hydro2D only)
  • Central scheme (adapted from [Kurganov2002], see also Central station C++ code, CentPack, Hydro2D only)
  • Relaxing TVD scheme (adapted from [Pen2003], Hydro2D only)

An early version of RAMSES-GPU for 2D simulations using mono-GPU implementation is described in [Kestener2010].

Future possible developments

  • Improve memory footprints for MHD-CT scheme, using a slab-by-slab procedure (DONE).
  • Implement cylindrical / spherical coordinate systems
  • Implement higher order schemes (WENO-based ??)