Merge pull request #28 from seahorce-scidac/update_docs_for_particles

get particle example working, clean out a lot of unused stuff from ER…
seahorce-scidac · Sep 18, 2023 · ac2091b · ac2091b
2 parents 6ce096e + 8fe5237
commit ac2091b
Show file tree

Hide file tree

Showing 30 changed files with 606 additions and 366 deletions.
diff --git a/Docs/sphinx_doc/Checkpoint.rst b/Docs/sphinx_doc/Checkpoint.rst
@@ -9,9 +9,6 @@ Checkpoint / Restart
 .. toctree::
    :maxdepth: 1
 
-**Note: Checkpoint and restart capability are currently untested and may produce
-unexpected behavior.**
-
 ROMSX has a standard sort of checkpointing and restarting capability and
 uses the native AMReX format for reading and writing checkpoints.
 In the inputs file, the following options control the generation of
@@ -29,10 +26,10 @@ List of Parameters
 | Parameter                       | Definition     | Acceptable     | Default        |
 |                                 |                | Values         |                |
 +=================================+================+================+================+
-| **romsx.check_file**              | prefix for     | String         | “*chk*”        |
+| **romsx.check_file**            | prefix for     | String         | “*chk*”        |
 |                                 | restart files  |                |                |
 +---------------------------------+----------------+----------------+----------------+
-| **romsx.check_int**               | how often (by  | Integer        | -1             |
+| **romsx.check_int**             | how often (by  | Integer        | -1             |
 |                                 | level-0 time   | :math:`> 0`    |                |
 |                                 | steps) to      |                |                |
 |                                 | write restart  |                |                |
@@ -46,7 +43,7 @@ Restarting
 | Parameter                       | Definition     | Acceptable     | Default        |
 |                                 |                | Values         |                |
 +=================================+================+================+================+
-| **romsx.restart**                 | name of the    | String         | not used if    |
+| **romsx.restart**               | name of the    | String         | not used if    |
 |                                 | file           |                | not set        |
 |                                 | (directory)    |                |                |
 |                                 | from which to  |                |                |

diff --git a/Docs/sphinx_doc/Particles.rst b/Docs/sphinx_doc/Particles.rst
@@ -0,0 +1,52 @@
+
+ .. role:: cpp(code)
+    :language: c++
+
+ .. _Particles:
+
+Particles
+=========
+
+ROMSX has the option to include Lagrangian particles in addition to the mesh-based solution.  Currently the
+particle functionality is very simple -- the particles are initialized randomly, one per mesh cell
+in a particular plane, and are advected by the velocity field.
+
+However, the AMReX particle data structure is very general and particles may take on a number of
+different roles in future.
+
+To enable the use of particles, set
+
+::
+
+   USE_PARTICLES = TRUE
+
+in the GNUmakefile if using gmake, or add
+
+::
+
+   -DROMSX_ENABLE_PARTICLES:BOOL=ON \
+
+to the cmake command if using cmake.  (See, e.g., ``Build/cmake_with_particles.sh``)
+
+One must also set
+
+::
+
+   romsx.use_tracer_particles = true
+
+in the inputs file or on the command line at runtime.
+
+Currently, by default, the particles are initialized at cell centers, one per cell when the cell index is
+(3,j,k), with zero initial velocity.  They are advanced in time every time step.
+
+Caveat: the particle information is currently output when using the AMReX-native plotfile format, but not
+when using netcdf.  Writing particles into the netcdf files is a WIP.
+
+To see an example of using the particle functionality, build the executable using gmake in Exec/ParticlesOverSeaMount.
+
+To visualize the number of paritcles per cell as a mesh-based variable, add ``particle_count`` to the line in the inputs file
+
+::
+
+   romsx.plot_vars_1 =
+
diff --git a/Docs/sphinx_doc/building.rst b/Docs/sphinx_doc/building.rst
@@ -41,30 +41,33 @@ or if using tcsh,
 
 #. Edit the ``GNUmakefile``; options include
 
-   +-----------------+--------------------------------+------------------+-------------+
-   | Option name     | Description                    | Possible values  | Default     |
-   |                 |                                |                  | value       |
-   +=================+================================+==================+=============+
-   | COMP            | Compiler (gnu or intel)        | gnu / intel      | None        |
-   +-----------------+--------------------------------+------------------+-------------+
-   | USE_MPI         | Whether to enable MPI          | TRUE / FALSE     | FALSE       |
-   +-----------------+--------------------------------+------------------+-------------+
-   | USE_OMP         | Whether to enable OpenMP       | TRUE / FALSE     | FALSE       |
-   +-----------------+--------------------------------+------------------+-------------+
-   | USE_CUDA        | Whether to enable CUDA         | TRUE / FALSE     | FALSE       |
-   +-----------------+--------------------------------+------------------+-------------+
-   | DEBUG           | Whether to use DEBUG mode      | TRUE / FALSE     | FALSE       |
-   +-----------------+--------------------------------+------------------+-------------+
-   | USE_NETCDF      | Whether to compile with NETCDF | TRUE / FALSE     | FALSE       |
-   +-----------------+--------------------------------+------------------+-------------+
-   | PROFILE         | Include profiling info         | TRUE / FALSE     | FALSE       |
-   +-----------------+--------------------------------+------------------+-------------+
-   | TINY_PROFILE    | Include tiny profiling info    | TRUE / FALSE     | FALSE       |
-   +-----------------+--------------------------------+------------------+-------------+
-   | COMM_PROFILE    | Include comm profiling info    | TRUE / FALSE     | FALSE       |
-   +-----------------+--------------------------------+------------------+-------------+
-   | TRACE_PROFILE   | Include trace profiling info   | TRUE / FALSE     | FALSE       |
-   +-----------------+--------------------------------+------------------+-------------+
+   +-----------------+----------------------------------+------------------+-------------+
+   | Option name     | Description                      | Possible values  | Default     |
+   |                 |                                  |                  | value       |
+   +=================+==================================+==================+=============+
+   | COMP            | Compiler (gnu or intel)          | gnu / intel      | None        |
+   +-----------------+----------------------------------+------------------+-------------+
+   | USE_MPI         | Whether to enable MPI            | TRUE / FALSE     | FALSE       |
+   +-----------------+----------------------------------+------------------+-------------+
+   | USE_OMP         | Whether to enable OpenMP         | TRUE / FALSE     | FALSE       |
+   +-----------------+----------------------------------+------------------+-------------+
+   | USE_CUDA        | Whether to enable CUDA           | TRUE / FALSE     | FALSE       |
+   +-----------------+----------------------------------+------------------+-------------+
+   | DEBUG           | Whether to use DEBUG mode        | TRUE / FALSE     | FALSE       |
+   +-----------------+----------------------------------+------------------+-------------+
+   | USE_NETCDF      | Whether to compile with NETCDF   | TRUE / FALSE     | FALSE       |
+   +-----------------+----------------------------------+------------------+-------------+
+   | USE_PARTICLES   | Whether to compile with particle | TRUE / FALSE     | FALSE       |
+   |                 | functionality enabled            |                  |             |
+   +-----------------+----------------------------------+------------------+-------------+
+   | PROFILE         | Include profiling info           | TRUE / FALSE     | FALSE       |
+   +-----------------+----------------------------------+------------------+-------------+
+   | TINY_PROFILE    | Include tiny profiling info      | TRUE / FALSE     | FALSE       |
+   +-----------------+----------------------------------+------------------+-------------+
+   | COMM_PROFILE    | Include comm profiling info      | TRUE / FALSE     | FALSE       |
+   +-----------------+----------------------------------+------------------+-------------+
+   | TRACE_PROFILE   | Include trace profiling info     | TRUE / FALSE     | FALSE       |
+   +-----------------+----------------------------------+------------------+-------------+
 
    .. note::
       **Do not set both USE_OMP and USE_CUDA to true.**
@@ -116,8 +119,20 @@ An example CMake configure command to build ROMSX with MPI is listed below:
           -DCMAKE_Fortran_COMPILER:STRING=mpifort \
           .. && make
 
-Note that CMake is able to generate makefiles for the Ninja build system as well which will allow for faster building of the executable(s).
+An example CMake configure command to build ROMSX with MPI and particles is listed below:
+
+::
+
+    cmake -DCMAKE_BUILD_TYPE:STRING=Release \
+          -DROMSX_ENABLE_MPI:BOOL=ON \
+          -DCMAKE_CXX_COMPILER:STRING=mpicxx \
+          -DCMAKE_C_COMPILER:STRING=mpicc \
+          -DCMAKE_Fortran_COMPILER:STRING=mpifort \
+          -DROMSX_ENABLE_PARTICLES:BOOL=ON \
+          .. && make
+
 
+Note that CMake is able to generate makefiles for the Ninja build system as well which will allow for faster building of the executable(s).
 
 Perlmutter (NERSC)
 ~~~~~~~~~~~~~~~~~~

diff --git a/Docs/sphinx_doc/index.rst b/Docs/sphinx_doc/index.rst
@@ -49,6 +49,7 @@ In addition to this documentation, there is API documentation for ROMSX generate
    theory/Variables.rst
    theory/Equations.rst
    theory/VerticalMixing.rst
+   Particles.rst
 
 .. toctree::
    :caption: IMPLEMENTATION

diff --git a/Exec/ParticlesOverSeaMount/CMakeLists.txt b/Exec/ParticlesOverSeaMount/CMakeLists.txt
@@ -0,0 +1,13 @@
+set(romsx_exe_name particles_over_sea_mount)
+
+add_executable(${romsx_exe_name} "")
+target_sources(${romsx_exe_name}
+   PRIVATE
+     prob.H
+     prob.cpp
+)
+
+target_include_directories(${romsx_exe_name} PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+
+include(${CMAKE_SOURCE_DIR}/CMake/BuildROMSXExe.cmake)
+build_romsx_exe(${romsx_exe_name})
diff --git a/Exec/ParticlesOverSeaMount/GNUmakefile b/Exec/ParticlesOverSeaMount/GNUmakefile
@@ -0,0 +1,40 @@
+# AMReX
+COMP = gnu
+PRECISION = DOUBLE
+
+# Profiling
+PROFILE       = FALSE
+TINY_PROFILE  = FALSE
+COMM_PROFILE  = FALSE
+TRACE_PROFILE = FALSE
+MEM_PROFILE   = FALSE
+USE_GPROF     = FALSE
+
+# Performance
+USE_MPI   = TRUE
+USE_OMP   = FALSE
+USE_CUDA  = FALSE
+USE_HIP   = FALSE
+USE_SYCL  = FALSE
+
+TEST = TRUE
+USE_ASSERTION = TRUE
+
+USE_NETCDF    = FALSE
+USE_PARTICLES = TRUE
+
+# Debugging
+DEBUG = FALSE
+DEBUG = TRUE
+
+# Physics
+SALINITY = TRUE
+
+# GNU Make
+Bpack := ./Make.package
+Blocs := .
+
+ROMSX_HOME := ../..
+ROMSX_PROBLEM_DIR = $(ROMSX_HOME)/Exec/ParticlesOverSeaMount
+
+include $(ROMSX_HOME)/Exec/Make.ROMSX
diff --git a/Exec/ParticlesOverSeaMount/Make.package b/Exec/ParticlesOverSeaMount/Make.package
@@ -0,0 +1,2 @@
+CEXE_headers += prob.H
+CEXE_sources += prob.cpp
diff --git a/Exec/ParticlesOverSeaMount/inputs b/Exec/ParticlesOverSeaMount/inputs
@@ -0,0 +1,81 @@
+# ------------------  INPUTS TO MAIN PROGRAM  -------------------
+max_step = 300
+stop_time = 30000.0
+
+amrex.fpe_trap_invalid = 1
+
+fabarray.mfiter_tile_size = 1024 1024 1024
+
+# PROBLEM SIZE & GEOMETRY
+geometry.prob_lo     =      0.     0.    -150.
+geometry.prob_hi     =  41000. 80000.       0.
+
+amr.n_cell           =  41     80      16
+
+# periodic in x to match WRF setup
+geometry.is_periodic = 1 1 0
+#ylo.type = "SlipWall"
+#yhi.type = "SlipWall"
+zlo.type = "SlipWall"
+zhi.type = "SlipWall"
+
+# TIME STEP CONTROL
+romsx.fixed_dt       = 300.0 # Timestep size (seconds)
+# NDTFAST  = 30.0 # Number of baratropic steps => 300.0/30.0 = 10.0
+romsx.fixed_fast_dt  = 10.0 # Baratropic timestep size (seconds)
+# romsx.fixed_fast_dt  = 300.0 # Baratropic timestep size (seconds) testing value
+romsx.fixed_ndtfast_ratio  = 30 # Baratropic timestep size (seconds)
+
+romsx.use_tracer_particles = 1
+
+# DIAGNOSTICS & VERBOSITY
+romsx.sum_interval   = 1       # timesteps between computing mass
+romsx.v              = 0       # verbosity in ROMSX.cpp (0: none, 1: print boxes, etc, 2: print values)
+amr.v              = 1       # verbosity in Amr.cpp
+
+# REFINEMENT / REGRIDDING
+amr.max_level       = 0       # maximum level number allowed
+
+# CHECKPOINT FILES
+romsx.check_file      = chk        # root name of checkpoint file
+romsx.check_int       = -57600      # number of timesteps between checkpoints
+
+# PLOTFILES
+romsx.plot_file_1     = plt        # prefix of plotfile name
+romsx.plot_int_1      = 5            # number of timesteps between plotfiles
+romsx.plot_vars_1     = omega salt temp x_velocity y_velocity z_velocity particle_count
+romsx.plotfile_type   = amrex
+
+# SOLVER CHOICE
+romsx.use_gravity = true
+romsx.use_coriolis = true
+romsx.horizontal_advection_scheme = "upstream3" # upstream3 or centered4
+romsx.use_rayleigh_damping = false
+romsx.spatial_order = 2
+
+romsx.les_type         = "None"
+#
+# diffusion coefficient from Straka, K = 75 m^2/s
+#
+romsx.molec_diff_type = "ConstantAlpha"
+romsx.rho0_trans = 1.0 # [kg/m^3], used to convert input diffusivities
+romsx.dynamicViscosity = 75.0 # [kg/(m-s)] ==> nu = 75.0 m^2/s
+romsx.alpha_T = 75.0 # [m^2/s]
+
+# PROBLEM PARAMETERS (optional)
+prob.R0 = 1027.0
+prob.S0 = 35.0
+prob.T0 = 14.0
+
+# PROBLEM PARAMETERS (shear)
+prob.rho_0 = 1.0
+prob.T_0   = 1.0
+prob.A_0   = 1.0
+prob.u_0   = 0.0
+prob.v_0   = 0.0
+prob.rad_0 = 0.25
+prob.z0    = 0.1
+prob.zRef  = 80.0e-3
+prob.uRef  = 8.0e-3
+
+prob.prob_type = 12
diff --git a/Exec/ParticlesOverSeaMount/prob.H b/Exec/ParticlesOverSeaMount/prob.H
@@ -0,0 +1,58 @@
+#ifndef _PROB_H_
+#define _PROB_H_
+
+#include "AMReX_REAL.H"
+
+struct ProbParm {
+  // Linear equation of State parameters
+  amrex::Real R0 = 1028; // background density value (Kg/m3) used in Linear Equation of State
+  amrex::Real S0 = 35.0; // background salinity (nondimensional) constant
+  amrex::Real T0 = 5.0; // background potential temperature (Celsius) constant
+
+  /*
+      // Background salinity (nondimensional) constant.
+    amrex::Real S0;
+
+    // Background Temperature (Celsius) constant.
+    amrex::Real T0;
+  */
+
+    // mod_scalars values
+
+    // Mean density (Kg/m3) used when the Boussinesq approximation is inferred.
+    amrex::Real rho0 = 1025.0;
+
+    // physical constant for gravity
+#ifdef SOLITON
+    amrex::Real g = 1.0;                // non-dimensional
+#elif defined WBC_1 || defined WBC_2 || defined WBC_3
+    amrex::Real g = 9.8;                // m/s2
+#elif defined CIRCLE
+    amrex::Real g = 3.92e-2;            // m/s2
+#else
+    amrex::Real g = 9.81;               // m/s2
+#endif
+
+
+  amrex::Real rho_0   = 0.0;
+  amrex::Real Theta_0 = 1.0;
+  amrex::Real A_0     = 1.0;
+  amrex::Real B_0     = 0.0;
+  amrex::Real u_0     = 0.0;
+  amrex::Real v_0     = 0.0;
+  amrex::Real rad_0   = 0.0;
+  amrex::Real z0      = 0.1;  // Surface Roughness
+  amrex::Real zRef    = 80.0; // Reference Height
+  amrex::Real uRef    = 0.0;  // Reference Wind Speed
+
+  amrex::Real xc_frac = 0.5; // Location of "center" of scalar (multiplies domain length)
+  amrex::Real yc_frac = 0.5; // Location of "center" of scalar (multiplies domain length)
+  amrex::Real zc_frac = 0.5; // Location of "center" of scalar (multiplies domain length)
+
+  int prob_type = -1;
+}; // namespace ProbParm
+
+extern ProbParm parms;
+
+#endif
+
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		CEXE_headers += prob.H
		CEXE_sources += prob.cpp