Document requirements and Implement model run(s) for CO-OPS

[Michael-Lalime]

@ZacharyWills @KatherinePowell-NOAA

We received requirements from CO-OPS regarding needs for a development space. I want to ask your advice about setting up a space or possibly even a separate head node for them.

What do you think? Can we make something like this work in the near-ish future?

Problem Statement
CO-OPS currently has two AWS servers for OFS applications; ofs-east1 is an operational server to generate all OFS operational products; ofs-work-east1 is presently used as a developmental server for software and product development and testing, and to create real-time products for all OFS which are run in a semi-operational mode for development and testing. Therefore, ofs-work-east1 is mainly used as a parallel/staging server. There is an urgent need to build a new developmental AWS server that the modeling team can use for OFS software and product development and testing.

This document contains requirements for ofs-dev-east1 to support CO-OPS development work. Any requests made herein to increase license count, CPU, etc., will need to be reviewed and/or approved and will impact the system's delivery date.
Requirements
Hardware requirements (Same hardware as ofs-work-east1, number of CPUs, memory, etc.)
Modeling team (9 modelers) heavily relies on ofs-dev
OS Change
For awareness the OS will change from RHEL 7 which is running on ofs-work-east1 and ofs-east1 to RHEL 8
Disk space
5TB EBS for /opt/archive (software, and semi-operational OFS under development)
8 TB EBS mount as /opt/ofsdev/users - used by modelers for R&D related projects, such as testing, developing, skill assessment etc.
+space exclusive to the SCI/SA project
What models will run on this server? Only OFS under testing is run on dev server, no semi-operational OFS runs on dev using cron jobs.
Mount /opt/ofs-bdp and /opt/ofs-all S3 buckets for data access (READ ONLY)
Other NOAA s3 bucket of data sources (e.g. satellite data, NCEI archives, etc.) might be needed to be mounted on. It can be figured out as needed in the future.
1TB /cache drive
This is required for the requested s3fs mounts

List of Software and Compilers
Matlab (1 concurrent license)
IDL (1 license)
ifort Fortran compiler
Icc C compiler
Totalview or similar debugger
Eclipse software development kit
Java
NCL command
Emacs
GIT ()
Anaconda with Python 3.9 or newer, Jupyter Notebook & Library
NCO NetCDF operators
Latest version NetCDF C and Fortran
NetCDF commands (e.g. ncdump)
Xmgrace
xmgredit5
Special Needs: commands saved in /usr/bin, nedit, gedit, gthumb
Possibly Allow CO-OPS external users to access the dev server
co-development with partners critical for closing the O2R cycle

OFS Data Processing System (SCI/SA) Requirements
Compute

Storage/Data:
Local disk storage (DEV possibly 1TB - it is covered by 8 TB under /opt/ofsdev/users )
Need 2 years of historical OFS data.
Historical OFS data at the NODD arn:aws:s3:::noaa-nos-ofs-pds
/opt/ofs-bdp will be mounted for READ ONLY
Note the SCI/SA project requires NODD S3 access to various data sets, including STOFS-2D-Global, STOFS-3DAtlantic, GFS, HRRR, etc.)

Software:

Apache Tomcat
Python 3 (3.9 or newer)
Anaconda and Pipenv

[mwengren]

I think some further discussion about CO-OPS needs/goals would be helpful. These requirements, for me at least, bring up the question of scope of the Sandbox, because I think these are broader than what we initially planned for.

It may be that some aspects belong in the Sandbox, whereas others do not.

Also, I don't see in these two descriptions the need to actually run ocean models, which is our primary mission:

• ofs-east1 is an operational server to generate all OFS operational products;
• ofs-work-east1 is presently used as a developmental server for software and product development and testing, and to create real-time products for all OFS which are run in a semi-operational mode for development and testing

I don't know how best to have this conversation. Perhaps we can do it here in this GitHub issue. I think once we can get a feel on the scope question, it may make sense to break these requirements down into more manageable pieces to implement the parts we feel belong in the Sandbox.