Why?
Improve efficiency by running 10s - 1000s of models ‘simultaneously’.
2021 Southwest Pacific Ocean swordfish stock assessment
9,300 model runs totalling ~46 months of computation time.
Why?
- Efficiency
- Knowledge acquisition
- Automation, transparency, reproducibility & portability
- Multi-model inference
Software containers
Better science
How?
High-throughput computing (HTC)
- Set-up to handle running many jobs simultaneously
- Ideal for running short, small, independent (embarrassingly parallel) jobs.
High-performance computing (HPC)
- Can handle HTC workflows (in theory)
- Can also handle long running, large, multi-processor jobs (true parallel processing)
2024 North Pacific shortfin mako shark assessment: Used HTC resources to complete ~4 months months of Bayesian simulation-estimation evaluations (18,000 model runs using RStan) in ~3 hours (1027 \(\times\) faster) during a working group meeting.
Example: Fitting large spatiotemporal model in R using TMB required 128 CPUs & 1TB RAM.
How?
Available resources
High-throughput computing (HTC)
High-performance computing (HPC)
OpenScienceGrid (OSG): OSPool
How?
OpenScienceGrid (OSG)
- Uses HTCondor distributed computing network (no shared file system between compute nodes) to implement HTC workflows
- Free to use for US based researchers affiliated with academic/government organization and using OSG for research/education efforts
- Should not be used to analyze protected data
NOAA Hera
- Uses Slurm to schedule HPC (or HTC) workflows
- Shared file system between compute nodes
- Allocation determines access
- NOAA resource so no restrictions on acceptable use/analyzing protected data if working on mission related tasks
Both use software containers
Software containers
Many may already be using containers in existing cloud-based workspaces such as GitHub Codespaces or Posit Workbench
- Application: set up identical, custom software environments on OSG and Hera
- Application: use to “version” analyses by “freezing” packages/libraries
Software containers
Apptainer
- Secure, portable and reproducible software container for Linux operating systems
- Easy to use
- Doesn’t require root privileges to build making it ideal for HTC/HPC environments
- Plays nice with existing containers (e.g., Docker)
Let’s walk through an example
In this case we will use NOAA Hera to conduct a quick retrospective analysis of all models in the Stock Synthesis (SS3) testing suite.
More complete documentation of this example can be found on our GitHub website.
Example results
The example ran 90 jobs (18 test models \(\times\) 5 runs each; base + 4 peels), with only one job ‘timing out’ at 1-hour limit.
\(~\) \(~\)
Excluding the job that timed out the 89 jobs run on Hera completed 3.15 hours of calculations (2.12 minutes per job) in an elapsed time of 14.48 minutes or \(\sim\) 13 \(\times\) faster.
Depletion estimates across retrospective peels from the SS3 testing model suite examples. Mohn’s \(\rho\) values are printed in each panel.
Start and stop time for jobs run on Hera, excluding the job that timed out.
Is this ‘time-savings’ worth it?
Benchmark testing
Using a baseline SS3 file, run 500 alternative models with different fixed parameter values of natural mortality and steepness (uses SS3 and R; r4ss)
Use the delta-MVLN approach to generate uncertainty in predictions so that models can be combined in an ensemble (uses R; ss3diags)
Run 5 retrospective peels (\(t-1\) to \(t-5\)) for each of the 500 models and calculate Mohn’s \(\rho\)
Note: Retrospective peels were treated as separate jobs for the purpose of the benchmark testing giving 3000 unique jobs.
Stock status plots from the model ensemble: A spawning biomass relative to the spawning biomass at MSY, and B fishing mortality relative to the fishing mortality at MSY. The median is showed in the solid line, darker band shows the 50th percentile and lighter band the 80th percentile.
Mohn’s \(\rho\) for alternative parameter combinations of natural mortality (M) and steepness (h). The solid black lines denote the original M (vertical line) and steepness (horizontal line).
Benchmark testing - OSG
All 3000 jobs hit the queue at the same time from a single array job submission and began executing almost immediately
Small fraction of jobs had bad file transfer and had to be relaunched
Total computation time was \(\sim\) 25 days, but elapsed time was \(\sim\) 3.5 hours (166 \(\times\) faster)
Start and stop time for jobs run on OpenScienceGrid (OSG).
Benchmark testing - Hera (as HTC)
Queue and job_array_id limits required multiple (staggered) job submissions
Large proportion of jobs suffered from resource competition during memory/disk intensive portion of SS3 calculations (SD calcs) and produced incomplete outputs.
Note SS3 does not appear to crash/trigger an error when it runs out of memory/disk but instead writes out available output which could appear complete.
Start and stop time for jobs run on NOAA Hera.
Benchmark testing - Hera (parallel)
The Hera workflow was re-configured to run parallel jobs within 15 compute nodes using the gnu parallel utility1.
All jobs hit the queue at the same time and executed as resources became available within each node.
Each node had 200 jobs allocated to it spread out over 40 CPUs resulting in the \(\sim\) 5 distinct waves of job execution.
Total computation time for 3000 jobs was \(\sim\) 25.5 days, but elapsed time was \(\sim\) 1.75 hours
2998 of 2999 (99.99 %) completed models produced identical output to OSG
Start and stop time for jobs run on NOAA Hera using gnu parallel utility.
Contact us
nicholas.ducharme-barth at noaa.gov
megumi.oshima at noaa.gov
