Parallel Runs: Local and Multi-Node MPI

This page explains how to choose and size parallel runs in corgihowfsc.

corgihowfsc supports two ways to run in parallel:

Local run: run on one machine using multiprocessing.
Multi-node (MPI) run: runs with MPI ranks, usually on a cluster, across multiple nodes.

For the full YAML field descriptions, see Parameter Reference. For details on how the local multiprocessing and MPI setups are arranged, see Parallel Setup Details.

Quick Start

Local Run

Use local mode on a laptop, workstation, or single cluster node.

Note

The values below are examples. Choose num_imager_worker, num_proper_process, and num_jac_process based on the CPUs available on your machine. See Sizing a Local Run.

runtime:
  use_mpi: false
  num_imager_worker: 3  # null: serial image generation; set > 1 for local image workers
  num_proper_process: 5    # PROPER processes inside each image worker
  num_jac_process: 6       # local Jacobian processes

Run:

python run_corgisim_nulling_gitl.py --param_file default_param.yml

Multi-node (MPI) Run

Use an MPI run when one machine is not enough and you want to spread the simulation across multiple cluster nodes. MPI (Message Passing Interface) is a standard communication protocol for parallel programs that coordinates separate processes across a cluster, including across nodes with separate memory, by passing messages between them.

In an MPI run, one Python process acts as the manager. It runs the GITL loop and sends image generation or Jacobian calculations to worker processes. The worker processes wait for tasks, run them, and send the results back to the manager.

This is useful because CorgiSim/PROPER image generation and Jacobian computation can be slow and resource intensive. MPI lets several workers run these tasks at the same time across a cluster allocation.

MPI documentation often calls these processes “ranks”. This page uses “manager process” and “worker processes” to explain the run.

runtime:
  use_mpi: true
  num_imager_worker: 21    # MPI worker processes
  num_proper_process: 5    # PROPER processes per worker process
  num_jac_process: 6       # ignored in MPI mode

For this example, we launch 22 total MPI processes:

1 manager process + 21 worker processes = 22 total MPI processes

Example Slurm launch:

#SBATCH --nodes=4
#SBATCH --ntasks=22
#SBATCH --cpus-per-task=5

export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

mpirun -np $SLURM_NTASKS python run_corgisim_nulling_gitl.py --param_file example_mpi_param.yml

Note

Example MPI submission scripts are available in corgihowfsc/scripts/example_mpi_*.sh. Use them as templates, but check the module loads, conda environment, --ntasks, and --cpus-per-task values for your cluster.

Choose Between Local and MPI

Situation	Recommended Run Type
Laptop or workstation	Local run
First test run or debugging	Local run
Single-node run with enough CPUs	Local run
Multi-node cluster allocation	MPI run
Slow CorgiSim image generation across many iterations	MPI run

Start with a local run. Switch to MPI when the per-iteration wall time is the bottleneck and you have a cluster allocation available.

Parallel Parameters

Parameter	Local Run (`use_mpi: false`)	MPI Run (`use_mpi: true`)
`num_imager_worker`	Number of local image worker processes. `null` means serial/default.	Number of MPI worker processes. Usually `--ntasks - 1`.
`num_proper_process`	Number of PROPER subprocesses inside each local image worker.	Number of PROPER subprocesses inside each MPI worker process. Usually matches `--cpus-per-task`.
`num_jac_process`	Number of local Jacobian processes.	Ignored. MPI Jacobian chunks use the MPI worker processes capped by `num_imager_worker`.
`use_mpi`	`false`	`true`

Sizing a Local Run

For local image generation, the approximate peak CPU use is:

num_imager_worker x num_proper_process

For example:

num_imager_worker: 3
num_proper_process: 5

can use up to:

3 x 5 = 15 CPU cores

num_jac_process controls a separate local Jacobian process pool. It is not active at the exact same time as image generation, but it should still be no larger than the CPUs available for the job.

At the start of the local run, it checks for oversubscription and warns if the requested process count exceeds the CPUs visible to the process. Oversubscription means asking for more parallel processes than available CPU cores, which can make the run slower rather than faster because processes compete for the same cores.

For a first local run, start conservatively:

runtime:
  use_mpi: false
  num_imager_worker: null
  num_proper_process: 5
  num_jac_process: 6

Increase num_imager_worker only if image generation is the bottleneck and the machine has enough CPUs for the additional workers.

MPI Run Model

In MPI mode, one process is the manager. It runs the main loop and sends work to the workers.

process 0       manager
process 1..N-1  workers

Worker processes are initialized once and reused for image generation and Jacobian chunks.

The manager process does not compute image or Jacobian tasks. Therefore:

total MPI processes = num_imager_worker + 1

Important

In MPI mode, launch num_imager_worker + 1 total MPI processes. Rank 0 is the manager and does not run frame or Jacobian tasks.

Example:

num_imager_worker: 21

requires:

mpirun -np 22 ...

or in Slurm:

#SBATCH --ntasks=22

MPI Process and CPU Sizing

For MPI mode, use this mapping:

--ntasks         = num_imager_worker + 1
--cpus-per-task  = num_proper_process

Example YAML:

runtime:
  use_mpi: true
  num_imager_worker: 21
  num_proper_process: 5

Example Slurm settings:

#SBATCH --ntasks=22
#SBATCH --cpus-per-task=5

This means:

manager process
worker processes
CPUs available to each MPI process

Each worker process may spawn up to num_proper_process PROPER subprocesses.

Choosing `num_imager_worker`

For image generation, the useful maximum is the number of images in one framelist:

n_wvl x (2 x n_probe_pairs + 1)

For HLC band 1 with 3 wavelengths and 3 probe pairs:

3 x (2 x 3 + 1) = 21

So a natural MPI setup is:

num_imager_worker: 21

and:

#SBATCH --ntasks=22

Using more than 21 worker processes will not speed up image generation for this case because there are only 21 image tasks in one framelist. Extra workers may still be initialized, but they will sit idle during image collection.

Process-Per-Core Mode

The intended parallel model is process-based:

MPI processes x PROPER processes

Threaded numerical libraries should be kept single-threaded because this workflow uses process-based parallelism:

export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

PROPER’s multi-run path uses a multiprocessing pool for NCPUS work and sets its multi-process FFT thread count to 1 internally. The environment variables above prevent threaded math libraries from adding extra parallelism around the process-based model.

Do not use:

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

Slurm Launch Notes

Prefer an explicit MPI process count:

mpirun -np $SLURM_NTASKS python run_corgisim_nulling_gitl.py --param_file example_mpi_param.yml

On systems where mpirun correctly uses the scheduler allocation, you can also omit -np:

mpirun python run_corgisim_nulling_gitl.py --param_file mpi_param.yml

Warning

Check that the MPI process count launched by Slurm matches num_imager_worker + 1. Extra MPI processes are initialized but remain idle; too few MPI processes reduce the requested parallelism.

If you use:

#SBATCH --ntasks-per-node=6

some MPI launchers may infer the total process count from:

nodes x ntasks-per-node

For example:

#SBATCH --nodes=4
#SBATCH --ntasks-per-node=6

This can launch 24 MPI processes. If your YAML has:

num_imager_worker: 21

you only wanted 22 total MPI processes, so 2 worker processes will be initialized but idle.

Common Startup Warnings

More MPI Processes Than Requested Workers

MPI launched 23 worker processes, but num_imager_worker=21.
Extra worker processes will be initialized but remain idle.

This means MPI launched more processes than your YAML will use. The run is still correct, but some allocated processes are wasted.

Fix by launching exactly:

num_imager_worker + 1

total MPI processes.

Fewer MPI Processes Than Requested Workers

MPI requested 21 active worker processes, but only 9 worker processes were launched.
The MPI task queues will use 9 workers.

The run is still correct, but slower than expected.

Fix by increasing the MPI process count or reducing num_imager_worker.

PROPER Processes Exceed CPUs Per Task

num_proper_process exceeds the CPU affinity visible to the MPI process

This usually means:

num_proper_process

is larger than:

#SBATCH --cpus-per-task

Fix by making them match, or by reducing num_proper_process.