SMP (symmetric multi-threaded) jobs

An SMP job requests multiple CPU cores to run multi-threaded programs within a single compute node. One can use this mechanism to run either the multi-threaded programs, or MPI programs.

In the case of MPI programs, all MPI processes will run within a compute node.

The key job scheduler parameter is:

#$ -pe shared N

where N is the number of CPU cores to use for the job. This value depends on how the program uses the multiple cores. A greater N value does not necessarily imply the program running faster.

Key points:

  • A sequential program does not automatically become multi-threaded just because it is submitted as a shared-memory job

  • Use -pe shared N

  • The h_data value is per-core.

  • Total memory is (h_data) * (-pe shared N). The job will not start if this product is “too big”.

  • If N is too large and the scheduler cannot find an available compute node, the job will not start.

Consider this example:

#!/bin/bash

#$ -cwd
#$ -l h_rt=8:00:00,h_data=4g
#$ -pe shared 16
#$ -o Stdout.$JOB_ID

This job requires 4g*16 = 64g of memory on a compute node to start.

  • It will not start on compute nodes having only 32G of memory.

  • Even if it could start on a 64GB (or greater) compute node, there will be some wait time because the scheduler needs to “drain” a whole 64GB spot to run this job.

  • If the program is sequential (not multi-threaded), 15 CPU cores will be idle (wasted!)

  • With -pe shared, all requested cores will be on one compute node. Nothing will run across compute nodes.

  • If your program is sequential but needs a big amount of memory, do not use -pe shared! Just use, e.g. -l h_data=64G (without -pe shared N) to run large-memory, sequential programs.

previous

Exercises

next

Exercises