Example
#!/bin/bash
#$ -cwd
#$ -l h_rt=1:00:00,h_data=2g
#$ -pe dc* 36
#$ -j y
#$ -o stdout.$JOB_ID
source /u/local/Modules/default/init/modules.sh
module load intel/18.0.4
cat $PE_HOSTFILE
which mpirun
mpirun -n 36 hostname
# Alternatively:
# mpirun -n $NSLOTS hostname
In one run, it ran on one 36-core compute node:
$ cat stdout.475739
n7645 36 pod_ib100.q@n7645 <NULL>
/u/local/compilers/intel/18.0.4/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin/mpirun
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
n7645
In another run, it ran across two compute nodes – 24 cores on one compute node and 12 cores on another:
$ cat stdout.475872
n6365 24 pod_ib56_xgold.q@n6365 <NULL>
n7050 12 pod_ib56_xgold.q@n7050 <NULL>
/u/local/compilers/intel/18.0.4/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin/mpirun
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n6365
n7050
n7050
n7050
n7050
n7050
n7050
n7050
n7050
n7050
n7050
n7050
n7050