Research IT

Using STAR-CCM+ on Nova

Introduction

STAR-CCM+ is used for solving problems in Computational Fluid Dynamics (CFD) and related multiphysics.  On Nova, users typically execute STAR-CCM+ analysis as batch jobs, though it can also be used interactively from Nova OnDemand.

Full STAR-CCM+ User Guide

Complete STAR-CCM+ User Guide is available as a PDF file that is available at the path:

/shared/hpc/starccm/starccm-userguide_19.04.009_en.pdf 

GPU Support

STAR-CCM+ has excellent support for GPUs.   Many STAR-CCM+ jobs that use a GPU will execute up to four times faster than a job that only uses CPU processors. 

Note also that running STAR-CCM+ with a GPU still requires CPUs for many operations, particularly with post-processing analysis.  It is recommended that you request at least 4 CPUs for each GPU requested.

STAR-CCM+ Versions

Use the module spider command to see the available versions:  

$ module spider starccm
----------------------------------------------------------------------------------------------------------------------------------------
 starccm:
----------------------------------------------------------------------------------------------------------------------------------------
    Versions:
       starccm/16.04.012-R8
       starccm/16.04.012
       starccm/17.04.008-R8
       starccm/17.04.008
       starccm/18.02.008-R8
       starccm/18.02.008
       starccm/19.04.007-R8
       starccm/19.04.007

 

Single Versus Double Precision

Note that the versions ending in -R8 are the double precision versions.   The -R8 versions use double precision floating numbers for all calculations.  Double precision is recommended for complex geometries.

Sample Batch Script for STAR-CCM+ Using a GPU

The following Slurm batch script shows a typical job that uses a GPU.   Some key points to note about how this job is launched:

  • The job requests 1 GPU from Slurm:   #SBATCH --gres=gpu:a100:1 
    You can request more than one GPU, of course.  It is recommended that you start with 1 GPU first and expand to additional CPUs as needed.
  • We use the option -batchsystem slurm to tell STAR-CCM+ to query the Slurm scheduler to determine the hardware assigned for the job.
  • We tell STAR-CCM+ not to use the Nvidia MPS system:    -gpgpu auto:1:nomps 
    (The Nvidia MPS system can conflict with the Slurm scheduler, so it is best not to use it.)
  • The job uses OpenMPI:  -mpi openmpi   
    (In earlier versions of STAR-CCM+,  Intel MPI was recommended over OpenMPI, but the latest versions of STAR-CCM+ works best with OpenMPI).
#!/bin/bash
#SBATCH -A myaccount
#SBATCH -J StarCCM-GPU
#SBATCH -D /work/mygroup/myworkdir/GPU
#SBATCH -N 1
#SBATCH -n 6
#SBATCH --partition=nova
#SBATCH --ntasks-per-node=6
#SBATCH --gres=gpu:a100:1
#SBATCH --mem=128G
#SBATCH --time=10:00:00
#SBATCH --error=StarCCM-GPU.%j.error
#SBATCH --output=StarCCM-GPU.%j.output
#SBATCH --mail-type=BEGIN,FAIL,END
#SBATCH --mail-user=username@iastate.edu

cd /work/mygroup/myworkdir/GPU
# load the double precision version:
module load starccm/19.04.007-R8

# This example shows how to use a Java macro file (Mesh.java).  
starccm+ -batch -batchsystem slurm -mpi openmpi -gpgpu auto:1:nomps -np 6 Mesh.java SlurmGPU.sim