ANSYS Installations on Tetralith & Sigma

Official homepage: https://www.ansys.com

In order to use ANSYS software, you need a licence. Either you belong to a LiU research group that has a license for this software, or you have to provide your own licence.

ANSYS Documentation

ANSYS provides a customer portal: https://support.ansys.com
We recommend the customer portal to search for documentation, examples, etc. If you have a licence for an ANSYS product, e.g. Fluent or CFX, you probably also have access to the customer portal.

Installed ANSYS Versions

Note: From 2019 on, ANSYS changed the naming of the versions. We apply the official naming convention, e.g. version 2019R1.

ANSYS License

At NSC, we provide ANSYS license information only for a few research groups. If you do not belong to one of these groups, you will see the following warning when loading the module:

Found no valid NSC-ANSYS license for you.
You can use ANSYS, providing your own license.

It is only a warning, that NSC does not provide any ANSYS license to you. In this case, you have to provide correct license information by yourself. Typically, the license server is run by the IT department of your university, e.g. at Chalmers, Uppsala University etc.

To connect to the license server, ANSYS needs the environmental variable ANSYSLMD_LICENSE_FILE. For example

    export ANSYSLMD_LICENSE_FILE=port@servername

port is a four digit number, and servername is the IP-address of the license server.

If the variable ANSYSLMD_LICENSE_FILE cannot be provided, there is a second option:

1. Create a file, that contains the following information, e.g. $HOME/license.ansys:

   SERVER <servername> ANY <FlexNet_port>
   VENDOR ansyslmd
   USE_SERVER
   INCREMENT any ansyslmd any
   

2. Set ANSYSLMD_LICENSE_FILE to point to the license file

   export ANSYSLMD_LICENSE_FILE=$HOME/license.ansys
   

The servername and FlexNet_port are provided by the IT department that runs the license server.

As a reference, we refer to the following article in the Ansys Learning Forum: License file for ANSYS users

How to run

Load the ANSYS module corresponding to the version you want to use, e.g

module load ansys/2021R2-nsc1

This gives you access to all ANSYS software, including Fluent, CFX, ANSYS workbench etc.

How to start the ANSYS workbench interface

ANSYS provides many different software packages. The ANSYS Workbench provides a general graphical interface to access all available software components. ANSYS Workbench can be started by the command:

runwb2

To use graphical hardware acceleration on a login node, we recommend to use vglrun:

vglrun runwb2

ANSYS Fluent

Settings for Parallel Tasks, Intel MPI and Omni-Path Network

Number of Parallel tasks

ANSYS Fluent version before 19.0
Until ANSYS version 19, one has to specify a machine file using the option -cnf=<machinefile>

ANSYS Fluent version after 19.0
From ANSYS version 19 onward, the option -cnf is not used anymore. Instead, one has to specify the number of parallel tasks using the option -t<number of tasks>, e.g. -t32 to use 32 parallel tasks.
In a batch script, it is inconvenient to manually specify the number of tasks, since this information is already specified by the number of nodes/cores, according to the SLURM options (#SBATCH -N, #SBATCH -n).

The number of parallel tasks can be determined from the SLURM options as following:

tasks=$(hostlist --repeat-slurm-tasks="$SLURM_TASKS_PER_NODE" "$SLURM_NODELIST" -e | wc -l)

fluent -t$tasks ...

Intel MPI and Omni-Path

Tetralith nodes are interconnected using Intel Omni-Path network. Our previous system Triolith used Infiniband. Therefore, the correct settings for the interconnect fabric differ between Tetralith and Triolith. If you manually provide the employed MPI and interconnect fabric for your ANSYS application, you might have to adjust them accordingly.

Example for Fluent using Intel MPI and Omni-Path:

-mpi="intel" -pib.infinipath

When using the NSC Fluent wrapper, you do not have to specify the options -mpi,-pib,-t. The wrapper takes care about the correct settings. The description is only intended for completeness and if you do not use our wrapper. The information is useful, when you want to run Fluent on other systems as well.

How to pass MPI Environmental Variables

Several MPI environmental variables can be passed to Fluent via the environmental variable FLUENT_MPIRUN_FLAGS. It is set in the following way:

export FLUENT_MPIRUN_FLAGS="-genv VAR1_NAME VAR1_VALUE -genv VAR2_NAME VAR2_VALUE"

For example, to set Intel MPI debugging information:

export FLUENT_MPIRUN_FLAGS="-genv I_MPI_DEBUG 5"

Fluent Wrapper

Fluent should be started using the NSC wrapper. The NSC wrapper uses the preferred Intel MPI library and the correct settings for Omni-Path fabric, so you do not have to take care about it. That means, you should not add any option -mpi or -pib, since these are added automatically. If you do not use the NSC wrapper, you have to adjust your mpi and interconnect settings as decribed in the previous section. Further, you do not need to specify the number of tasks (option -t). This is automatically done within the wrapper as well.

We provide the environment variable

$NSC_FLUENT_WRAPPER

which is available, after loading the ANSYS module.

Alternatively, you can use the full path:

Fluent 2021R2:  /software/sse/manual/ansys/2021R2/nsc1/fluent_nsc.sh
Fluent 2021R1:  /software/sse/manual/ansys/2021R1/nsc1/fluent_nsc.sh
Fluent 2020R2:  /software/sse/manual/ansys/2020R2/nsc1/fluent_nsc.sh
Fluent 2020R1:  /software/sse/manual/ansys/2020R1/nsc1/fluent_nsc.sh
Fluent 2019R1:  /software/sse/manual/ansys/2019R1/nsc1/fluent_nsc.sh
Fluent 19.0:    /software/sse/manual/ansys/19.0/nsc1/fluent_nsc.sh
Fluent 18.2:    /software/sse/manual/ansys/18.2/nsc1/fluent_nsc.sh
Fluent 17.2:    /software/sse/manual/ansys/17.2/nsc1/fluent_nsc.sh

Note: The NSC_FLUENT_WRAPPER can only be used on a compute node, but not on a login node.

How to run Fluent Interactively

When running the Fluent/ANSYS GUI, NSC recommends using ThinLinc to access Tetralith.
For more information on how to use ThinLinc, please see: Running graphical applications using ThinLinc ThinLinc is avaible for all platforms (Macintosh, Windows, Linux).

Load the ANSYS version that you would like tu use, e.g.

module load ansys/2019R1-nsc1

Start Fluent interactively, using the NSC wrapper script:

$NSC_FLUENT_WRAPPER

Meshing in Fluent

Fluent has an integrated module to create meshes. It can be accessed via the Fluent Launcher, when starting Fluent interactively. In the GUI of the Fluent Laucher, you have to choose: Options -> Meshing Mode. In the section “Processing Options” you have to choose “Parallel (Local Machine)” and adjust the number of Meshing Processes not to be 0. Do not choose “Serial”, even if you are only using one core. The “Serial” mode does not work on Tetralith.

Example Batch Script

To run Fluent in batch mode, it has to be submitted via a SLURM batch script, as the following example illustrates:

#!/bin/bash
#SBATCH -n 64
#SBATCH -t 00:20:00
#SBATCH -J jobname
#SBATCH -A SNIC-xxx-yyy

module load ansys/2021R2-nsc1

JOURNAL=your_fluent_journal_file.scm

$NSC_FLUENT_WRAPPER -gu 3ddp -i $JOURNAL

Submit the script via the command: sbatch <script file>

You have to adjust the batch script as follows:

1. Specify the number of cores via the option #SBATCH -n<cores>

2. Specify the required time #SBATCH -t <time>

3. Change the jobname #SBATCH -J <job name>

4. Specify your account name #SBATCH -A <account name>

5. Adjust the name of your JOURNAL file (testcase)

6. The option “-gu” starts Fluent without a graphical interface

7. The Option “3ddp” starts Fluent in 3D mode, using double precision

A list of available runtime options for Fluent can be accessed via

$NSC_FLUENT_WRAPPER -help

Common Fluent Options

The following options are commonly used as arguments, when starting Fluent in a batch script

Graphical Output in Batch Mode

To run Fluent in batch mode, one can choose the options -g and -gu.

Option: -gu

The option -gu only turns off the GUI. It still allows to call graphics functions, e.g. to generate and export a graphical output of the solution as an image file, e.g. PNG or JPG.

To create images in batch mode, it is important that the Fluent journal file contains the command “/display/set-window 1”. Without this command, it is not possible to output an image in batch mode.

Snippet of a Fluent journal file to export an image in PNG format:

    (ti-menu-load-string (format #f "/display/set-window 1 \n"))
    (ti-menu-load-string (format #f "/display/set/picture/driver png \n"))

Without the command set-window you will get an error message as following:

    (ti-menu-load-string (format #f "/display/save-picture ./solution_iso.png \n"))

    Error: hardcopy-window: No active window.
    Error Object: #f
    /display/save-picture ./solution_iso.png

The save-picture command will fail, since no active window has been set. Running Fluent interactively may work without set-window. This is an inconsistency between interactive mode and batch mode.

Option: -g

The option -g turns off the GUI and does not allow any graphics output. Any command in the journal file that is related to graphics output will fail. For example:

    /display/set-window 1
    Error: No graphics functions are available.

Since the option -gu is less restrictive, we recomment to use -gu instead of -g, since it allows to save images in batch mode.

Potential Problems

1. When running a simulation via the GUI or a batch script, you should choose the same floating point precision, namely single or double precision. In the batch script, the option “3ddp” starts a 3D calculation with double precision mode. In the Fluent Laucher GUI, you find the Option “Double Precision”, which may not be selected by default. This may potentially lead to a different behaviour of your solution process and accuracy when you compute interactively and in batch mode, if you do not choose the same precision.

2. In the Fluent GUI, there are options to check the mesh quality. Under the menu item Mesh , you find the menu items Check and Check Quality to ensure that the mesh quality is okay. An insufficient mesh quality may lead to poor results, slow convergence or even a breakdown of the solution process.

3. Paraview and Ensight Gold Format. In version ANSYS 2019R1, it seems that the export to the ensight gold format has changed. Fluent adds a new section “SCRIPTS” and some metadata information to the ensight gold case file. Paraview does not know these additional options and will complain about it. In this case, you have to remove these lines in the ensight gold case file.

ANSYS CFX

ANSYS CFX is started via the command

cfx5solve

An overview of the available command line options can be obtained via

cfx5solve -help | more

Note on more command: Press spacebar to scroll, type q to exit

Common CFX Options

The following options are commonly used as arguments, when starting CFX

Batch Script Example

#!/bin/bash
#SBATCH -n 64
#SBATCH -t 00:20:00
#SBATCH -J jobname
#SBATCH -A SNIC-xxx-yyy

module load ansys/2020R2-nsc1

# Full path of input file
DEF_FILE=inputfile.def


# Typically, one does not need to change the lines below

CFXHOSTLIST=$(hostlist --repeat-slurm-tasks="$SLURM_TASKS_PER_NODE" "$SLURM_NODELIST" -e | \
              sort | uniq -c | awk '{print $2 "*" $1}' | paste -sd, -)

cfx5solve \
-def $DEF_FILE \
-parallel \
-par-dist "$CFXHOSTLIST" \
-start-method "Intel MPI Distributed Parallel" \
-batch \
-double

Submit the script via the command: sbatch <script file>