Richard Gilmore's Research: Setting up Fire Dynamics Simulator on HPC

Configuring the working programming environment for FDS

Edit the .bashrc file in your user home directory and add the environment variables for FDS6 to the path variable by adding this line:

    export FDSPATH=/opt/FDS/FDS6

And adding this to the path export line:


We can also add loading the FDS bashrc_fds file, which will include a few other important variables. I use the Linux editor "vim", but you could also download the file edit it, and then upload it back. Here are the commands to get this done with vim.

    Editing .bashrc

Press 'i' to enter -- INSERT -- mode, which will show up at the bottom. Then move the cursor with the arrow keys and type or copy and paste the lines (right-click the mouse button to paste).

Also add the lines to load the modules for the Intel MPI and Programing Environment:

    module load openmpi/intel/1.6.5 >& /dev/null
    module load PrgEnv/intel/default >& /dev/null

Press 'Esc' to exit insert mode, then ':wq' to write the changes and quit. Below is the first few lines of my .bashrc file.

    New .bashrc with FDS path

The file /opt/FDS/FDS6/bashrc_fds adds some other important variables that FDS will need. The MPI Intel environment, the bin directory that contains the FDS executables, the infiniband network that FDS will communicate to other nodes through, add to the LD_LIBRARY_PATH variable /opt/FDS/FDS6/bin/LIB64, and sets the variable for OMP_NUM_THREADS to 4. Below are the contents of that file:

    bashrc_fds source file

This concludes setting up the environment for FDS and now we are ready to setup the SBATCH SLURM script that will be submitted to the job scheduler. Copy the premade script from the FDS6 install directory to your bins directory and edit the file.

    Copying FDS Hybrid Script

Looking at the top 8 lines the SLURM scheduler starts an executable bash session with '-x' flag, then the following lines specify parameters that are read by SBATCH. The first parameter gives a job name that is shown in 'squeue' for other users to read. This is followed by the number of compute nodes, the number of MPI task per node, and the total number of MPI task. The exclusive flag insures that no other jobs will use unused resources on the node. Remove this flag if you want to run several FDS jobs on the same node. The export=ALL flag passes the environmental variables we setup earlier to the running job environment, and finally, the maximum time is specified before the SLURM job scheduler kills the running job.

    Remove lines in FDS Hybrid Script

Remove the lines 10-15 of the code, because we already placed them previously into your .bashrc file. Once again I'm using vim here, but this time I move the cursor to line 10, then press 'v' to enter -- VISUAL -- mode and move the cursor to the end of line 15. Then by pressing 'd' these lines will be deleted. If you make a mistake press 'u' to undo.

Next edit the end of line 35 and change the 'tail -1' command to this 'sed' command:

    sed "s/[^0-9]//g"

The little tick mark at the end and near the begin are not single quote marks, but rather back-ticks found on the tilde, ~ key, without pressing the shift key.

    Hybrid script change tail to sed

Lines 44-46 we will leave unchanged for now, but later you will need to change them to your FDS file name and have them copy from your scratch/runs directory.

Further down on line 56 is where the FDS program is started. The line begins with 'srun' and starts the EXE variable that we set for fds_impi, inputs the TEST.fds to FDS, and directs the output from standard error stream #2 to standard out, '2>&1', and saves it to the file output appended with the job ID number '> output.JOBID

Now you are ready to write and quit ':wq' and test running the Flowflelds FDS example.

Running FDS

From you home directory move to your scratch directory and make a new directory call 'fds-test', then start your SLURM batch program by typing 'sbatch' followed by the scripted name. If you successfully submitted the job you will get the job number on the next line. In this case my job number is 1439523, and the directory with this number contains all the files.

    FDS-test RUN

Entering the directory 1439523 and getting a listing of all the files when the job is running you will get a whole bunch of files including the output.1439523 file, and the symmetry_test_mpi.out file. It may take a while for the directory list to show up because of the number of files and they are constantly increasing in size.

    FDS files

To watch the FDS program run you can go into the JOBID directory and look at the output file. Here I use the 'tail -f' command to follow the output file as FDS runs. The output is the time step number and simulated time. Press 'Ctrl-C' to exit the tail command. To stop the simulation gracefully you can create a file named symmetry_test_mpi.stop and FDS will save and exit. Or you can use SLURM to kill the job with by typing 'scancel' follows by the JOBID.

    FDS tail and watch job

When the job is complete the end of the output file will look like this:

    FDS completed job

Optimizing an FDS Job

Make a new directory in your scratch directory and copy or upload the FDS file you want to run into it. Here I still copying the same FDS example, but this time I'm going to run it on more nodes and find the optimum combination of nodes and threads. Changing the SBATCH variable nodes to 4 will have the job request 4 nodes, and changing the ntasks-per-node to 2 MPI per node for the total of 8 MPI tasks. More instructions for testing node and thread combinations are at the bottom of this script.

    Node and MPI changes

If you get all 8-core nodes (see the Mio nodes configuration page) then the OMP_NUM_THREADS calculated by lines 35-36 will be 4, however, if you get 12, 16, 20 core machines then it will be 6, 8 and 10, respectively. This adversely affects the speed of the program as shown the graph. Where I have made request for a mixture of nodes that have unbalanced core counts or over requested threads the performance plummets. The optimal settings with the available 8-meshes in the FDS file is the 4 nodes with 8 cores (4x8) using 8 MPI processes (8-cores), with 4 threads per MPI process (4-threads). Once I change the number of available meshes to 64 you can see that again the 4-threads per MPI process is optimal. However, the mesh size also had to change to make this a functional FDS file.

    Optimizing FDS to Number of Nodes and Threads

Setting up to run your own FDS files

Given that FDS runs optimally with 4-threads per MPI process the default setting placed in the bashrc_fds file loaded by your .bashrc should be retained and the lines in the hybrid_script file should not be needed anymore. Remove them by adding a pound sign '#' in front of each of the lines. Also since you are running your own FDS file the copying of the example is no longer needed, but instead copy the FDS file into the JOBID directory.

Change the line to (that is a space and period at the end):

    cp ../$TEST.* .

Now, just change the 'TEST=' line to the name of your FDS CHID and upload this script and your .fds file to a working directory in your scratch and you are all set.


Useful Links