How to Set Up and Submit ANSYS Fluent Batch Jobs to Mio from Off-Campus

Although ANSYS gives you the option to run Fluent on your local installation and perform the computations remotely (using Mio, for example), I have found several limitations to this, especially if you are like me and do most of your work from off-campus:

  • Any interruption in your network connection will cause Fluent to fail
  • You need to use VPN to access Mio from off-campus, which is subject to session time restrictions.
  • Overall, Fluent seems to run slower in this configuration. I'm not sure what causes this, but it is something I have noticed.
  1. Set up Mio Account and VPN

  2. Assuming that you have performed an initial set up of your Mio account as described in the Mio user's guide, there are a couple of additional things to know. First you need to establish a means of communicating with Mio from off-campus. To do this you will need to set up a Virtual Private Network (VPN) which you can read about here. You will need to use the Network Connect applet to establish a full tunnel between your computer and the CSM network.

    At a minimum you will need to be familiar with logging in to Mio from the command line. This can be accomplished using the ssh utility in the Terminal application for OSX or Linux users, and the puTTY application for Windows users. However, depending on how much you like or dislike working from the command line, you may want to also use a graphical ftp program such as Cyberduck or WinSCP for file management on your Mio directories. Also, make sure you understand the file system on Mio and it's use and restrictions.

    Once you are comfortable with working on Mio in general, you need to set up a license preference file to indicate that you are using the proper license. This of course will vary depending on the terms of your license. Also, to use Fluent you will need load a set of environmental variables available using the command

    $ module load fluent/121

    or

    $ module load fluent/130

    depending on whether you use version 12.1 or version 13.0. This line can be added to your .bash_profile or .bashrc file so you don't have to type it every time you log in.

  3. Set Up a Case

  4. Since the Fluent graphical interface features a window for text commands, you get the best of both worlds! For this reason, it is helpful to perform the initial case setup in a local ANSYS installation utilizing the GUI. Start out in Workbench, since this seems to be the only way to access the Design Modeler and the Mesh applications. Set up your Fluent case in serial mode (it seems to run better this way) and use it to define things like initial conditions, monitors, solver settings, etc. Once this step is done, save/export the case and close Fluent.

  5. Write a Fluent Journal File
  6. The journal file is basically a series of text commands that tell Fluent what to do. Everything that can be done with the Fluent graphical interface (GUI) can also be done using the text interface (TUI), and Fluent has the capability to open and immediately start performing whatever commands exist in a specified journal file upon startup. Since you've already set up and saved the case, the journal file can be very simple. All it really needs to do is tell Fluent to read the case, initialize the solution, iterate, save the solution, and exit. An example journal file might look like this:

    fluent journal file

    You can create one using a text editor such as TextWrangler (OSX) or Notepad++ (Windows). Make sure you set the file format to Unix for use on Mio. Note that the commands are exactly what you would enter into the text window if you were running Fluent in graphical mode. Comment lines begin with a semicolon. If you are solving this case on a local installation of Fluent, then the case file and a journal file are all that is required, and the problem could be solved by running Fluent from the command line using something like this (Linux/Unix) example:

    $ fluent 3d -g -t8 < example.jou > example.trn

    This would tell Fluent to open in 3d mode, without the GUI, in parallel using 8 processors, execute the commands specified in the journal file example.jou, and write standard output (everything that would appear in the text window if you were running in graphical mode) to the transcript file example.trn.

    (Note: The journal and transcript files are basically text files, and it is fine to use the extension .txt for these instead. Also, the Windows version uses slightly different syntax, not included here since Mio is a Linux/Unix system.)

    If you are submitting this job to Mio, however, you need to perform a few important preliminary steps, as discussed below.

  7. Write a Shell Script File

  8. Fluent needs to be run on compute nodes to take advantage of Mio's parallel computing capabilities and to keep it separate from all the other proccesses in the cluster. This is very important! All computational and memory intensive programs need to be run on compute nodes.

    To do this you need to use Portable Batch System (PBS) commands to define the resources you need for your job and set basic parameters such as the job name, output file path, etc. These commands are submitted to Mio using the Unix utility qsub. Note that an interactive session can be started by adding the -I flag. A simple practice session might look like this:

    qsub interactive session

    The line

    $ qsub -I -V -l nodes=1:ppn=1

    tells Mio to start an interactive session, export all environmental variables to the batch job, reserving one node and one processor on that node. The line

    $fluent 3d -g

    opens Fluent in serial mode without the graphical interface. At this point we are ready to read a case, initialize, iterate, etc. interactively on Mio using text commands. This approach is useful to familiarize yourself with how it works, but is not practical for real jobs, since it requires you to remain logged in while the job executes or a fatal error will occur. Ultimately you want to be able to submit your job and "walk away" from it - allowing you to work on something else or maybe just pack up your computer and take the rest of the day off - all while Mio is still chugging away at the problem and working on getting your results.

    You can accomplish this using a shell script (extension .sh) that includes the PBS commands followed by the fluent execute command, and run it using qsub. Again, you can create one using a text editor that can be set to Unix format. An example might look like this:

    example script file

    The lines beginning with # are comment lines, however note those which begin with #PBS...these lines are actually qsub options, and when utilized in this way act as if they were on the command line following the qsub command. Respectively, they tell Mio to reserve one node and eight processors, set a time limit of one hour for the job, name the job "example", join the standard output and error streams, print the output stream to the file "example.out", and finally to export environmental variables to the batch job. The line

    cd $PBS_O_WORKDIR

    sets the working directory for the output of the batch job to the directory that you start the job from, and finally the fluent execute command is the same example that is used above.

  9. Upload Files to Mio

  10. Now you are ready to upload your files to Mio and start your job. It might be easiest to use a graphical ftp application for this. Start your VPN session and log in to Mio. Navigate to your working directory (probably some subfolder in your SCRATCH directory) and upload your Fluent case, journal, and shell script files. It's probably a good idea to check the load on Mio at this point to see what is available. You can do this via the Mio jobs page.

  11. Starting a Job

  12. If sufficient resources are available, go ahead and log in to Mio using ssh and navigate to the directory containing the files you just uploaded. All that is required now is to submit your job using qsub.

    qsub command

    When youe run qsub, notice that the job number is printed to the console. Once your job is submitted, you can track it's progress by monitoring the transcript file specified in your journal file, since it is updated as Fluent solver progresses. A useful tool here is the Unix tail utility, which can be set to print the last 10 lines of a file and to follow it as it is updated.

    tail command
  13. Stopping a Job

  14. If you notice a problem and need to stop Fluent before it is finished, you can inturrupt it and cleanly exit using the cleanup script for the job. This should have been generated automatically and placed in your working directory. It can be executed with the sh command.

    cleanup commandNote the status of the job before and after running the script, which both stops and exits the Fluent process, and then deletes the cleanup script itself. If you stop a script in this way, no further action is necessary.

  15. Deleting a Job

  16. For some reason, if Fluent finishes by reaching the end of your journal file and there is still time left on the job (as specified by the walltime option), the batch job does not close on it's own. It will continue until the walltime is exceeded. At this point the cleanup script will have been deleted automatically by Fluent, but you probably still want to manually stop the job to free up the nodes and license for others to use. You can do this with the qdel command.

    qdel command
  17. Getting Your Results

  18. Once your job is finished, you should now have a Fluent data file and several output files in your working directory. All you have to do now is use your ftp program to download the data file to your home computer, and you are ready for post-processing!

Questions? Comments?

I wrote this document because I am new at high performance computing, and it took me quite some time and a lot of dialogue with the (very patient and helpful) folks at the Global Energy Computing Organization (GECO) at CSM to figure all this out. Hopefully this will be a useful reference for others in a similar situation. I am no expert, however, and if you find an error in this document please let me know! My email is:

jobush (at) mines.edu

Useful Links