New file system and file system policy for CSM HPC platforms

See: http://inside.mines.edu/mio/newpolicy.html for the HTML version of this note.

We are deploying a new file system for Mio and RA. The file system is up but not yet available for use. We are building directory structures and let you know when you can move your data. The file system on BlueM will have the same policies as outlined here.

Along with the new install we are implementing new policies to prevent many of the issues we have seen in recent time with the current file system. This policy is discussed here.

Important notes:

Each user will have three base directories:

In addition a group may have a $SETS directory which is designed for semipermanent data sets that will be used repeatedly by the group. $SETS can contain things like equations of state or velocity fields. It may also contain programs used by multiple members of a group. $SETS will be readable on the compute nodes.

$HOME - Should be kept very small, having only start up scripts and other simple scripts. Output from parallel jobs can not be directed to $HOME. It will be read only from compute nodes..

$DATA - Should contain programs users have built for personal use and small data sets and run scripts. Output from parallel jobs can not be directed to $DATA It will be read only from compute nodes..

$SCRATCH - The main area for running applications. Output from parallel runs should be done to this directory.

$HOME and $DATA will be bigger than they have been in the past but still much smaller that $SCRATCH.

$HOME and $DATA will have quotas:

User Directory Quota
$HOME 20 Gbytes
$DATA 200 Gbytes

$SCRATCH will not have quotas. However, data in $SCRATCH should be cleaned after runs to prevent the the file system from filling.

Users have two options for keeping $SCRATCH clean. Users will be placed in one of two classes: Automatic or Manual. The default class will be Manual. A group's PI can be asked people to be placed in either class by sending email to hpcinfo@mines.edu.

For those users in the Automatic class:
  • Files in $SCRATCH older than 30 days can be automatically purged.
  • For those users in the Manual class:
  • There will be no automatic purging of files.
  • $SCRATCH usage will be monitored an on individual basis. If it is excessive then users will be prevented from running parallel jobs until their $SCRATCH space is cleaned.
  • Monitoring for the users in the Manual class will consist of the system periodically running a command to determine the amount of space being used. Usage will be deemed excessive if on two successive usage checks one of the following occurs:

    1. The amount of space used is greater than a fair share of the available space.
    2. The user has so many files that the amount of space being used can not be determined in a reasonable time.

    Notes:

    Policy with respect to expired accounts

    When a person leaves CSM the PI is responsible for that person must designate a recipient of the persons data. The recipient will be "given" ownership of the data. They will be free to move it or delete it. After 3 months the directories which belonged to the person who has left will be removed.