Linux for HPC
Timothy H. Kaiser, Ph.D.
Spring 2014


Linux for HPC
• Overview
• File system (more details next slide)
• Logging in
• ssh
• Some tools for doing ssh
• What happens
• Environment
• what it effects
• how it gets set
• how to change it
• modules


Files Linux, more details
• The file system
• moving around
• listing
• hidden files
• “wildcards”
• deleting
• creating/removing directories
• creating files
• touch
• edit
• pipe
• redirect


Editing
• gedit (GUI)
• gvim (GUI)
• emacs (GUI)
• nano
• Remote editing
• Not covered
• vi
• emacs


Creating programs
• compilers
• make
• configure
• cmake


Advanced ssh
• Setting up keys
• .ssh/config file
• alias
• Specific configurations
• Tunneling
• External tunneling


HPC and Paral el Programming
• What is it?
• How do you run paral el


Not covered…
• Running HPC applications
• Shell programming…
• See: http://geco.mines.edu/scripts/
• This needs to be updated for our current
system
• Skip everything between “Running Bash Scripts”
and “Redirection”


Let's get going. .


Operating Systems
• What is an operating system
• What a user uses to use the computer
• It is the interface between a user and the computer
• Controls the computer
• Presents “stuff” to the user
• Interface
• GUI
• Text
• Voice/Sound


From Unix to Linux
• The Unix operating system was conceived and
implemented by Ken Thompson and Dennis Ritchie
(both of AT&T Bell Laboratories) in 1969 and first
released in 1970.
• The History of Linux began in 1991 with the
commencement of a personal project by a Finnish
student, Linus Torvalds, to create a new free operating
system kernel.
• Linux has been ported to al major platforms and its
open development model has led to an exemplary pace
of development.
• http://en.wikipedia.org/wiki/History_of_Linux


Many different “versions”
• Linux
• http://en.wikipedia.org/wiki/Linux_distribution
• Linux/Unix like:
• Mach (OSX - IOS)
• AIX
• Unicos
• HP-UX


Some cool things
• Linux has many smal tools that can be combined
to do complex tasks
• You can write simple programs “scripts” to
automate common tasks
• You learn Linux you wil :
• Be better at working in OSX
• Be comfortable at just about any HPC site
• Windows?
• Built in help for most commands


Some links
• Tutorials:
• http://www.tutorialspoint.com/unix/index.htm
• http://www.ee.surrey.ac.uk/Teaching/Unix/
• https://www.cac.cornell.edu/VW/Linux/default.aspx?id=xup_guest
• http://til e.garrels.be/training/bash/
• See: http://geco.mines.edu/scripts/
• General Interest
• http://en.wikipedia.org/wiki/History_of_Linux
• http://en.wikipedia.org/wiki/Linux_distribution


Basic Interactions
• We wil be talking mostly about text based interactions
• Even text based interactions need a terminal window
• Linux - X11 based
• OSX - Terminal
• Windows
• Putty
• Cygwin
• ZOC
• Terminal programs
• http://en.wikipedia.org/wiki/List_of_terminal_emulators


Files Linux, more details
• The file system
• moving around
• listing
• hidden files
• “wildcards”
• deleting
• creating/removing directories
• creating files
• touch
• edit
• pipe
• redirect


File System
• Tree structure for storing files:
• Most GUI interfaces to a file system show the
structure as a col ection of folders and subfolders
• The folders are a visual depiction of a “Directory”
• On a text based interface the structure is shown as
a list of files and directories
• Directories can contain files and more directories.


Directory tree structure
• In a text based interface you are always
somewhere in the tree structure
• The base of the structure is /
• “/“ is a directory that contains the whole tree
• There are commands for moving around, the
structure and viewing/creating/deleting/moving/
changing/listing “stuff”
• Users have a home directory which the system
puts them in when they login


The file system
• moving around
• listing
• “wildcards”
• hidden files
• deleting
• creating/removing directories
• creating files
• touch
• edit
• pipe
• redirect


Moving around
• The primary command for moving around in the
file system is “cd” - change directory
• cd . Don’t go anywhere “.” is the current
directory
• cd . Go up one level “.” is up one . /. = two levels
• cd ~ Go to your home directory
• cd adir Go to a subdirectory
• cd adir/bdir Go to a subdirectory two levels down
• cd /u/pa/ru/tkaiser Go to an absolute location


Listing what’s in a directory
• ls is the primary command for listing files in a
directory
• Has many options
• ls by itself just gives names


Some ls options
-a, --all
do not ignore entries starting with .
! --color
colorize the output.
! -l use a long listing format
! -F, --classify
append indicator (one of */=>@|) to entries
! -r, --reverse
reverse order while sorting
! -R, --recursive
list subdirectories recursively
! -s, --size
print the allocated size of each file, in blocks
! -S sort by file size
! -X sort alphabetically by entry extension


ls -X
[tkaiser@mc2 ~]$ ls -X!
after lib slurmnodes tintel.f90!
ALIAS local stack bgxlc.html!
allinea makefile tau bgxlf.html!
atest mc2 testddt mod.html!
aun mc2_script helloc.c smap.html!
auto_launch_tag onmc2 Futil_getarg.f util_getarg.o!
b4 privatemodules get_inp_file.F a.out!
bashrc remote util_getarg.F ddt.out!
bin scratch util_getenv.F batch.qtf!
bins scripts color.f90 1009_210203vestalac1.tgz!
ddt-script serial docol.f90 example.tgz!
getnew setbonk dosin.f90!
hist setbonk2 f90split.f90!
HPM slurmjobs junk.f90!
[tkaiser@mc2 ~]$ !


ls -a
[tkaiser@mc2 ~]$ ls -a!
. bins .history scripts!
.. .cache HPM serial!
1009_210203vestalac1.tgz color.f90 junk.f90 setbonk!
after .config .kshrc setbonk2!
ALIAS .dbus .lesshst slurmjobs!
allinea ddt.out lib slurmnodes!
.allinea ddt-script local smap.html!
a.out docol.f90 .local .ssh!
atest dosin.f90 makefile stack!
aun .emacs mc2 .subversion!
auto_launch_tag example.tgz mc2_script tau!
b4 f90split.f90 mod.html testddt!
.bash_history .fontconfig .mozilla tintel.f90!
.bash_logout Futil_getarg.f .nedit util_getarg.F!
.bash_profile .gconf onmc2 util_getarg.o!
bashrc .gconfd .pki util_getenv.F!
.bashrc get_inp_file.F privatemodules .vim!
batch.qtf getnew .qt .viminfo!
bgxlc.html .gnome2 .recently-used.xbel .Xauthority!
bgxlf.html helloc.c remote!
bin hist scratch!
[tkaiser@mc2 ~]$ !


ls -R
[tkaiser@mc2 ~]$ ls -R scripts!
scripts:!
do1d example mc2_script serial set1!
!
scripts/example:!
aun_script docol.f90 helloc.c mc2_old_script!
color.f90 example.tgz makefile mc2_script!
!
scripts/serial:!
728 a.out fort_000001 fort_006792 mc2_script slurm-728.out small.py!
729 do_thread fort_006762 hello.f90 simple_test slurm-729.out!
!
scripts/serial/728:!
bonk.out env.728 script.728 submit!
!
scripts/serial/729:!
bonk.out env.729 script.729 submit!
[tkaiser@mc2 ~]$ !


ls -l
[tkaiser@mc2 ~]$ ls -l scripts/serial/!
total 11152!
drwxrwxr-x 2 tkaiser tkaiser 512 Jan 8 14:17 728!
drwxrwxr-x 2 tkaiser tkaiser 512 Jan 8 14:18 729!
-rwxrwxr-x 1 tkaiser tkaiser 6453905 Dec 23 10:44 a.out!
-rw-rw-r-- 1 tkaiser tkaiser 2216 Jan 8 14:18 do_thread!
-rw-rw-r-- 1 tkaiser tkaiser 43 Dec 23 10:44 fort_000001!
-rw-rw-r-- 1 tkaiser tkaiser 29 Dec 23 10:44 fort_006762!
-rw-rw-r-- 1 tkaiser tkaiser 43 Dec 23 10:44 fort_006792!
-rw-rw-r-- 1 tkaiser tkaiser 350 Dec 23 10:44 hello.f90!
-rw-rw-r-- 1 tkaiser tkaiser 2492 Dec 23 10:44 mc2_script!
-rwxrwxr-x 1 tkaiser tkaiser 4926419 Dec 23 10:44 simple_test!
-rw-rw-r-- 1 tkaiser tkaiser 2184 Jan 8 14:17 slurm-728.out!
-rw-rw-r-- 1 tkaiser tkaiser 2194 Jan 8 14:18 slurm-729.out!
-rwx------ 1 tkaiser tkaiser 351 Dec 23 10:44 small.py!
[tkaiser@mc2 ~]$ !


Link
[tkaiser@mc2 561]$ ls -l!
total 192!
-rw-rw-r-- 1 tkaiser tkaiser 4670 Dec 23 14:13 env.561!
-rw-rw-r-- 1 tkaiser tkaiser 2470 Dec 23 14:13 script.561!
-rw-rw-r-- 1 tkaiser tkaiser 4303 Dec 23 14:13 srun_1!
-rw-rw-r-- 1 tkaiser tkaiser 1818 Dec 23 14:14 srun_4!
-rw-rw-r-- 1 tkaiser tkaiser 2135 Dec 23 14:14 srun_8!
lrwxrwxrwx 1 tkaiser tkaiser 13 Dec 23 14:13 submit -> /bins/tkaiser!
-rw-rw-r-- 1 tkaiser tkaiser 657 Dec 23 14:13 tests!
[tkaiser@mc2 561]$ !


* is a wildcard for al file operations
[tkaiser@mc2 561]$ ls!
env.561 script.561 srun_1 srun_4 srun_8 submit tests!
[tkaiser@mc2 561]$ ls srun*!
srun_1 srun_4 srun_8!
[tkaiser@mc2 561]$ !
[tkaiser@mc2 561]$ !
[tkaiser@mc2 561]$ ls *561!
env.561 script.561!
[tkaiser@mc2 561]$ !


Set the accessibility for a file
• Flags:
• 1 - an executable program or script
• 2 - readable
• 4 - writable
• These can be added
• 1+2+4=7= an executable program or script that is readable
and writalbe
• There are 3 flags
• You
• Group
• Everyone


Setting accessibility
• chmod 700 afile
• afile is an excitable program that only you can run, read, or
change
• chmod 755 afile
• afile is an excitable program that anyone can run but only
change 5=4 (run) +1 (read)
• Strange but for someone to see a directory it must have settings
of at least 5 or 7
• chmod 640 afile
• You - read/change
• Your group read
• Everyone else - nothing


Creating files
• Touch afile
• Creates and empty file cal ed “afile”
• Piping
• The “>” symbol “redirects” the output of a command into a
file
• “>>” appends output
• date > alisting
• ls -lt /bin >> alisting
• cp - makes a copy of a file
• mv - rename or move a file


Seeing files
• The file command tells what type a file you have


[tkaiser@bluem ~]$ file alisting
alisting: ASCII text
[tkaiser@bluem ~]$
• If a file is a text file you can do the fol owing
• cat - types the whole file
• head - end of a file
• tail - beginning of a file
• less - page through a file (q to end)
• more - similar to less (q to end)
• sort - sorts a file


More Piping < , | , >&
• The | between two Linux commands means to take the
output from the first command and use it as input to the
second command
• cat alisting | sort
• The < between a command and a file means to use
the file as input for a command
• sort < listing
• >& puts errors from a command to a file, /dev/null is
the “bit bucket”
• command >& errors
• command >& /dev/null


Removing Files
• The command for removing files is “rm”
• Syntax
• rm anoldfile
• Removes the anoldfile
• rm *f90
• Removes al files ending in f90
• rm -rf adir afile
• -r recursive remove (directories also)
• -f don’t give and error if the file does not exist


Online manual pages
man rm
RM(1) User Commands RM(1)!
!NAME!
rm - remove files or directories!
!SYNOPSIS!
rm [OPTION]... FILE...!
!DESCRIPTION!
This manual page documents the GNU version of rm. rm removes each!
specified file. By default, it does not remove directories.!
! If the -I or --interactive=once option is given, and there are more!
than three files or the -r, -R, or --recursive are given, then rm!
prompts the user for whether to proceed with the entire operation. If!
the response is not affirmative, the entire command is aborted.!
! Otherwise, if a file is unwritable, standard input is a terminal, and!
the -f or --force option is not given, or the -i or --interac-!
tive=always option is given, rm prompts the user for whether to remove!
the file. If the response is not affirmative, the file is skipped.!
!OPTIONS!
…!
…!
man -k can be used to search for commands


A few cool commands
• echo
• just write something a string or variable
• date
• date - can format it
• sed
• read a file and write a new one with changes
• nslookup
• find an address associated with a machine name


A few cool commands
• sort
• alias
• make an alias for a command
• export
• set a variable
• which
• tells the path to a command that you might run
• wget
• download something from a given http (web) address


ssh
• Command for getting on a Linux box from another
• Basic syntax from a terminal window:
• ssh machine_name
• ssh bluem.mines.edu
• ssh username@machine_name
• ssh tkaiser@bluem.mines.edu
• ssh username@machine_name command
• ssh tkaiser@bluem.mines.edu ls
• To enable a remote machine to open a local window you
need to add the -Y option
ssh -Y bluem.mines.edu


Digression:
• There are a number of GUI clients that wrap ssh
• Windows - putty
• GUI for making connections
• GUI for set up
• Also provides a “normal” terminal window
• Instructions
• http://geco.mines.edu/ssh/puttyra.html
• Firefox plugin FireSSH (very cool)


ssh
• Reads a local configuration file ~/.ssh/config (if it exists)
• Alias
• Special password settings
• Tunnels
• Sets up an encrypted connection between your local and
remote machines
• “Normal y” asks for a password (MultiPass)
• Opens up a session on the remote host in which you can
enter commands
• Type exit to quit


The first time you go to a new machine…
• You maybe asked to set up ssh keys
• Usual y just hit return for the default selection
• More on this later
• First time you go to bluem.mines.edu you may be
asked to transfer keys
• This sets up easy access to MC2 and AuN
• Wil ask you for your (MultiPass) password
• Your normal “campus” password


Let’s Do It


• ssh -Y bluem.mines.edu
• ls
• ls -a
• ls /


The Environment
• You interact with the machine via a program cal ed
the shell
• Several shell programs: csh, tcsh, zsh, bash…
• We wil be using bash
• When bash starts up it reads several files to set up
the environment
• .bashrc - “sourced” when you start bash
• .bash_profile - “sourced” when you login


The Environment
• The environment is “set up” by setting various
environmental variables
• The fol owing commands wil show what is set
• export
• printenv
• The difference is that “export” shows them in a
form that can be reused and export can also be
used to set a variable


Setting a variable
osage:~ tkaiser$ export BONK="abcd"!
osage:~ tkaiser$ printenv BONK!
abcd!
!
osage:~ tkaiser$ echo $BONK!
abcd!
!
!
!
!
declare can also set variables
osage:~ tkaiser$ declare -x BONK="12345"!
osage:~ tkaiser$ printenv BONK!
12345


[tkaiser@mc2 ~]$ export!
declare -x HISTSIZE="1000"!
declare -x HOME="/u/pa/ru/tkaiser"!
declare -x HOSTNAME="mc2"!
declare -x INCLUDE="/bgsys/drivers/ppcfloor/comm/include:/opt/ibmcmp/xlf/bg/14.1/include:/
opt/ibmcmp/vacpp/bg/12.1/include"!
declare -x LANG="en_US.UTF-8"!
declare -x LD_LIBRARY_PATH="/bgsys/drivers/ppcfloor/comm/lib:/opt/ibmcmp/xlf/bg/14.1/
lib64:/opt/ibmcmp/vacpp/bg/12.1/lib64"!
declare -x LIBRARY_PATH="/bgsys/drivers/ppcfloor/comm/lib:/opt/ibmcmp/xlf/bg/14.1/lib64:/
opt/ibmcmp/vacpp/bg/12.1/lib64"!
declare -x LOADEDMODULES="PrgEnv/IBM/VACPP/12.1.bgq:PrgEnv/IBM/XLF/14.1.bgq:PrgEnv/IBM/
default:PrgEnv/MPI/IBM/default:Core/Devel"!
declare -x LOGNAME="tkaiser"!
declare -x MANPATH="/opt/ibmcmp/xlf/bg/14.1/man/en_US:/opt/ibmcmp/vacpp/bg/12.1/man/
en_US:/usr/share/man"!
declare -x MODULEPATH="/usr/share/Modules/modulefiles:/etc/modulefiles:/opt/modulefiles"!
declare -x MODULESHOME="/usr/share/Modules"!
declare -x MPI_BIN="/bgsys/drivers/ppcfloor/comm/bin/xl"!
declare -x MPI_COMPILER="mpicc"!
declare -x MPI_HOME="/bgsys/drivers/ppcfloor/comm"!
declare -x MPI_INCLUDE="/bgsys/drivers/ppcfloor/comm/include"!
declare -x MPI_LIB="/bgsys/drivers/ppcfloor/comm/lib"!
declare -x MPI_SUFFIX="_mpich"!
declare -x OLDPWD!
declare -x PATH="/bgsys/drivers/ppcfloor/comm/bin/xl:/opt/ibmcmp/xlf/bg/14.1/bin:/opt/
ibmcmp/vacpp/bg/12.1/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/
sbin:/usr/sbin:/sbin:/u/pa/ru/tkaiser/bin"!
declare -x PWD="/u/pa/ru/tkaiser"!
declare -x SCRATCH="/scratch/tkaiser"!
declare -x SHELL="/bin/bash"!
declare -x USER="tkaiser"!


Important variables
• PATH
• Where to look for programs to run
• LD_LIBRARY_PATH
• Where to look for libraries to use when
running programs
• MANPATH
• Where to look for man pages


Setting up your environment
• You can run “export” form the command line
• If you want to have an environment set every time
you login or start bash you set that in
• .bashrc
• .bash_profile


Example…
• Say I want
• PATH to include ~/bin and “.”
• LD_LIBRARY_PATH to include ~/lib


Example…
osage:~ tkaiser$ ssh petra!
tkaiser@petra's password: !
Last login: Wed Mar 12 08:59:58 2014 from osage.mines.edu!
[tkaiser@petra ~]$ printenv PATH!
/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin!
[tkaiser@petra ~]$ printenv LD_LIBRARY_PATH!
[tkaiser@petra ~]$!
Original .bashrc
Becomes
# .bashrc!
!
# .bashrc!
# Source global definitions!
!
if [ -f /etc/bashrc ]; then!
# Source global definitions!
. /etc/bashrc!
if [ -f /etc/bashrc ]; then!
fi!
. /etc/bashrc!
!
fi!
# User specific aliases and functions!
!
!
# User specific aliases and functions!
export PATH=.:~/bin:$PATH!
!
export LD_LIBRARY_PATH=~/lib!


On next login…
osage:~ tkaiser$ ssh petra!
tkaiser@petra's password: !
Last login: Wed Mar 12 08:59:58 2014 from osage.mines.edu!
osage:~ tkaiser$!
osage:~ tkaiser$!
!
[tkaiser@petra ~]$ printenv PATH!
.:/home/tkaiser/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin!
osage:~ tkaiser$!
osage:~ tkaiser$!
[tkaiser@petra ~]$ printenv LD_LIBRARY_PATH!
/home/tkaiser/lib!
[tkaiser@petra ~]$ !


Modules
• Some systems (bluem, Mio001, AuN, Mc2) have sets
of variables combined into modules
• To see what modules are available run
• module avail
• To load the set you run the load module command:
• module load
• You can put module load commands in .bashrc


Modules
[tkaiser@mio001 ~]$ module avail!
!
———————— /usr/share/Modules/modulefiles ————————!
dot module-cvs module-info modules null use.own utility!
!
———————— /opt/modulefiles ————————!
PrgEnv/intel/13.0.1 impi/gcc/4.1.1!
PrgEnv/intel/default impi/intel/4.1.1!
PrgEnv/libs/fftw/gcc/3.3.3 openmpi/gcc/1.6.5!
PrgEnv/libs/fftw/intel/3.3.3 openmpi/gcc/default!
PrgEnv/libs/opencl/1.2 openmpi/intel/1.6.5!
PrgEnv/python/Enthought/2.7.2_v7.1-2 openmpi/intel/1.6.5_test!
ansys/fluent/15.0 openmpi/intel/default!
[tkaiser@mio001 ~]$ !
[tkaiser@mio001 ~]$ !
!
[tkaiser@mio001 ~]$ which mpicc!
/opt/openmpi/1.6.5/intel/bin/mpicc!
[tkaiser@mio001 ~]$ !
[tkaiser@mio001 ~]$ !
!
[tkaiser@mio001 ~]$ module load impi/intel/4.1.1!
[tkaiser@mio001 ~]$ which mpicc!
/opt/intel/impi/4.1.1.036/intel64/bin/mpicc!
!


Moving files to/from machines
• scp - secure copy
• related to ssh
• Full syntax
• scp source destination


scp username@machine:path_to_file username@machine:path_to_file
• Can usual y shorten this
• I have a helpful utility /opt/utility/scpath


Shorter forms
• scp afile tkaiser@bluem:/u/pa/ru/tkaiser/tmp
• copy a local file to bluem
• scp afile bluem:/u/pa/ru/tkaiser/tmp
• same as above
• scp afile bluem:~
• copy to home directory
• scp bluem:/u/pa/ru/tkaiser/tmp/afile .
• copy from bluem to local directory
• scp -r bluem:/u/pa/ru/tkaiser/tmp .
• copy a complete directory to your local directory


A useful utility
• /opt/utility/scpath
• Gives full paths for scp
• scp then becomes a copy/past activity
[tkaiser@bluem bins]$ ls!
abinit amber examples.tgz gromacs matrix nwchem quick wu!
abinit-6.10.2 enthought fft grow memory petsc siesta!
acc.tgz examples fftw guide nbody ppong utility!
[tkaiser@bluem bins]$ !
[tkaiser@bluem bins]$ !
[tkaiser@bluem bins]$ scpath!
tkaiser@bluem:/u/pa/ru/tkaiser/remote/aun/bins!
[tkaiser@bluem bins]$ !
[tkaiser@bluem bins]$ !
[tkaiser@bluem bins]$ scpath amber!
tkaiser@bluem:/u/pa/ru/tkaiser/remote/aun/bins/amber!
[tkaiser@bluem bins]$ !
[tkaiser@bluem bins]$ scpath *tgz!
tkaiser@bluem:/u/pa/ru/tkaiser/remote/aun/bins/acc.tgz!
tkaiser@bluem:/u/pa/ru/tkaiser/remote/aun/bins/examples.tgz!
[tkaiser@bluem bins]$ !


scp GUI clients
• These are very useful in the context of editing a
file
• There are a number of good ones:
• WinSCP (Windows)
• Filezil a (cross platform)
• FireFTP (Firefox extension) See: FireSSH also
• Yummy (OSX)
• Fetch (OSX)


Editing
• nano
• gedit (GUI)
• gvim (GUI)
• emacs (GUI)
• Remote editing
• Not covered
• vi (Available on every Linux box)
• emacs (Text based version of emacs)


Nano
• Text based editor
• Relatively easy to use
• Online help
• Copy at http://hpc.mines.edu/nano.html
• ^ - implies the control key


Nano Important commands
• ^w - find
• ^\ - find and replace
• ^k - cut text (aline or marked text)
• ^u - paste text
• ^^ - mark text for cutting. In this case the second ^
is not the control character but the “real” ^, usual y
shift 6


GUI base editors
• These are available on Mio,BlueM, AuN, Mc2
• gedit
• gvim
• emacs
• Require X-windows but gvim and emacs wil fal
back to a text version
• Launch them in the background with the &
option and put errors into /dev/null
[tkaiser@bluem ~]$ gedit alisting >& /dev/null &!








Remote Editing
• Idea: Copy the file to your desktop machine and
edit it local y. Then send it back
• Some (most) of the scp GUI clients support this
almost automatical y
• Example with:
• Filezil a
• Yummy (OSX)
• FireFTP (not 100% automatic)


Remote Editing
• For FireFTP I set up a special directory in which I
am doing edits.
• Copy file
• Do Edits
• Copy back
• For Yummy and Filezil a you can double click on a
file to edit it.
• May need to select your local editor
• After that, it is automatic


Back to ssh


Local ssh pages
• Setting up ssh, including putty
• http://geco.mines.edu/ssh/
• Tunneling
• http://geco.mines.edu/ssh/tunneling.html
• http://hpc.mines.edu/bluem/transfer.html#scp
• http://hpc.mines.edu/bluem/multistage.html


ssh keys
• Setting up keys
• Keys are like two part passwords
• Private part - on the machine you are coming
from
• Public part - on the machine you are going to
• You can give someone your public key
• They put it on a machine in:
• ~.ssh/authorized_keys
• You now have access


Private key - Pass Phrase
• Private keys have a pass phrase which must be
entered to al ow its use
• Can have a pass phrase that you enter like a
password every time
• Can have a blank pass phrase which wil al ow
getting on to a machine without needing a
password. (This is a lot more common than you
think.)
• Can enter a pass phrase with a timeout feature
• Once a pass phrase is validated you can use it on al
machines that have the public key


Set up keys and copy them to “bluem”
Get the address for bluem
osage:.ssh tkaiser$ nslookup bluem!
Server:! !

138.67.1.2

!
Address:!138.67.1.2#53!
!
Name:!bluem.mines.edu!
Address: 138.67.132.239!
Create our ~/.ssh/config with
the fol owing lines
Host bluem bluem.mines.edu
HostName 138.67.132.239
User tkaiser
Identityfile2 ~/.ssh/brock


More on keys…
• The command to generate a key set is ssh-keygen
• -t option tells what “type” of key
• ssh-keygen -tdsa
• Keys are normal y stored in a hidden directory
~/.ssh
• You can give a key set a non-default name
• You can associate a key set with a machine in
the file ~/.ssh/config


A slight digression - .ssh/config
Host petra petra.mines.edu !
HostName 138.67.4.29!
User tkaiser!
Identityfile2 ~/.ssh/arock!
When you ssh to petra or petra.mines.edu you:
!
• Connect to a machine at 138.67.4.29
• Username is tkaiser
• Use the keys found in ~/.ssh/arock


Set up keys and copy them to “bluem”
Copy the public key to bluem
cd ~/.ssh!
cat brock.pub | ssh bluem.mines.edu "cat >> .ssh/authorized_keys!
tkaiser@138.67.132.239's password: !
osage:.ssh tkaiser$ !


Set up keys and copy them to “bluem”
Create a key set:
osage:.ssh tkaiser$ ssh-keygen -tdsa!
Generating public/private dsa key pair.!
Enter file in which to save the key (/Users/tkaiser/.ssh/id_dsa): /Users/tkaiser/.ssh/brock !
Enter passphrase (empty for no passphrase): !
Enter same passphrase again: !
Your identification has been saved in /Users/tkaiser/.ssh/brock.!
Your public key has been saved in /Users/tkaiser/.ssh/brock.pub.!
The key fingerprint is:!
e9:bb:b4:20:2c:af:fc:5c:4d:e7:c4:50:3c:82:db:64 tkaiser@osage.mines.edu!
osage:.ssh tkaiser$!
!!
osage:.ssh tkaiser$ ls -lt brock*!
-rw------- 1 tkaiser staff 751 Mar 12 11:24 brock!
-rw-r--r-- 1 tkaiser staff 613 Mar 12 11:24 brock.pub!
osage:.ssh tkaiser$ !
!!


Next time you login…
• You wil be asked for a pass phrase instead of a
pass word
• What has this bought you?
• You can validate a key for some time and you
wil not need to reenter it until the time
expires
• This validates a key for 8 hours:

keys ~/.ssh/brock


Here it is…
osage:.ssh tkaiser$ ssh-add -t 28800 ~/.ssh/brock!
Enter passphrase for /Users/tkaiser/.ssh/brock: !
Identity added: /Users/tkaiser/.ssh/brock (/Users/tkaiser/.ssh/brock)!
Lifetime set to 28800 seconds!
!!!!
osage:.ssh tkaiser$ ssh bluem!
Last login: Wed Mar 12 11:30:23 2014 from osage.mines.edu!
[tkaiser@bluem ~]$ !
You wil want to add the fol owing to your .bashrc file
alias keys="ssh-add -t 28800 ~/.ssh/brock"!
alias killkeys="ssh-add -D"!


More ~/.ssh/config magic
• Motivation
• Make your life easier
• Tunneling:
• Get to one machine by going through another
• Second machine might only be accessible via the first
• Mc2 and AuN can only be seen from bluem
• Machine might be behind a firewal
• Mio, BlueM
• Most campus machines


A simple tunnel
Host golden!
ProxyCommand ssh bluem.mines.edu nc 2>/dev/null aun.mines.edu %p!
!
Host energy!
ProxyCommand ssh bluem.mines.edu nc 2>/dev/null mc2.mines.edu %p!
• ssh to golden goes to bluem and then is forwarded to aun.mines.edu
• ssh to energy goes to bluem and then is forwarded to mc2.mines.edu


Getting to bluem from off campus
This would go on your machine at home:
Host bluem!
ProxyCommand ssh imagine.mines.edu nc 2>/dev/null bluem.mines.edu %p!
• ssh to bluem from off campus goes to
imagine.mines.edu and then is forwarded to
bluem.mines.edu


We can combine tunnels
This would go on your machine
at home to get to AuN or Mc2
Host energy2!
ProxyCommand ssh step2 nc 2>/dev/null mc2.mines.edu %p!
!
Host golden2!
ProxyCommand ssh step2 nc 2>/dev/null aun.mines.edu %p!
!
Host step2!
ProxyCommand ssh step1 nc 2>/dev/null bluem.mines.edu %p!
!
Host step1!
Hostname imagine.mines.edu
• ssh to energy2 from off campus goes to imagine.mines.edu and
then is forwarded to bluem.mines.edu then final y to
mc2.mines.edu


Automatical y forward X11 connections
ForwardAgent yes
ForwardX11 yes


An obscure feature
Host bluem bluem.mines.edu !
HostName 138.67.132.239!
User tkaiser!
Identityfile2 ~/.ssh/brock!
ControlMaster auto!
ControlPath /Users/tkaiser/.ssh/tmp/%h_%p_%r!
After you have one login session open to a machine any
new connections wil get piped transparently through the
first connection.
!
If you have two part authentication this might save you
some work


Building Programs
• Compilers
• make
• configure
• cmake


Compilers - build programs from source
• Primary language of HPC
• Fortran (90,2000,2003,77)
• C
• C++
• There are special versions of these for paral el
applications
• Need to match the machine


X86 compilers (Mio/Aun)
• Intel
• Portland
• ifort
• pgf77,pgf90,
ppgf95,
• icc
• pgc
• icpc
• pgc++
• gnu
• NAG
• gfortran
• nagfor
• gcc
• g++


Power Compilers (Mc2)
IBM Fortran Compilers:
IBM "C" Compilers:
• gnu







gfortran









gcc







g++












!
http://hpc.mines.edu/bgq/compilers/
Note: The compute nodes on Mc2 have different processors than the head node so
programs compiled for one might not work on the other


Compiling
Good idea to build/test with multiple versions of compilers
Start with optimization level -O0
Normal good optimization level is -O3
[tkaiser@aun001 ~]$ ifort -O0 stringit.f90 -o stringit!
[tkaiser@aun001 ~]$ ls -lt stringit*!
-rwxrwxr-x 1 tkaiser tkaiser 668896 Mar 12 12:37 stringit!
-rw-rw-r-- 1 tkaiser tkaiser 551 Oct 3 13:42 stringit.f90!
[tkaiser@aun001 ~]$ ./stringit


Python
• Python is a scripting/programming language for
quick tasks
• Good mixture of numeric and string (text)
processing capabilities
• Easy to learn and use
• Can be run interactively
• Can be used like a calculator
• GUI and Graphics libraries
• http://www.python.org


make
• Make is a system for managing the building of
applications
• Reads a makefile
• dependancies
• instructions
• Cal s compilers and similar software to do the
build


L1= charles.o darwin.o ga_list_mod.o global.o init.o laser_new.o!
L2= mods.o more_mpi.o mpi.o numz.o unique.o wtime.o!
!
OPT= -O3 -free!
SOPT= !
LINK= -lesslbg -L/bgsys/ibm_essl/prod/opt/ibmmath/lib64!
!
PF90=mpixlf90_r!
!
darwin: $(L1) $(L2)!
!
$(PF90) $(SOPT) $(L1) $(L2) $(LINK) -o darwin!
!
.f.o:!
!
$(PF90) $(SOPT) $(OPT) -c $<!
!
wtime.o : wtime.c!
!
$(CC) -DWTIME=wtime -c wtime.c!
!
mpi.o: mpi.f !
!
numz.o:numz.f!
!
more_mpi.o: more_mpi.f numz.o mpi.o!
!!


!
charles.o: charles.f mods.o global.o more_mpi.o mpi.o numz.o !
!
darwin.o: darwin.f ga_list_mod.o global.o more_mpi.o mpi.o numz.o mods.o !
!
ga_list_mod.o: ga_list_mod.f!
!
global.o: global.f!
!
init.o: init.f global.o more_mpi.o mpi.o numz.o!
!
laser_new.o: laser_new.f ga_list_mod.o global.o more_mpi.o mpi.o numz.o !
!
mods.o: mods.f mpi.o numz.o !
!
unique.o:unique.f mpi.o numz.o!
!
clean:!
!
/bin/rm -f *o *mod $(L1b) $(L2b)!


configure & cmake
• configure and cmake are utilities for creating makefile
• Idea:
• A person that creates an application also creates a
configure or cmake file
• configure or cmake are run to create a make file
• make is run to build the application
• Ideal world:
• configure and cmake discover enough about your system
to create a working makefile
• You “may” want to specify options to tune to your system


HPC and Paral el Programming
• Concept is simple
• If a problem takes N hours on 1 processor why
not run it on N processors and finish in an hour?
• Has al of the advantages and disadvantages of
working on a committee
• Mc2 - 8192 processors in 512 nodes
• AuN - 2304 processors in 144 nodes
• Programming across multiple nodes and processors
on a node requires special languages and/or
compilers


Programming
• Thread programming for cores on a node
• Message passing for programming using multiple
nodes
• Hybrid between the two for scalable machines


Threads
Time
1
41.9
2
21.7
4
10.9
8
5.6
16
2.9

Linux links
• Tutorials:
• http://www.ee.surrey.ac.uk/Teaching/Unix/
• https://www.cac.cornell.edu/VW/Linux/default.aspx?id=xup_guest
• http://til e.garrels.be/training/bash/
• See: http://geco.mines.edu/scripts/
• General Interest
• http://en.wikipedia.org/wiki/History_of_Linux
• http://en.wikipedia.org/wiki/Linux_distribution


Local ssh pages
• Setting up ssh, including putty
• http://geco.mines.edu/ssh/
• Tunneling
• http://geco.mines.edu/ssh/tunneling.html
• http://hpc.mines.edu/bluem/transfer.html#scp
• http://hpc.mines.edu/bluem/multistage.html


More Links
• Home Page
• hpc.mines.edu
• Blog
• http://inside.mines.edu/~tkaiser/data/books/
rabook.html
• BlueM
• http://hpc.mines.edu/bluem/
• Mio
• http://inside.mines.edu/mio/


More Links
• BlueM Load
• http://mindy.mines.edu
• Module links:
• http://inside.mines.edu/mio/mio001/mod.html
• http://mindy.mines.edu/modules/aun/
• http://mindy.mines.edu/modules/mc2/


Setting up Keys
• Details are specific for gaining access to BlueM,
Aun, and Mc2
• Same concepts wil hold for other sites
• Shows how to do multistep tunnels without VPN
• From: http://hpc.mines.edu/access/


f 8 o
e 2gPa
3/26/14, 11:38 AM
du,
ac)
up
re
to
be
s.e
(M
the
ed
e the

us
can
ine
X
setting
and
ty
.m
S
nt
allow
can
ut
m
O
re
P
ue
or
for
ers
ffe
not
us
ith w
box
edure
e are
pus
ty
access bl
le is di
opl
ling
proc

cam
inux
ox
he
pe
f
ith-put
ls to
L
T
tion
M
of
tunne
b
e)
s
X
u
d.
S
ras
gura
ling-w
h
tunne
re
and
O
in
on-CS
ty
sp
is set up
unne
as
and
camp
n
confi
ut
e: N
N
P
-t
x or
ys
on
u
the
ot
P
n
ty p
ke
N
V
up
an
show
but
.
er
mp
up

are
ft
s Liu
ed
ilar
low
A
.
s/ssh-port
from
er
be
. N
setting
age
setting
P
amp
se an
du
ed
on
ill ent
is sim
us
indow
ent
e V
w
C
for
s.e
w
pus
ot u
to/
de
ine
ling
um
t us
ation
on
o n
em
us
am

lu
gui
you
how
c2.m
m
r (D
ck
m
ssh/
ands
a tunne
du/
s is doc
om an
ai
inform
cat ~/.ssh/forbluem.pub | ssh bluem.mines.edu "cat >> ~/.ssh/authorized_keys"
m
hi
and
du/
u.e
ey to b
and
T
d for on c
ers
s.e
ey p
ssh-keygen -f $HOME/.ssh/forbluem -tdsa
a a qui
pus
com
line
r k
lic k
s us
ine
cs.ne
cess fr
b
ing
du,
he
cam
u
/access/
s access vi
step.
to.c
ac
u
s.e
T
u
ra
o
out
p
you
low
.
t
co.m
e p
ine
indow
es.ed
ne
ext
l int
g u
in
fol
amp
edure
W
//ge
//how
n
erate
y th
e
c.m
he
achi
or
tp:
tp:
en
op
p
T
aun.m
m
off c
is an
tunne
proc
F
found a
ht
and
ht
etti
G
C
Guid
S
://h
er
osage:~ joeuser$
Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/joeuser/.ssh/forbluem.
Your public key has been saved in /Users/joeuser/.ssh/forbluem.pub.
The key fingerprint is:
67:60:3c:5e:42:64:23:c5:79:70:62:d1:da:74:97:45 joeuser@osage.mines.edu
The key's randomart image is:
+--[ DSA 1024]----+
| .+@=. +E|
| *o++ . o |
| *=.. . |
| o.=. |
| S o |
| o |
| |
| |
| |
+-----------------+
osage:~ joeuser$
osage:.ssh joeuser$
ttp
Us
h


f 8 o
e 3gPa
3/26/14, 11:38 AM
stxi
ot e
s noe
t df ie i
on
reate
le. C
g fifi
/con
le.
sh.s
g fifi
~/
on
r c
you
es to youin
s on
g lin
ssion
low
/access/u
ermi
e fol
e p
es.edin
th
e
d
c.m
d
et th
p
A
S
Guid
://h
er
Host bluem
HostName 138.67.132.239
Identityfile2 ~/.ssh/forbluem
Host bluem.mines.edu
HostName 138.67.132.239
Identityfile2 ~/.ssh/forbluem
Host golden
ProxyCommand ssh bluem.mines.edu nc 2>/dev/null aun.mines.edu %p
Host energy
ProxyCommand ssh bluem.mines.edu nc 2>/dev/null mc2.mines.edu %p
chmod 600 ~/.ssh/config
ttp
Us
h


f 8 o
e 4gPa
3/26/14, 11:38 AM
of
tem
can
r
e sys
you
eithe
es your
n
by
t som
s copi
ys
s logi
:
ord
hiT c2.
s tha
thi
ne
ssw
ure
er
nd M
ft
achi
and.
a pa
m
a
e your ke
ens
A
n
op m
for
uN
c2.
sabl
d
com
M
skt
s logi
ing
nd di
hi
or
aske
een by A
ey:
T
low
s
e a
uN
your de
are
bl
r k
fol
ord.
A
na
you
le:
the
an be
o e
you
to
ssw
from
n
If
y c
rc fi
t
run
h
t on
he
ent
a pa
logi
ands
c2.
m
as
M
o t
imi
er
to
and
s
r .b
ronm
r l
om
ent
c
m
tem
ou
try
and
nvi
to
he
ue
ys
uN
bl
8 h
ed
A
o
you
le s
g to you
ne
o
o run t
int
d fi
in
o you e t
et an
sh
fore
e t
ont
ck
re
g s
not
low
d
be
bl
ha
a
ba
s
ands
g to s
sin
ctly re
e fol
m
in
be
he
shoul
di
s log
t
th
om
he
d
c
low
em u
is done
d
ne
/access/
lu
d now
o t
ad
he
u
t
:
t
e fol
e you
achi
to b
.
logge
ys
al
dds
es.ed
n
houl
on
in
th
s tim
l setup
be
ke
ti
s a
e
n
se m
w
c.m
u
hi
ou s
p
hi
p
R
Logi
T
leve
logout
Y
and
the
ne
O
T
Guid
://h
er
ssh-add -t 28800 ~/.ssh/forbluem
ssh bluem
ssh golden
ssh energy
/opt/utility/appendkeys
alias keys='ssh-add -t 28800 ~/.ssh/forbluem'
alias killkeys='ssh-add -D'
ttp
Us
h


f 8 o
e 5gPa
3/26/14, 11:38 AM
ox bXS e)
O
rash
x or
sp
u
as
n
ty p
s Liu
mp ed
amp
se an
off c
ot u
en
on
o n
magi
r (D
om an
ai
cat ~/.ssh/forbluem.pub | ssh imagine.mines.edu "cat >> ~/.ssh/authorized_keys"
ey to i
ey p
ssh-keygen -f $HOME/.ssh/forbluem -tdsa
cess fr
r k
lic kbu
/access/
ac
u
p
you
e p
es.ed
g u
in
n
erate
y th
e
c.m
en
op
p
etti
G
C
Guid
S
://h
er
petra:~ joeuser$
Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/joeuser/.ssh/forbluem.
Your public key has been saved in /Users/joeuser/.ssh/forbluem.pub.
The key fingerprint is:
67:60:3c:5e:42:64:23:c5:79:70:62:d1:da:74:97:45 joeuser@osage.mines.edu
The key's randomart image is:
+--[ DSA 1024]----+
| .+@=. +E|
| *o++ . o |
| *=.. . |
| o.=. |
| S o |
| o |
| |
| |
| |
+-----------------+
petra:~ joeuser$
petra:.ssh joeuser$
ttp
Us
h


f 8 o
e 6gPa
3/26/14, 11:38 AM
sne
achi
st
m
xi
M
ot e
CS
s n
on
oe
ve
t d
ha
f ie i
you n
on
tha
le.
g fi
reate
l box
onfi
le. C
erna
g fifi
ext
/con
your
s for your c
le.
sh.s
e thi
g fifi
~/
e on m us
on
but
r c
erna
you
es to you
us
low
in
nt
be
re
s on
g lin
ffe
ctions
ssion
low
a di
re
/access/
di
u
ve
ermi
e fol
ha
he t
e p
es.edin
th
e
d
you
low
c.m
d
et th
p
A
If
fol
S
Guid
://h
er
ForwardAgent yes
Host imagine
HostName 138.67.132.196
Identityfile2 ~/.ssh/forbluem
Host imagine.mines.edu
HostName 138.67.132.196
Identityfile2 ~/.ssh/forbluem
Host step1
Hostname imagine.mines.edu
Identityfile2 ~/.ssh/forbluem
Host step2
ProxyCommand ssh step1 nc 2>/dev/null bluem.mines.edu %p
Host golden2
ProxyCommand ssh step2 nc 2>/dev/null aun.mines.edu %p
Host energy2
ProxyCommand ssh step2 nc 2>/dev/null mc2.mines.edu %p
Host bluem
ProxyCommand ssh step1 nc 2>/dev/null bluem.mines.edu %p
chmod 600 ~/.ssh/config
ttp
Us
h


f 8 o
e 7gPa
3/26/14, 11:38 AM
se
your
ssphra
er
pa
ent
to
the
ys
ed
er
:ne
ill ne
ill ent
w
c2:
achi
e your ke
w
ou
ou
Y
nd M
op m
sabl
Y
. m
skt
y.
un a
ue
nd di
ke
bl
o A
e a
ey:
to
t
bl
m
your de
r k
your
y
na
w
ue
ke
le:
o e
you
ne
w
bl
from
t
re
ne
rc fih
ent
t on
to
ands
as
d
y from
m
c2.
imi
your
r .b
ronm
r l
ede
om c
nd M
nvi
ou
copy
he
a
as ne
8 h
y.
to
uN
opy your ke
g to you
run
ke
sh
and
o run t
o A
in
o your e t
et an
o c
be
he
g s
m
e t
d
low
sin
tail -1 ~/.ssh/authorized_keys | ssh bluem.mines.edu "cat >> ~/.ssh/authorized_keys"
ing t
ssh bluem /opt/utility/appendkeys
bl a
ands
g to s
com
et up t
ctly ont
e fol
m
in
e u
shoul
low
be
re
o s
n
ing
th
om
d
c
low
fol
d di
and
low
/access/
ed t
magi
he
d now
ad
he
u
m
:
t
e fol
fol
ogge
to i
l
al
dds
es.ed
n
ord.
the
houl
on
in
th
s com
n run t
ti
s a
e
n
t you us
ssw
c.m
u
hi
he
ou s
p
hi
p
R
T
tha
Logi
Run
pa
T
Y
and be
O
T
Guid
://h
er
ssh-add -t 28800 ~/.ssh/forbluem
ssh imagine
[joeuser@imagine ~]$
[joeuser@imagine ~]$
ssh golden2
ssh energy2
alias keys='ssh-add -t 28800 ~/.ssh/forbluem'
alias killkeys='ssh-add -D'
ttp
Us
h


f 8 o
e 8gPa
3/26/14, 11:38 AM
mue ace
bl
pl
Re
on
.
ve
low
ha
ben
you
es
n
inh
tha
le as show
mac
l box

al
g
rn
erna
xte
ext
confi
e
the
s on
your
e to
ame
e on
m
m
sern
na
erna
t u
er
usM
ren
usnt CS
iffe
re
e.
d
ffe
m
@step1 nc 2>/dev/null bluem.mines.edu %p
@step2 nc 2>/dev/null aun.mines.edu %p
@step2 nc 2>/dev/null mc2.mines.edu %p
@step1 nc 2>/dev/null bluem.mines.edu %p
your
le for
a di
erna
ve
add
g fi
joeuser
joeuser
joeuser
joeuser

ha
to
on
ed
/access/
c
you
u
ith your us
if
ate
ill ne
es.ed
w
er w
in
rn
lly,
e
us
joeuser
joeuser
joeuser
joeuser
joeuser
joeuser
joeuser
c.m
lte
ina
p
A
F
you
joe
Guid
://h
er
ForwardAgent yes
Host imagine
HostName 138.67.132.196
User
Identityfile2 ~/.ssh/forbluem
Host imagine.mines.edu
HostName 138.67.132.196
User
Identityfile2 ~/.ssh/forbluem
Host step1
Hostname imagine.mines.edu
User
Identityfile2 ~/.ssh/forbluem
Host step2
User
ProxyCommand ssh
Identityfile2 ~/.ssh/forbluem
Host golden2
User
ProxyCommand ssh
Identityfile2 ~/.ssh/forbluem
Host energy2
User
ProxyCommand ssh
Identityfile2 ~/.ssh/forbluem
Host bluem
User
ProxyCommand ssh
Identityfile2 ~/.ssh/forbluem
ttp
Us
h