Using XSEDE

Setting up XSEDE and Stampede Accounts
Using Stampede
Help and Resources
XSEDE and Stampede Account Set-up

Getting an xsede account

You should set up your xsede account if you have not yet done so: Follow these Steps:
  1. go to https://portal.xsede.org/
  2. choose "Create Account" button to request a new account: 
    Organization: Swarthmore
Department: Computer Science
Registration Key:  pick the 6 first chars your swarthmore user name
  3. When you have a choice to select a user name, pick your swarthmore user name (ex. tnewhal1 is mine)
It will take a day or so for your account to be activated.

Setting up Logging into xsede portal and stampede system

  1. go to https://portal.xsede.org/ and login with your user name, request a password. You will receive email about how to submit a password.
  2. log into the portal with your user name and password. Choose Accounts, and see what your user name on stampede is.
  3. try connecting to stampede using the single sign-on directions here.

    You will get a message with a url for how to request/rest your stampede password. Follow those directions. You will get emails from stampede with directions for setting your password.

    It will take about 30 minutes for your stampede password to work.

Logging into stampede once you have a password set up on their system

You can log into stempede from the xsede portal or directly ssh in:
  1. One way to log into stampede (from xsede portal directions here)
    1. ssh into xsede with your xsede user name and login: 
      ssh your_xsede_username@login.xsede.org
    2. gsissh into stampede: 
      gsissh -p 2222 stampede.tacc.xsede.org
  2. The other is to directly ssh into stampede using your stampede user name and password (note: your stampede user name may be different from your xsede user name, check your account information in the portal to see). 
    ssh your_stampede_username@stampede.tacc.xsede.org
Using Stampede and submitting jobs

You can use scp to copy files to/from stampede to our system.

To copy over my helloworld example with a hello.mpi batch submision script, from stampede do: 

  scp you@cs.swarthmore.edu:/home/newhall/public/XSEDE_MPI.tar .
  tar xvf XSEDE_MPI.tar
  cd XSEDE_MPI
  make

slurm

Stampede runs the SLURM resource manager to submit jobs. 
showq             # list all jobs in the job queue (there will be lots)

sbatch  job.mpi   # to submit a batch job, you will get a jobId 
scancel jobId     # to kill a queued job 
squeue -j jobId   # veiw queue status of job with specified jobId

man slurm         # the man page for slurm
/share/doc/slurm/ # example job scripts  (see job.mpi)

module avail      # lists available mpi software available on

Example

Open hello.mpi and you can see an example submission script that includes our allocation ID. Notice how the runqueue (development) and the number of nodes and processes is set. Also note the approximation for the runtime of the application (the job scheduler uses these values to order jobs in the queue).

Here is an example of submiting the mpi helloworld job using sbatch and then viewing its status in the queue using squeue -j jobId: 

sbatch hello.mpi 
-----------------------------------------------------------------
              Welcome to the Stampede Supercomputer              
-----------------------------------------------------------------

login3.stampede(54)$ sbatch hello.mpi 
-- Verifying valid submit host (login3)...OK
-- Verifying valid jobname...OK
-- Enforcing max jobs per user...OK
-- Verifying availability of your home dir (/home1/01351/newhall)...OK
-- Verifying availability of your work dir (/work/01351/newhall)...OK
-- Verifying availability of your scratch dir (/scratch/01351/newhall)...OK
-- Verifying valid ssh keys...OK
-- Verifying access to desired queue (development)...OK
-- Verifying job request is within current queue limits...OK
Submitted batch job 6614602

login3.stampede(55)$ squeue -j 6614602
JOBID   PARTITION     NAME     USER   ST       TIME  NODES NODELIST(REASON)
6614602 development  myhello  newhall CG       0:15      8 ...
When it is done running, there will be a file with its output in the directory from which you submitted it: 
$ ls
myhello.6614602.out
Use the development queue for developement and the normal queue for experiements(or gpudev and gpu).
Useful Functions and Resources