Slurm

(jump link)

From fasseood or a login node you can submit a job via the sbatch command using the --wrap flag. An example is:

sbatch -p fasse --mem=1024 -t 240 -o /ncf/mylab/mysubjects/outfiles/reconall_mysubj1_%j --wrap="recon-all -all -subjid mysubj1

--wrap takes the executable script you want to run, followed by any flags it takes, all in double quotes.

If the script you want to submit is not in your path, such as when you make your own script to run, you need to make sure you give the full pathname to the script and that it is executable (chmod u+x scriptname). If it is a standard script like procfast or recon-all this is not necessary.

(jump link)

For this method, you will create a file, say via gedit, (gedit my_first_script.sh)that contains the flags discussed above. This is a bash script (as our cluster uses bash, as opposed to tcsh, or csh). Therefore, the first line of your file will always have the line below, which specifies it is a bash script. The next several lines specify all the flags talked about above, and then the last lines would be what you want to run. This example generates (via $RANDOM) a bunch of random numbers, puts them in a file, and then sorts them. You could also simply call recon-all, by putting recon-all -all -subjid mysubj1, after the last #SBATCH line.

#!/bin/bash
#
#SBATCH -p fasse # partition (queue)
#SBATCH --mem 100 # memory
#SBATCH -t 0-2:00 # time (D-HH:MM)
#SBATCH -o myscript_%j_output.out # STDOUT
#SBATCH --mail-type=END # notifications for job done

for i in {1..100000}; do
echo $RANDOM >> SomeRandomNumbers.txt
done

sort SomeRandomNumbers.txt

To run you script, in a terminal, type below from the location where it is, or you have to specify the full path to where it is.

sbatch my_first_script.sh

(jump link)

This is useful to know so that you can request the appropriate amount of time and memory when you run your script, or something similar, again. You want to be as accurate as possible so that resources can be fairly spread across multiple jobs. In addition, if the cluster is being heavily used, and you request a bunch of memory, it make take awhile for the requested memory to become available. Your script will stay pending until your requested memory is available.

sacct --format JobID,ReqMem,MaxRSS,Elapsed,State

This will return something that looks like:

JobID            ReqMem     MaxRSS    Elapsed     ExitCode      State 

------------ ---------- ---------- ---------- ----------- ------------ 

425616             1000M           8.82M   00:01:45    0:0    COMPLETED 

So your script took 1 min and 45 seconds, and 8.82 MB of memory. Your batch job specified that it would take 2 hours and 1000 MB (--mem=1000 -t 120 … remember memory is in MB and this way of specifying time is in minutes), way too long and too much memory. So if you were to run it again, you might use a command that requested slightly more memory (~20%) and time than it needed, which would look like:

sbatch -p fasse --mem=10.5 -t 4 --wrap="/ncf/mylab/myspace/myscript.sh"

(jump link)

To see your priority score:

sshare -U

Your priority score is the last number that comes up (FairShare). Larger is better. Generally a priority score above .5 is considered good, below .5 is bad. This score is currently based on the combined information from all the members of the group.

(jump link)

To see your pending jobs, you can type:

squeue

This should return something like:

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
73271197 fasse myscript mcmains R 0:30 1 fasse22

If it is pending, the reason will usually be Resources or Priority. If it is resources, it means there aren’t enough free nodes/cores or enough memory to run your job. If its priority, it means their are people above you. See here for more about priority.

To see how many people are ahead of you in the queue:

showq-slurm –p ncf –o

Jobs listed at the top are next in line.

(jump link)

This is only complicated because of all the quotes. This example script that takes four inputs, two strings followed by two numbers.

sbatch -p fasse -n 1 -t 04-15:01 --mem=2000 --wrap="matlab -nodisplay -nosplash -nojvm -r $'myscript(\'${subject}\',\'test\',9,0);exit'"

Generally, everything following the -r gets put in quotes. Given the –wrap command is in quotes, we enter quote unhappiness. The first thing that will come after the -r is a $', followed by your script name. If you need to use quotes in your function call, such as around strings, you need to put a \ before the single quote. The whole thing then ends with a single quote followed by a double quote. In this example, the first input passed is a bash variable that is set somewhere else to be a string, hence the quotes, followed by a string (test) and two numbers, (9,0). The exit makes sure that MATLAB closes after it finishes.

(jump link)

The max amount of memory you can request on the regular cluster is about 250 G. However, it might take a long time for your job to run because this is taking a substantial portion of the available memory on the cluster. If you find yourself needing a lot of memory (> 30 G) you might consider using the big memory nodes, that have about 2 T of available memory:

sbatch -p fasse_bigmem --mem=30000 -t 0-02:00 -o ./myoutput_%j --wrap="/ncf/mylab/myspace/myscript.sh"

The key is using either the fasse_bigmem (up to 499 GB) or fasse_ultramem (up to 2000 GB) queues. To submit a job to these queue you must request at least 30 gigs.

Also, please keep in mind that this is not the exact same hardware as the regular compute cluster. Therefore, the numbers you get back might be slightly different than the ones you would get if you ran it on the regular cluster. So make sure to not, say, run all your controls here and all your patients on the regular compute.

That being said, if you need it for one subject, but don’t need 50 gigs for all subjects, you could request 50 for the others just to end up on this node. Keep in mind repeated over requesting will hurt your priority and might end you up on the monthly ‘bad’ list, that will result in very friendly people from RC contacting you to make sure you know what you are doing.

(jump link)

A variety of problems can arise when running jobs on FASSE. Many are related to resource mis-allocation, but there are other common problems as well. Even your script seems to have finished successfully, you should look at the last line of the output file to make sure it wasn’t killed by the job handler.

tail myscript_71827678.out

This will show you the last 5 lines of the output file. You don’t want the last line to say something like:

slurmstepd: error: Exceeded step memory limit at some point.

(jump link)

If you have a script that can take advantage of multiple cores, you can request them via sbatch. There are several important flags. Keep in mind that requesting more than 1 core only helps you if your script utilizes some kind of parallelization.

(jump link)

There are two ways to accomplish this:

• via the --wrap flag, or
• via a batch script.

Regardless of how you do it, you want to practice good etiquette: don’t submit a bunch of jobs that run for less than 5 min, and pause between submitting each script.

Via the –wrap flag

You want to create a script, ie text file that will contain your script, using a text editor like sublime. This can be in any language you want, here I will demonstrate it with bash.

#!/bin/bash
#set your subjects
subjects=(150101_subj1 150102_subj2 150102_subj3)
#loop over your subjects
for subj in ${subjects[*]}; do
echo $subj
sbatch -p fasse -t 2-0:00 --mem=1024 -o ${subj}_%j.out --wrap="recon-all -subjid ${subj} -all"
sleep 1 # pause to be kind to the scheduler
done

You would then want to make this executable (chmod u+x my_script.sh). And would run it via the command line:
./my_script.sh

Via a batch script

This requires creating two scripts. One similar to what we showed above, that loops over your subjects, and one that contains the sbatch flags (your batch script), which is like what is described here (submitting via a batch script). First, lets make our script to loop over subjects, my_script.sh:

#!/bin/bash
#set your subjects
subjects=(subj1 subj2 subj3)
#loop over your subjects
for subj in ${subjects[*]}; do
echo SUBJ: $subj
sbatch -o ${subj}_%j.out mybatch_script.sh ${subj}
#you can follow your batch script call with any number of inputs it needs, in this case we are passing it one with the subject ID
sleep 1 # pause to be kind to the scheduler
done

Now we can write our batch script (mybatch_script.sh):

#!/bin/bash
#SBATCH -p fasse # partition (queue)
#SBATCH --mem 1024 # memory
#SBATCH -t 2-0:00 # time (D-HH:MM)

recon-all -subjid ${1} -all

This will take an argument (subj). As it is a bash script, it will automatically parse inputs you give it when it is called, and place them in variables (1,2,3), reflecting the order in which they followed the script name. In this case, $1 gets assigned the subject ID. After making sure your script is executable (chmod u+x my_script.sh), you can run it:
./my_script.sh