Information for Current Faculty and Students
This page contains information to assist researchers with getting started with High Performance Computing (HPC). To request and account, create a ticket at the University of Memphis helpdesk (Self Service -> Research and HPC Software -> HPC Account). You can also reach out to us at hpcadmins@memphis.edu if you have any questions.
Getting Started
The HPC system uses your normal University of Memphis credentials for authentication--the same username and password you use for university email, OneDrive, etc. However, before you can log in, an account needs to be created for you on the HPC cluster itself. To do that, please create a ticket to request an account (see above). HPC accounts are available to enrolled students, faculty and staff of the university. External researchers and collaborators affiliated with the University can ask their collaborator to sponsor an account.
Logging In
Once you have an HPC account, Linux and Mac users can open a terminal window and start a secureshell (ssh) connection to the login node by running the following command:
ssh [username]@bigblue.memphis.edu
Windows users can use the Windows subsystem for linux (WSL), or install a terminal emulation application such as PuTTY or MobaXTerm--SSH clients which offer a graphical user interface where you can input the necessary login information, such as your username, password, and host name. An SFTP (Secure File Transfer Protocol) client such as WinSCP or Tunnelier will also be necessary for uploading/downloading files on the cluster.
For access to the HPC from outside the University, connect first to the University's VPN and then connect to the cluster.
We have recently added the web-based interface from OpenOnDemand, which can be accessed from Campus/VPN IP addresses, at the URL https://bigblueweb.memphis.edu. This interface allows for much fo the operability of the cluster from your web browser.
Submitting Jobs
ALL jobs must be submitted to the SLURM job scheduler as either batch job (through a submission script) or interactively.
Batch Scripts (Templates for batch jobs with explanations on how to modify)
Interactive Submission (Without GUI and with GUI information)
Scheduler Control Commands (A selection of optional commands available for customization)
Python (How to use python on the cluster)
Matlab (How to use matlab on the cluster)
Note that with the web interface, BigBlueWeb, you can start an interactive GUI job from your browser.
Best Practices
Jobs
- The login nodes are meant for editing scripts and submitting jobs, and if a command takes longer than a few minutes to run, it is probably appropriate to run an interactive job with the SLURM srun or salloc commands
- Don't modify or delete your running or pending jobs' files or directories! This will cause your job to exit without errors being reported
- Make sure you define the number of shared processing cores with --cpus-per-task
- If you need to use MPI, make sure to include an MPI module (openmpi, mpich, intel/mpi, etc...) and use --ntasks to allocate cores
- Use job names, with --job-name, that help you identify what jobs you are running
- Use the least amount of time, with --time, you think your job will take. E-mail us with the jobId if you need more time
- Run a few small jobs that get progressively larger to determine appropriate resource, such as memory or time, sizes
- Over 300 other users run jobs on the cluster, and if your job doesn't start immediately, it will be in pending state until resources are available
- Try not to include the % character in your jobs. It is usually a control character in SLURM. I.e. %a, %j, and %A are all sbatch related
Storage
- Cluster storage in /project, /scratch, and /home is for cluster jobs, not backup
- Files in /scratch take up twice as much space due to GPFS replication
- If you have a lot of past job files you would like to keep, consider compressing them, tar -czf [jobFolder].tar.gz [jobFolder], or ask us about archive storage
- Storage in your home and project directories, /home/$USER/ and /project/$USER/, is backed up weekly, but your scratch, /scratch/$USER/, is never backed up. Also, note that backups of data are in the same data center as the original storage--there is no off-site backup facility.
- Files deleted using the rm command are gone unless they've been in /project or /home at least a day, in which case then they might be in the daily GPFS snapshots.
- Previous cluster home directories (/oldhome) are no longer available on the cluster as of May of 2025.
- Consider using a self contained folder for jobs that include the job batch script, input data, and output data files.
- Spaces in file and directory names usually need an escape character, Hello\ HPC, or quotes, "Hello HPC".
- High speed storage is available in /scratch/$USER/ and the quota is initially 10 TB.
- Longer term storage is available in /project/$USER/ and the quota is initially 1 TB.
- User storage is available in /home/$USER/ and the quota is initially 50 GB.
Network
- We have a very fast connection inbound and outbound from the cluster. Feel free to upload and download as much as you need.
- Our very fast connection is only intended for cluster data. We cannot host your server or database on the internet.
- Our internal network is extremely fast, especially between nodes.
Software
- Our operating system is Rocky Linux 8.8, and we cannot directly run software intended for other operating systems (Windows, Mac, Ubuntu, etc).
- Singularity is the container system we currently use. If you need a docker container, it is often trivial to use with Singularity.
- You can install software in your home directory. Generally, installs use the make
toolchain:
- ./configure --prefix=$HOME
- make
- make install
- If you need software that requires root/sudo, make a ticket under Self Service->Research and HPC Software->HPC Account.
- In general, we will not install software that gives users root/sudo access or interferes with the scheduling software.
- Learn what you need about Linux commands, but generally, you only need to know cd, ls, rm, mkdir, rmdir, scp/rsync, ssh, and vi/nano/emacs commands.
- Linux programs are generally case sensitive, i.e. a and A are different, unlike Windows.
- Be careful when using the $ character in scripts and command line because it usually indicates a variable. Use the escape sequence, \$threeFifty, or single quotes, '$threeFifty', instead.
Visualization
- Although we have GPU nodes on the cluster for general purpose computing (CUDA or OpenCL), they do not have the ability to speed up displayed applications through OpenGL.
- The mesa module uses CPUs to display OpenGL content. While the CPUs are quite powerful, they are not quite as fast as some dedicated GPUs.
- BigBlueWeb uses noVNC to allow OpenGL visualization on an XFCE desktop in your browser
Useful Links
Resources
Intel Compiler Directives for Fortran
Intel Preprocessor Options for C++
