Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.nyc-ai.app/llms.txt

Use this file to discover all available pages before exploring further.

The HPCC clusters run Linux. If you’ve never used a Linux shell before, this page covers the commands you’ll use every day. If you’re already comfortable with the command line, skip ahead to Software & modules or Job submission.

​
The shell prompt

After logging in, you’ll see a prompt like: 
[username@arrow ~]$
The ~ means you’re in your home directory (/global/u/<username>). Everything after $ is where you type.
CommandWhat it does
pwdPrint the current working directory.
lsList files in the current directory.
ls -lhList with details (permissions, size in human-readable form).
ls -aShow hidden files (those starting with .).
cd /path/to/dirChange to a directory.
cd ~Go home.
cd ..Go up one level.
mkdir mydirCreate a directory.
mkdir -p a/b/cCreate nested directories in one step.
rmdir mydirRemove an empty directory.

​
Managing files

CommandWhat it does
cp source destCopy a file.
cp -r sourcedir destdirCopy a directory (recursive).
mv old newMove or rename.
rm fileDelete a file.
rm -r dirDelete a directory and everything inside it.
touch fileCreate an empty file (or update its timestamp).
rm on Linux is permanent; there is no trash folder or Recycle Bin. Double-check paths before deleting, especially with wildcards (rm *.log).

​
Viewing file contents

CommandWhat it does
cat filePrint the entire file to the terminal.
less fileScroll through a file (press q to quit, /pattern to search).
head -n 20 fileShow the first 20 lines.
tail -n 20 fileShow the last 20 lines.
tail -f logfileFollow a file as it grows (useful for watching job output).
wc -l fileCount lines.

​
Searching

CommandWhat it does
grep "pattern" filePrint lines in file that match pattern.
grep -r "pattern" dirRecursive search across all files in dir.
grep -i "pattern" fileCase-insensitive search.
find . -name "*.py"Find files matching a pattern below the current directory.

​
File permissions

Every file has an owner, a group, and a permission string like -rw-r--r--. The three groups of rwx bits map to owner, group, and others. 
ls -l myscript.sh
# -rwxr-x--- 1 username groupname 4096 Apr 1 12:00 myscript.sh
CommandWhat it does
chmod 755 fileOwner can read/write/execute; group/others can read/execute.
chmod 600 fileOwner can read/write; nobody else can touch it. Use for private keys.
chmod +x script.shAdd execute permission for everyone.
chown user:group fileChange owner (usually needs sudo).
SLURM scripts must be readable by the scheduler but don’t have to be executable. sbatch handles that. If you call them directly (./script.sh), you do need chmod +x.

​
Text editors

Nano is the easiest to learn because it shows commands at the bottom: 
nano myfile.txt
# Ctrl+O → save   Ctrl+X → quit   Ctrl+K → cut line   Ctrl+U → paste
Vim is more powerful but has a learning curve: 
vim myfile.txt
# Press i to enter insert mode   Esc to exit   :w to save   :q! to quit without saving

​
Redirects and pipes

command > output.txt        # write stdout to a file (overwrites)
command >> output.txt       # append stdout to a file
command 2> errors.txt       # write stderr to a file
command > out.txt 2>&1      # write both stdout and stderr to one file
command1 | command2         # pipe stdout of command1 into command2
Examples: 
module avail 2>&1 | grep -i python      # search module list for Python
cat data.csv | sort | uniq -c > counts.txt

​
Variables and the environment

MY_VAR="hello"
echo $MY_VAR                # prints: hello

export DATA_DIR=/scratch/$USER/dataset
cd $DATA_DIR
Important environment variables already set on HPCC nodes:
VariableValue
$USERYour username.
$HOMEYour home directory.
$SLURM_JOB_IDJob ID (set by SLURM inside a job).
$SLURM_SUBMIT_DIRDirectory where sbatch was called.
$SLURM_CPUS_PER_TASKCores allocated to the task.

​
Disk usage

df -h ~                     # how full your home filesystem is
df -h /scratch/$USER        # how full scratch is
du -sh ~/mydir              # how big a specific directory is
du -sh ~/*                  # sorted breakdown of home contents

​
Process management

ps aux | grep $USER         # see your running processes
top                         # interactive process monitor (press q to quit)
htop                        # friendlier version of top (may need module load)
kill <PID>                  # terminate a process by ID
kill -9 <PID>               # force-kill
Do not run compute jobs on the login (head) node. Long-running processes found there will be killed and the account may be suspended. Use sbatch for batch jobs or srun --pty for interactive compute sessions.

​
Archiving and compression

tar -czvf archive.tar.gz mydir/    # create a gzip-compressed archive
tar -xzvf archive.tar.gz           # extract
tar -tzvf archive.tar.gz           # list contents without extracting
gzip file                          # compress in place (file → file.gz)
gunzip file.gz                     # decompress

​
Shell history and shortcuts

ShortcutWhat it does
Up / Down arrowScroll through command history.
Ctrl+RReverse-search history; type a fragment to find a past command.
Ctrl+CInterrupt (kill) the current command.
Ctrl+ZSuspend the current command.
TabAuto-complete file names, commands, and module names.
!!Repeat the last command.
!$Last argument of the previous command.

​
Basic bash scripting

A script is just a sequence of commands in a file: 
#!/bin/bash
# ── analysis.sh ──────────────────────────────────

INPUT=$1            # first argument passed to the script
OUTPUT=$2           # second argument

echo "Processing $INPUT → $OUTPUT"

grep "ERROR" "$INPUT" > "$OUTPUT"
echo "Done. $(wc -l < "$OUTPUT") errors found."
Run it: 
chmod +x analysis.sh
./analysis.sh logfile.txt errors.txt
SLURM scripts are bash scripts with #SBATCH comment-directives at the top. See Job submission for complete templates.

​
Next steps

Software & modules

Load compilers, MPI, Python, and scientific applications.

Job submission

Write and submit your first SLURM job.