Reducing Home Directory Usage on Nova

Table of Contents

Introduction

Each user on the Nova cluster is allocated a home directory with 10 GB of storage space.  This directory is intended to store configuration and login files, and therefore is much slower and smaller than other storage locations available to users.  Because of this, it is easy for files to accidentally consume a user's entire quota.  When this happens, it can cause some odd, seemingly unrelated errors, such as Nova OnDemand connections failing.  This occurs because many programs store small temporary files in the home directory, and they cannot run correctly if they are unable to write these files.  Therefore, it is imperative that users maintain some free space in their home directory.

Identifying and Removing Problematic Files

First, it is important to figure out which files and directories are consuming the most space so they can be dealt with.  To do this:

  1. Log into a terminal session on novadtn.its.iastate.edu either via SSH or Nova OnDemand.

  2. Run the following command in the terminal: du ~ -a -h -d 1 | sort -h -r

    How This Command Works:
	1. "du" is the Disk Usage command, which returns how much space files and directories are using on the disk.
	2. "~" tells teh command to check the home directory.  "~" is an alias for \home\<username>, which is the same as the $HOME environment variable.  The three may be used interchangeably.
	3. "-a" is an option for the du command that tells it to check the size on all files (which allows you to see the space used by hidden files).  This is important because hidden files and directories (denoted by a "." at the start of their name) are often responsible for consuming significant amounts of space.
	4. "-h" is an option for the du command that formats the output sizes in a humand readable format instead of raw block usage.
	5. "-d 1" is an option for the du command that instructs the command to check subdirectories up to 1 directory deep.  This is useful for understanding which subdirectories are consuming the most space.
	6. "|" redirects the output of a command (du in this case) to another command (sort in this case) for further processing.
	7. "sort" is a command that sorts the output of another command into a useful order.
	8. "-h" is an option for the sort command that instructs it to interpret the data size numbers as human readable data size values.
	9. "-r" is an option for the sort command that reverses the sorting order so the sizes are listed largest to smallest instead of smallest to largest.

  3. The command will return a result that looks something like the following.  Note, the folders and sizes you see will be different based on which software packages you use and how you organize your files.

    526M    /home/<username>
166M    /home/<username>/Data
113M    /home/<username>/.cache
56M     /home/<username>/.apptainer
41M     /home/<username>/ondemand
1.6M    /home/<username>/.local
837K    /home/<username>/.config
286K    /home/<username>/.ondemand
115K    /home/<username>/.ssh
17K     /home/<username>/.bashrc
17K     /home/<username>/.bash_profile
17K     /home/<username>/.bash_history
512     /home/<username>/.vscode-server
512     /home/<username>/.first-time-setup-complete
512     /home/<username>/.bash_logout
    1. The top line (/home/<username>) is the total usage of the home directory.  In this case the user is only using 526 MB of the allotted 10 GB, and wouldn't need to clean up the directory.  However, if this usage ever climbs above 7 GB, the user would want to free up space immediately to ensure the disk usage doesn't hit the 10 GB limit.  Hitting this limit can prevent some software from running correctly.
    2. The remaining lines represent files and directories and the total usage of their contents.  In this case /home/<username>/Data is the largest directory, and should be moved to another location (as data files should not be stored in the home directory) or deleted if the files are no longer needed.
    3. The files and directories that start with "." are hidden normally.  To list those files, you need to use the "-a" option on the ls command.
  4. To delete a file, at the terminal run: rm <filename>
  5. To delete a directory and all of its contents, at the terminal run: rm <directoryname> -r
  6. To delete an empty directory, at the terminal run: rmdir <directoryname>
  7. To move a file or directory, at the terminal run: mv <source> <destination>

Creating Symbolic Links for Large Folders

Some software packages automatically make files in hidden directories within the home directory. These files can quickly fill a user’s home directory if the user is not aware of this.  The best way to handle these files is to create a symbolic link to another storage location so it appears to the software as if the file is being stored in the home directory, but in reality, the files are stored in another location that has more space available and better performance.  In this example, the directories will be redirected to the user’s work directory; however, another storage directory (such as LSS) could be used. It is not recommended to use $TMPDIR or /ptmp for these files, as both those locations have their data automatically purged on a regular basis.  To do this:

  1. Make a directory where you would like the program to store its data instead of on the home directory.  To do this, run:

    mkdir -p /work/<your_group_working_directory>/<username>/<new_parent_directory>
    1. The “-p” option instructs the command to create any necessary parent directories for the directory you requested.
    2. If you already have an existing directory you would like to use for the data, you can skip this command.

  2. Move any existing data from the home directory to the newly created directory.  To do this, run:

    mv ~/<old_program_directory> /work/<your_group_working_directory>/<username>/<new_parent_directory>

  3. Create a symbolic link between the old location and the new one.  Note, if there are any existing files in the old directory that you did not move in step 2, this command will fail until you delete or move those files.

    ln -s /work/<your_group_working_directory>/<username>/<new_program_directory> ~/<old_program_directory>
    1. The “-s” option instructs the command to create a symbolic link instead of a hard link.  You must use a symbolic link in this case because the directories are on different physical drives, which hard links do not support.

Software Specific Recommendations

From past experience, we are aware that the following programs create and store a large amount of data on the home directory.  Recommended actions for mitigating these issues are listed below.  However, this is not an exhaustive list, and there may be other software packages that consume significant space in the home directory in addition to these.

Apptainer

Apptainer has two environment variables that control where temporary files are stored.  Because these files do not need to persist across sessions, it is recommended that $TMPDIR be used to store these temporary files. This is set by:

export APPTAINER_CACHEDIR=$TMPDIR
export APPTAINER_TMPDIR=$TMPDIR

These environment variables should be set automatically when the Apptainer module is loaded.  To verify they are set correctly you can use the following commands, which should all return the same path to the temporary storage for your job:

echo $TMPDIR
echo $APPTAINER_CACHEDIR
echo $APPTAINER_TMPDIR

Additionally, the image files that Apptainer uses (usually *.sif files) can be quite large.  When creating or pulling an image, make sure your current directory is your work directory to ensure the images are stored there instead of in your home directory.

Conda Virtual Environments

The environment and package cache directories for conda virtual environments can rapidly fill the home directory if they are not moved.  To do this:

  1. Create a new folder for your conda environments.  For example: mkdir -p /work/<your_group_working_directory>/<username>/.conda/envs
  2. Create a new folder for your conda packages.  For example: mkdir -p /work/<your_group_working_directory>/<username>/.conda/pkgs

  3. Edit the .condarc file to add the environment directory and package cache directory locations, as shown below using the example directories.  Be sure to use spaces and not tabs in the file.  By default this file is located at $HOME/.condarc.  If the file doesn't exist, create it.

    envs_dirs:
  - /work/<your_group_working_directory>/<username>/.conda/envs
  
pkgs_dirs:
  - /work/<your_group_working_directory>/<username>/.conda/pkgs

Python Virtual Environments

Python virtual environments contain copies of all the external packages you install, and can get quite large.  To prevent this filling up your home directory, simply specify a path in your work directory when creating the virtual environment.  For example:

python -m venv /work/<your_group_working_directory>/<username>/python_venv

Additionally, when installing packages in the virtual environment with pip, it is strongly recommended that you use the --no-cache-dir option to avoid the home directory being filled with temporary cache files.

RStudio Server via Nova OnDemand

When accessed through Nova OnDemand, RStudio Server stores packages you install in the /home/<username>/.ondemand/<username>/rstudio directory by default.  To avoid this filling your home directory, redirect this folder to your work directory.

Visual Studio Code Server via Nova OnDemand

When accessed through Nova OnDemand, VS Code Server stores extensions you install in the /home/<username>/.local/share/code-server directory by default.  To avoid this filling your home directory, redirect this folder to your work directory

Visual Studio Code via Remote Extensions

When accessing the cluster using Remote Extensions in Visual Studio code, the software stores extensions you install in the /home/<username>/.vscode-server directory by default.  To avoid this filling your home directory, redirect this folder to your work directory