Research IT

Python Virtual Environments

Contents

Overview

Python virtual environments are a useful tool that allow a user to configure a python environment for a specific purpose without introducing conflicts between packages or package versions for other applications. If a user had two applications that required different versions of a python package, they would be able to install each version in a virtual environment and avoid conflicts with the applications. More can be read about Python virtual environments here: https://docs.python.org/3.9/tutorial/venv.html

Return to Table of Contents

Creating a Python Virtual Environment

Login to a Compute Node

First, log in to Nova and open a shell on a compute node.

Load a python-venv module

We will need to load the Python virtual environment tools python-venv installing and using virtual-environments.   From the shell on the compute node, load the python-venv module:

$ module load python-venv     

This will select the default version.  To see what other packages this particular module loads, do:
$ module list

Create the Virtual Environment under Your Work or Ptmp Directory

Important:  Do not create the virtual environment in your home directory.  You don't have enough storage capacity there.  Use a path under your group's Work Directory (/work/<groupname>) or the Ptmp folder (/ptmp/<groupname>).

To create a python3 environment named Test1 in my work directory, I would do:

$ mkdir /work/ccresearch/jedicker/venv/
$ python -m venv /work/ccresearch/jedicker/venv/JupyterTest1

 The above commands will create a virtual environment called JupyterTest1 under the path /work/ccresearch/jedicke/venv.

Return to Table of Contents

Activating a Python Virtual Environment

Once the environment is created, it needs to be activated to install packages and be used. Activating the environment can be done by running the source command with the path to the activate script that was created with the environment. For the Test1 example created above, the command would be:

$ source /work/ccresearch/jedicker/venv/JupyterTest1/bin/activate

Running this will change the shell prompt to indicate that the environment is active by including the python environment name in round brackets. For the JupyterTest1 example it would display:

(JupyterTest1) $

Once the environment is active, packages can be installed in it and it can be used.

Return to Table of Contents

Installing Packages in a Python Virtual Environment

In an active environment, packages can be installed with pip. To install NumPy with pip in an active environment, the following command would be run:

$ pip --no-cache-dir install numpy

To uninstall NumPy from an active environment the following command would be run:

$ pip --no-cache-dir uninstall numpy

Note: The --no-cache-dir option is specified to prevent the default behavior of a cache files being created in the user's home directory, which can fill the directory. Documentation for pip's caching behavior can be found here. 

Return to Table of Contents

Deactivating a Python Virtual Environment

To stop using a virtual environment, a user just has to run deactivate from their shell.

$ deactivate

Return to Table of Contents

Using a Python Virtual Environment on Nova

Python virtual environments can be created and used both interactively and in slurm jobs. To use the environment in a slurm job, a user just has to include the source command. From the Test1 example above, the command would be:

### Script Above ###

source /work/ccresearch/jedicker/venv/JupyterTest1/bin/activate

### Script Below ###

Return to Table of Contents

Using Python Virtual Environments with OOD Jupyter Notebooks

Virtual environments can be used with Jupyter notebooks on the Open OnDemand platform with ISU clusters. This requires that the ipykernel package be installed and the config placed in the user's home directory. To do this, first:

Start an interactive session on a compute node to prevent excessive load on the login node.  For example, the following will request a session on a compute node with 16 processors and 64GB of memory for two hours:

$ salloc -N 1 -n 16 -t 2:00:00  --mem=64G

As above, choose a version of python, create a Python virtual environment, and activate it.  The example below assumes the venv is created in /work/ccresearch/jedicker/venv/Test1

$ module load py-pip
$ source /work/ccresearch/jedicker/venv/JupyterTest1/bin/activate

Install the ipykernel package. 

 $ python -m pip install ipykernel --no-cache-dir

Create the ipykernel config in the user's home directory. This is not an issue since the config is small. 

$ python -m ipykernel install --user --name "JupyterTest1"

Once the config is created, the environment can be used by:

  • Logging in to Nova OnDemand. See guide here
  • Creating a Jupyter Notebook Session
    • Select Interactive Apps
    • Select Jupyter Notebook from the list of apps
    • Select desired compute and partition settings.
    • Select Launch
    • Once it starts, select Connect to Jupyter
  • In the Jupyter Notebook Launcher, select JupyterTest1 from the Notebook or Console section. It can also be selected as the kernel when choosing the kernel for a new notebook or console. 

Return to Table of Contents