Creating and Using Python Environments on Clipper

Summary

How to create and access different types of Python environments on Clipper.

Body

Overview

Python environments are isolated installations that allow users to manage project-specific dependencies without affecting other users or projects on the system. They provide several key benefits:

Isolation: Each environment maintains its own separate set of Python packages and dependencies
Version Control: Different projects can use different versions of Python or packages without conflicts
Reproducibility: Environments can be recreated consistently across different systems
Security: Changes in one environment don't affect other users or the system Python installation

On Clipper, users have two main options for managing Python environments:

Conda Environments: A comprehensive environment management system that can handle Python and non-Python dependencies
Virtual Environments: A lightweight, built-in Python tool focused specifically on Python package management; Python virtual environments are the recommended way to manage Python projects on the Clipper HPC cluster.

Choose Your Environment Type

Conda Environments Show

Creating and Using Conda Environments on Clipper

Conda environments provide a flexible way to manage Python projects and their dependencies on the Clipper HPC cluster.

Conda environments are isolated Python installations that allow for easy management of project-specific dependencies. They enable users to create a personalized environments without affecting the global Python installation available to all Clipper users. This way, users can manage multiple environments without affecting other projects or users.

This section describes how to create and use Conda environments on Clipper and how to access them through Jupyter.

Creating Conda Environments

Conda environments can be stored in your /mnt/home folder or in project space allocated on /mnt/projects.

To create a Conda environment, first load the Miniconda3 module:

[hpcuser1@clipper ~]$ module load miniconda3

Then, create the Conda environment. This example creates an environment named "mycondaenv" in the user's home directory:

[hpcuser1@clipper ~]$ conda create --prefix /mnt/home/hpcuser1/mycondaenv

Using Conda Environments

Activation and Deactivation

Conda environments must be activated to make the isolated Python interpreter and packages available in your shell.

To activate the environment, run:

[hpcuser1@clipper ~]$ conda activate /mnt/home/hpcuser1/mycondaenv

Your shell prompt will change to indicate you are in the Conda environment:

(/mnt/home/hpcuser1/mycondaenv) [hpcuser1@clipper ~]$

To deactivate the environment and return to the default state, type:

conda deactivate

Installing Packages

Because the Conda environment has been activated, the miniconda3 module is no longer loaded. Module load is used to load a specific software package or tool into your current shell environment, and because that environment has been changed the module must be loaded again. Before installing packages inside the environment, run:

module load miniconda3

Conda can now be used to install packages inside the environment. After activation, run:

conda install <package>

For example, to install the ipykernel package:

conda install -c anaconda ipykernel

Conda will handle the installation of ipykernel and all necessary dependencies.

Packages can also be removed from the Conda environment:

conda remove <package>

Using Conda Environments with Jupyter

To use your Conda environment with Jupyter, ipykernel must be installed and registered to provide Jupyter with the IPython kernel. To install, run:

conda install -c anaconda ipykernel
python -m ipykernel install --user --name "mycondaenv"

This will make your Conda environment available as a kernel option in Jupyter notebooks.

To access your Conda environment in Jupyter notebooks, see Accessing Jupyter Lab on Clipper

Note on Environment Persistence

Each time you start a new session or switch environments, you may need to reload the miniconda3 module to ensure conda is available:

module load miniconda3
conda activate /mnt/home/hpcuser1/mycondaenv

Conda Version Note

As of May 2024, the Miniconda3 version available on Clipper is 24.1.2. This version includes Python 3.9.18 by default, but other Python versions can be installed within Conda environments as needed.

If you require a different Conda or Python version, please contact Academic Research Computing support with your request and justification.

Python Virtual Environments Show

Creating and Using Python Virtual Environments on Clipper

Python virtual environments are the recommended way to manage Python projects on the Clipper HPC cluster.

Virtual environments are isolated Python installations that allow for easy management of project-specific dependencies. Virtual environments allow individuals to create a tailored environment without affecting the global Python installation available to all Clipper users.

This section describes how to create a Python virtual environment on Clipper.

Creating Virtual Environments

Private virtual environments can be stored either in your /mnt/home folder or project space allocated on /mnt/projects.

Will this virtual environment be used for a class or a large number of users? Contact Academic Research Computing support to establish a globally-available Python virtual environment.

To create a virtual environment, use the built-in venv Python module. This example installs a virtual environment into a hypothetical /mnt/home/hpcuser1 directory.

Navigate to the directory in which you wish to create the virtual environment:

[hpcuser1@clipper ~]$ cd /mnt/home/hpcuser1

Create the virtual environment with the venv module:

[hpcuser1@clipper ~]$ python3 -m venv myproject

This will create a directory named myproject containing the virtual environment's files inside of /home/hpcuser1.

Using Virtual Environments

Activation and Deactivation

Virtual environments must be activated. Activating a virtual environment makes the isolated Python interpreter and packages available in your shell.

To activate the earlier-created environment, run:

[hpcuser1@clipper ~]$ source myproject/bin/activate

Once activated, your shell prompt will change to indicate you are in the virtual environment:

(myproject) [hpcuser1@clipper ~]$

To deactivate the environment and return to the system default Python environment, type:

deactivate

Installing Packages

Use the pip package manager to install packages inside the virtual environment. Once you have activated the environment, run:

pip install <package>

For example, to install the BeautifulSoup library, run:

pip install beautifulsoup4

pip will handle the installation of Beautiful Soup and all necessary dependencies automatically.

Packages can also be removed from the virtual environment:

pip uninstall <package>

Python Version Note

Clipper's default Python version is that of its underlying operating system, Red Hat Enterprise Linux 9. As of May 2024, the Python version available on Clipper is 3.9.18.

Additionally, Python 3.12.1 is available through the Lmod modules system.

Should you need a different version of Python, please contact Academic Research Computing support with your request and justification.

Best Practices

Project Isolation: Create separate environments for each project to prevent dependency conflicts
Version Control: If using Git, add environment directories to .gitignore
Documentation: Maintain a list of required packages (requirements.txt for virtual environments or environment.yml for Conda)
Regular Updates: Periodically update packages while testing for compatibility

System Information

As of January 2025:

Default Python version: 3.9.19 (RHEL 9)
Python 3.10.14, 3.11.9 and 3.12.5 available via Lmod modules
Miniconda3 version: 24.9.2
Miniforge3 version 24.3.0

For requests regarding different Python or Conda versions, contact Academic Research Computing support with your justification.

Details

Article ID: 21850

Created

Thu 10/24/24 1:44 PM

Modified

Thu 12/19/24 11:30 AM