Using Conda/Mamba on Berzelius

• Introduction
• Conda/Mamba on Berzelius
  • Conda
  • Mamba
• Conda/Mamba Best Practices
• Conda/Mamba Environment Relocation
• Cheat Sheet

Introduction

Conda is an open-source package management and environment management system primarily used for Python. It allows you to create isolated environments with specific sets of packages and dependencies to avoid conflicts between different projects.

The concepts of Conda, Miniconda, Anaconda, and Mamba are all related to package management and environment management in Python. Let's explore each concept:

• Conda is the core package and environment management tool.
• Miniconda is a minimal Conda installer, allowing you to create custom environments.
• Anaconda is a comprehensive distribution of Conda, focused on data science and scientific computing.
• Mamba is an alternative package manager that can be used as a replacement for the default Conda package manager, providing improved performance.
• Mambaforge is a distribution of Mamba that provides a pre-packaged and ready-to-use environment for Conda users.

The default channel for Conda is anaconda, and Mamba installs packages by default from the channel conda-forge.

We recommend to use Mamba for its improved performance.

Conda/Mamba on Berzelius

Conda

The Anaconda installation is available on Berzelius via the module system, for example, Anaconda/2023.09-0-hpc1-bdist.

A basic example of creating a Conda environment called myenv with Python 3.8, including the pandas and seaborn packages:

module load Anaconda/2023.09-0-hpc1-bdist
conda create -n myenv python=3.8 pandas seaborn
conda activate myenv

To find valid conda package names look at Anaconda repo package search.

To remove a Conda environment

conda env remove -n myenv

Please refer to the conda cheatsheet for the most important commands for using Conda.

Mamba

The "solving environment" stage is where Anaconda tries to determine the best way to install the packages you've requested, considering the other packages you have installed and their dependencies. This process can sometimes take a long time due to complex dependencies, outdated package versions, or network issues. It is highly recommended to use Mamba as an alternative to Conda.

Mamba is a reimplementation of the Conda package manager in C++ that offers the following benefits:

• Much faster dependency solving
• Parallel downloading of repository data and package files

The following table shows a comparison of installation times for the command:

install pytorch==2.3.0 torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

Mamba demonstrates a significantly faster installation time, completing in 1m17.021s compared to Conda's 6m26.264s.

Berzelius makes Mamba available via the Mambaforge miniforge distribution. For example, to set up a customized environment pytorch-2.3.0-python-3.10, install PyTorch and run a Python script python_script.py in it:

module load Mambaforge/23.3.1-1-hpc1-bdist
mamba create -n pytorch-2.3.0-python-3.10 python==3.10
mamba activate pytorch-2.3.0-python-3.10
mamba install pytorch==2.3.0 torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
python python_script.py

To remove a Mamba environment

mamba env remove -n pytorch-2.3.0-python-3.10

Conda/Mamba Best Practices

Issues can arise when conda and pip are used together to create an environment.

• Running conda after pip has the potential to overwrite and break packages installed via pip.
• Running pip after conda may upgrade or remove a conda-installed package.

A general best practice guideline is

• Create a conda environment for isolation.
• Try to only use conda packages.
• Install as many requirements as possible with conda, then use pip.

For more details, please read Using Pip in a Conda Environment

Conda/Mamba Environment Relocation

The default location for Conda/Mamba environments is ~/.conda in your home directory. This location can be problematic as these environments can become very large. It is recommended recommended to redirect this directory using a symbolic link to your project directory.

mv ~/.conda /proj/<your project>/users/<your username>
ln -s /proj/<your project>/users/<your username>/.conda ~/.conda

Cheat Sheet