Slurm User Guide

Using Conda in Slurm Jobs

Conda is a popular package management system and environment management system. When using Slurm on a cluster, you may need to activate and use Conda environments in your job scripts. Here's how to do it effectively:

1. Loading Conda

First, you need to ensure Conda is available in your Slurm job. This usually involves loading a module or sourcing a script to initialize Conda. 

module load anaconda3

2. Activating a Conda Environment

Once Conda is loaded, you can activate your environment. Here's how you might do this in a Slurm script: 


#!/bin/bash 
#SBATCH --job-name=conda_job 
#SBATCH --output=output_%j.log
#SBATCH --error=error_%j.log
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G

# Load Conda
module load anaconda3

# Activate your environment
conda activate myenv

# Run your Python script
python my_script.py

3. Creating Conda Environments on the Fly

Sometimes, you might want to create a Conda environment as part of your job. Here's an example: 


#!/bin/bash
#SBATCH --job-name=conda_create_job
#SBATCH --output=output_%j.log
#SBATCH --error=error_%j.log
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G

# Load Conda
module load anaconda3

# Create a new environment
conda create -n job_env python=3.8 numpy pandas -y

# Activate the new environment
conda activate job_env

# Run your Python script
python my_script.py

# Optionally, remove the environment at the end of the job
conda deactivate
conda env remove -n job_env -y

4. Best Practices for Using Conda with Slurm

  • Specify exact versions: When creating environments, specify exact versions of packages to ensure reproducibility.
  • Use environment files: For complex environments, use a `environment.yml` file to specify dependencies.
  • Clean up after your job: If you create temporary environments, make sure to remove them to free up space.

    • 6. Troubleshooting Conda in Slurm Jobs

    If you encounter issues:
  • Check your Slurm output and error logs for Conda-related errors.
  • Ensure Conda is properly initialized in your job script.
  • Verify that the specified Conda environment exists and contains the necessary packages.
  • Check for any conflicts between Conda and other modules loaded in your job.