This video explains how to create and load a python environment in a SLURM cluster environment, install Python libraries with pip, then submit a SLURM job that activates that new virtual environment and use the libraries you just installed. In this example, we create a new virtual environment and install the Dask library then run it on the cluster.
Commands to create the virtual environment:
module load anaconda/anaconda3
pip install virtualenv
python3 -m venv testenv
source ~/testenv/bin/activate
pip install dask
deactivate
SLURM Script:
#!/bin/bash
#SBATCH --account=YOUR_ACCOUNT
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=00:01:00
#SBATCH --job-name=daskex
#SBATCH --error=daskex-%J.err
#SBATCH --output=daskex-%J.out
##Load Modules
module load anaconda/anaconda3
Run job ##
source ~/testenv/bin/activate
srun python daskex.py
deactivate
21 окт 2024