Training Apertus on CSCS Clariden
JupyterHub workflow for training Apertus with QLoRA, LoRA and Full SFT on CSCS Clariden.
Use the same Apertus SFT notebook on Kaggle and CSCS Clariden.
Open Clariden JupyterHub and start JupyterLab with the settings provided for your course or project. The tested setup used Dedicated, the CSCS project account, nvcr.io/nvidia/pytorch:25.04-py3, and no Advanced options. The notebook saw 4 x NVIDIA GH200 120 GB GPUs.
Notebook
Use the Apertus 1.5 SFT notebook.
Select the training method in the notebook:
TRAINING_METHOD = "qlora"
# "qlora" | "lora" | "full"QLoRA, LoRA and Full SFT have been validated with the current Clariden JupyterHub workflow.
The notebook detects Clariden and the visible GPUs automatically. It keeps the target global batch size at 8, so the tested setups use:
Kaggle: 2 GPUs x batch 2 x accumulation 2 = 8
Clariden: 4 GPUs x batch 2 x accumulation 1 = 8Python dependencies
The CSCS NGC PyTorch image provides the NVIDIA/PyTorch runtime. The notebook installs the Hugging Face training stack itself, including Transformers, TRL, PEFT, Accelerate, Datasets and Safetensors. It adds bitsandbytes for QLoRA, Weights & Biases when enabled, and DeepSpeed for Full SFT.
CSCS exports a global PIP_CONSTRAINT that can pin older Hugging Face packages. The notebook disables this constraint only for its package-installation subprocess and leaves the NVIDIA/PyTorch runtime unchanged.
No separate virtual environment or custom Jupyter kernel is required for this workflow.
Credentials
If credentials are missing, the notebook asks interactively for HF_TOKEN and, when enabled, WANDB_API_KEY. Do not store tokens in notebook source code.
QLoRA on GH200
On the tested CUDA 12.9 environment, bitsandbytes reports that it loads its CUDA 12.8 binary. This warning did not prevent QLoRA from running on the GH200 GPUs. The notebook uses BF16 on Clariden when supported.
LoRA on GH200
LoRA has also been validated with the current JupyterHub workflow on Clariden. It uses the same notebook and training/evaluation flow without 4-bit quantization.
Full SFT on GH200
Full SFT can be selected with:
The validated setup uses DeepSpeed ZeRO-3 across 4 x NVIDIA GH200 120 GB GPUs. Full SFT trains all approximately 8.05 billion model parameters and uses the same notebook launcher and training/evaluation flow as the other methods.
The current CSCS NGC environment exposes an NVTX implementation that is incompatible with the DeepSpeed NVTX domain API. The notebook therefore applies a small Full-SFT-only workaround before DeepSpeed is initialized:
DeepSpeed may also print repeated linker messages such as cannot find -laio. In the validated run these messages were non-fatal and training continued normally.