Training Apertus
Guide for supervised fine-tuning Apertus 1.5, starting on Kaggle and scaling text-focused training workflows on CSCS Clariden.
Use the Apertus 1.5 SFT notebook on Kaggle as the common notebook for Kaggle and CSCS Clariden.
The notebook supports three training methods:
QLoRA — validated on Kaggle and CSCS Clariden;
LoRA — validated on CSCS Clariden;
Full SFT — validated on CSCS Clariden with DeepSpeed ZeRO-3.
Select the method in the notebook:
TRAINING_METHOD = "qlora"
# "qlora" | "lora" | "full"Recommended workflow
Define or reserve a held-out evaluation set.
Evaluate the base model as a baseline.
Fine-tune on the training data.
Evaluate the adapted model on the same held-out examples.
Compare loss, task behaviour, format adherence and task-specific metrics.
Start with QLoRA on Kaggle using 2 x T4 GPUs. Once the complete workflow works, use the same notebook on CSCS Clariden when more compute is needed.
Text-focused checkpoint
For the current teaching workflow, use andreasmartin/apertus-v1.5-8b-text.
CSCS Clariden
For larger experiments, use the same notebook through the Clariden JupyterHub. QLoRA, LoRA and Full SFT have been validated in the current JupyterHub workflow. The tested setup and CSCS-specific notes are documented in Training Apertus on CSCS Clariden.
Credentials
On Kaggle, use Kaggle Secrets. On CSCS JupyterLab, the notebook can ask interactively for missing Hugging Face and Weights & Biases credentials.
Do not place tokens, passwords, or private keys directly in notebook source code, repositories, screenshots, or shared chats.