> For the complete documentation index, see [llms.txt](https://docs.aisl.science/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.aisl.science/learning-and-resources/training-apertus/cscs-clariden.md).

# Training Apertus on CSCS Clariden

JupyterHub workflow for training Apertus with QLoRA, LoRA and Full SFT on CSCS Clariden.

Use the same Apertus SFT notebook on Kaggle and CSCS Clariden.

Open [Clariden JupyterHub](https://jupyter-clariden.cscs.ch/) and start JupyterLab with the settings provided for your course or project. The tested setup used `Dedicated`, the CSCS project account, `nvcr.io/nvidia/pytorch:25.04-py3`, and no Advanced options. The notebook saw 4 x NVIDIA GH200 120 GB GPUs.

## Notebook

Use the [Apertus 1.5 SFT notebook](https://www.kaggle.com/code/andreasmartinch/sft-apertus-v1-5-8b-hack-apertus-template).

Select the training method in the notebook:

```python
TRAINING_METHOD = "qlora"
# "qlora" | "lora" | "full"
```

QLoRA, LoRA and Full SFT have been validated with the current Clariden JupyterHub workflow.

The notebook detects Clariden and the visible GPUs automatically. It keeps the target global batch size at 8, so the tested setups use:

```
Kaggle:   2 GPUs x batch 2 x accumulation 2 = 8
Clariden: 4 GPUs x batch 2 x accumulation 1 = 8
```

## Python dependencies

The CSCS NGC PyTorch image provides the NVIDIA/PyTorch runtime. The notebook installs the Hugging Face training stack itself, including Transformers, TRL, PEFT, Accelerate, Datasets and Safetensors. It adds bitsandbytes for QLoRA, Weights & Biases when enabled, and DeepSpeed for Full SFT.

CSCS exports a global `PIP_CONSTRAINT` that can pin older Hugging Face packages. The notebook disables this constraint only for its package-installation subprocess and leaves the NVIDIA/PyTorch runtime unchanged.

No separate virtual environment or custom Jupyter kernel is required for this workflow.

## Credentials

If credentials are missing, the notebook asks interactively for `HF_TOKEN` and, when enabled, `WANDB_API_KEY`. Do not store tokens in notebook source code.

## QLoRA on GH200

On the tested CUDA 12.9 environment, bitsandbytes reports that it loads its CUDA 12.8 binary. This warning did not prevent QLoRA from running on the GH200 GPUs. The notebook uses BF16 on Clariden when supported.

## LoRA on GH200

LoRA has also been validated with the current JupyterHub workflow on Clariden. It uses the same notebook and training/evaluation flow without 4-bit quantization.

## Full SFT on GH200

Full SFT can be selected with:

```python
TRAINING_METHOD = "full"
```

The validated setup uses DeepSpeed ZeRO-3 across 4 x NVIDIA GH200 120 GB GPUs. Full SFT trains all approximately 8.05 billion model parameters and uses the same notebook launcher and training/evaluation flow as the other methods.

The current CSCS NGC environment exposes an NVTX implementation that is incompatible with the DeepSpeed NVTX domain API. The notebook therefore applies a small Full-SFT-only workaround before DeepSpeed is initialized:

```python
if TRAINING_METHOD == "full":
    import deepspeed.accelerator.cuda_accelerator as ds_cuda
    ds_cuda.nvtx = None
```

DeepSpeed may also print repeated linker messages such as `cannot find -laio`. In the validated run these messages were non-fatal and training continued normally.

## Related resources

* [Training Apertus](https://docs.aisl.science/learning-and-resources/training-apertus)
* [Apertus SFT notebook on Kaggle](https://www.kaggle.com/code/andreasmartinch/sft-apertus-v1-5-8b-hack-apertus-template)
* [CSCS Clariden](https://docs.aisl.science/external-tools-and-services/cscs-clariden)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.aisl.science/learning-and-resources/training-apertus/cscs-clariden.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.