Large-Scale Transformer Training on MareNostrum 5

System Architecture

Leveraged the GPU partitions of the MareNostrum 5 supercomputer. Orchestrated training workloads using Slurm batch scripts, optimizing distributed data loading and multi-GPU memory utilization to avoid node I/O bottlenecks.

Project Overview

As part of the Advanced Neural Networks and Deep Learning coursework at UPC, this project focused on scaling vision model training to supercomputing infrastructure. Access to the MareNostrum 5 supercomputer was provided by the Barcelona Supercomputing Center (BSC), ranked among the top 10 most powerful public supercomputers in the world.

The MAMe (Massive Art and Media) dataset consists of over 200 GB of high-resolution museum art images across hundreds of fine-grained categories, characterized by complex artistic textures, stylistic nuance, and heavy class imbalance.

Technical Highlights & Distributed Infrastructure

1. HPC Cluster & Workload Management

• Slurm Automation: Authored and tuned multi-GPU Slurm submission scripts (launcher.sh) managing job dispatching, resource allocation, environment isolation, and automated checkpoint recovery on failure.
• Distributed Data Loading: Designed efficient PyTorch data sharding and asynchronous prefetching to eliminate disk I/O bottlenecks across shared supercomputing storage nodes.

2. Vision Transformer Architecture & Fine-Tuning

• Model Backbones: Evaluated Vision Transformer (ViT) architectures against modern convolutional baselines (ResNet, ConvNeXt), analyzing self-attention receptive fields on fine-grained stylistic patterns.
• Optimization Strategy: Trained using AdamW with cosine learning rate scheduling, mixed-precision FP16 computing, and aggressive data augmentations (RandAugment, CutMix) to prevent overfitting on fine art categories.

Benchmark Results

• Classification Performance: Achieved a top test F1-score of 0.75 across fine-grained art classes, outperforming classical CNN baselines by +6% absolute.
• Scalability: Demonstrated linear speedups across distributed multi-GPU partitions with minimal inter-node synchronization overhead.

Key Engineering Challenges & Solutions

• Challenge: Large Dataset I/O Bottlenecks
  High-resolution images on shared supercomputing filesystems can bottleneck GPU compute if not loaded efficiently.
  ✓ Solution: Implemented optimized batching, worker prefetching, and multi-node sharding with PyTorch DataLoader and Slurm environment configs.

Key Takeaways & Insights

Working directly with Slurm and HPC infrastructure provided deep hands-on understanding of the operational realities of training large models at scale, where I/O throughput and distributed batching matter as much as model architecture.