RCP Workshop: Advanced Deep Learning with PyTorch and NVIDIA Ecosystem

This event is organized by the Research Computing Platform (RCP), which provides campus-wide IT infrastructure for the research community.

This two half-day workshop is designed for practitioners and researchers eager to deepen their understanding of advanced deep learning concepts, tools, and best practices using PyTorch, NVIDIA NeMo, CUDA, and other state-of-the-art frameworks. The agenda blends lectures, demonstrations, and collaborative discussions to provide a thorough exploration of generative AI, parallelism, checkpointing, resiliency and model deployment across multi-GPU and multi-node environments.

! Registration is required to attend this workshop. Access is restricted to EPFL participants. !

Registration form (restricted to EPFL email addresses): https://forms.office.com/e/g0sPMJzm2h

The two half-day sessions cover different content, and we strongly encourage participants to attend both sessions for the full learning experience.

AGENDA

Day 1: Sept 29 Afternoon (Half Day) - Room SV 1717

13:00 – 13:15 | Registration and Welcome

• Participant check-in
• Workshop objectives and introductions

• Introduction to modern GPU architectures
• Understanding memory hierarchies, bandwidth, and compute units
• Overview of CUDA C/C++ and CUDA Python

• An Introduction to Deep Learning
• How a Neural Network Trains
• Data Augmentation 
• Pre-Trained Models
• Generative AI

• An introduction to our project on large-scale self-supervised pre-training with graphs for histopathology.

• Concepts of data parallelism and distributed computing
• Implementing data-parallel strategies with PyTorch’s DDP (Distributed Data Parallel)

• Scaling and parallelizing  large neural networks
• Techniques for managing large-model memory footprints and optimizing training performance
• Introduction to model parallelism frameworks
• Example a transformer model using model parallelism

• Recap and open Q&A
• Preview of the afternoon session

• Summary of Day 1 key learnings
• Outline of Day 2 agenda

• Introduction to generative AI concepts and applications
• Exploring diffusion models and their significance
• Overview of NVIDIA’s generative AI tools

• Advanced NLP techniques
• Best practices for training and fine-tuning large language models
• Best practices for optimizations: speed and memory

• Snacks and networking

• How GPU-accelerated, large-scale protein design can help us learn more about biology.

• Deep dive into checkpointing + resiliency for model recovery, and efficient training
• Overview of checkpointing + resiliency tools in PyTorch, and NVIDIA frameworks
• Example of robust checkpointing based on Nemo and PyTorch

• Multi-GPU and Multi-Node Programming: Frameworks and Libraries
• Considerations for scaling across GPU cluster
•  Profiling and performance optimization strategies

• Open discussion and feedback
• Meet the experts. Talk 1:1 or 1:N about your projects and challenge.
• Resources for continued learning – DLI, teacher kit, ambassador program (Cristel)
• Certificate distribution and farewell