Analyzing the Behavior of Self-Supervised Learning Models
Event details
| Date | 08.07.2024 |
| Hour | 13:00 › 15:00 |
| Speaker | Haoqi Wang |
| Location | |
| Category | Conferences - Seminars |
EDIC candidacy exam
Exam president: Prof. Sabine Süsstrunk
Thesis advisor: Prof. Mathieu Salzmann
Co-examiner: Prof. Maria Brbic
Abstract
Self-supervised learning has become the foundation of both computer vision and NLP, exemplified by foundation models such as DINOv2 and Llama 2. Their learned representations have empirically proven to be of high quality and generalizable. The behaviors of SSL models, both abnormal and regular, have attracted many researchers to study them. The three selected papers cover this topic. The first paper provides the background of self-supervised learning. The second paper provides a treatment for the defective tokens of DINOv2. The third paper studies the emergence of the semantic clustering structure in self-supervised training. However, most of the existing studies focus on the empirical analysis of behaviors, yet few understandings have been gained on why their representations are superior. My research aims to analyze the behavior of self-supervised models, and my approach is to first study the abnormal behavior of self-supervised models. My preliminary research has provided a mathematical prediction of the high-norm defective tokens in DINOv2 and proposed an effective way to repair the defective tokens. I am currently working on extending the analysis of defective tokens to language models. After a thorough understanding of the defective tokens, I will move on to analyze the regular behavior of self-supervised models.
Background papers
1. iBOT: Image BERT Pre-Training with Online Tokenizer https://openreview.net/pdf?id=ydopy-e6Dg (page 1-9).
2. Vision Transformers Need Registers https://openreview.net/pdf?id=2dnO3LLiJ1 (page 1-9).
3. Reverse Engineering Self-Supervised Learning https://openreview.net/pdf?id=NsVEjx6YPd (page1-10).
Exam president: Prof. Sabine Süsstrunk
Thesis advisor: Prof. Mathieu Salzmann
Co-examiner: Prof. Maria Brbic
Abstract
Self-supervised learning has become the foundation of both computer vision and NLP, exemplified by foundation models such as DINOv2 and Llama 2. Their learned representations have empirically proven to be of high quality and generalizable. The behaviors of SSL models, both abnormal and regular, have attracted many researchers to study them. The three selected papers cover this topic. The first paper provides the background of self-supervised learning. The second paper provides a treatment for the defective tokens of DINOv2. The third paper studies the emergence of the semantic clustering structure in self-supervised training. However, most of the existing studies focus on the empirical analysis of behaviors, yet few understandings have been gained on why their representations are superior. My research aims to analyze the behavior of self-supervised models, and my approach is to first study the abnormal behavior of self-supervised models. My preliminary research has provided a mathematical prediction of the high-norm defective tokens in DINOv2 and proposed an effective way to repair the defective tokens. I am currently working on extending the analysis of defective tokens to language models. After a thorough understanding of the defective tokens, I will move on to analyze the regular behavior of self-supervised models.
Background papers
1. iBOT: Image BERT Pre-Training with Online Tokenizer https://openreview.net/pdf?id=ydopy-e6Dg (page 1-9).
2. Vision Transformers Need Registers https://openreview.net/pdf?id=2dnO3LLiJ1 (page 1-9).
3. Reverse Engineering Self-Supervised Learning https://openreview.net/pdf?id=NsVEjx6YPd (page1-10).
Practical information
- General public
- Free