Video-based action segmentation by learning world models from language
Event details
| Date | 03.06.2024 |
| Hour | 09:00 › 11:00 |
| Speaker | Sepideh Mamooler |
| Location | |
| Category | Conferences - Seminars |
EDIC candidacy exam
Exam president: Prof. Robert West
Thesis advisor: Prof. Antoine Bosselut
Thesis co-advisor: Prof. Alexander Mathis
Co-examiner: Prof. Mathieu Salzmann
Abstract
Many questions in biology, from development to neuroscience and medicine require the identification of fine-grained behaviors. We will develop novel computer vision and natural language processing technology to improve behavioral analysis in biology and medicine. Specifically, we will build deep learning models that can efficiently learn joint representations from video and heterogeneous data sources (e.g., textual descriptions, knowledge graphs). To do so, we will mine the written literature as well as video-sharing platforms to extract a knowledge graph of behavior and then learn tri-modal models based on vision, language and this knowledge graph. We believe that these models will be able to more robustly and efficiently generalize to various applications in biology.
Background papers
coming soon
Exam president: Prof. Robert West
Thesis advisor: Prof. Antoine Bosselut
Thesis co-advisor: Prof. Alexander Mathis
Co-examiner: Prof. Mathieu Salzmann
Abstract
Many questions in biology, from development to neuroscience and medicine require the identification of fine-grained behaviors. We will develop novel computer vision and natural language processing technology to improve behavioral analysis in biology and medicine. Specifically, we will build deep learning models that can efficiently learn joint representations from video and heterogeneous data sources (e.g., textual descriptions, knowledge graphs). To do so, we will mine the written literature as well as video-sharing platforms to extract a knowledge graph of behavior and then learn tri-modal models based on vision, language and this knowledge graph. We believe that these models will be able to more robustly and efficiently generalize to various applications in biology.
Background papers
coming soon
Practical information
- General public
- Free