BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Memento EPFL//
BEGIN:VEVENT
SUMMARY:Talk of Dr Antonio Orvieto (ELLIS Institute Tübingen)
DTSTART:20240112T140000
DTEND:20240112T150000
DTSTAMP:20260916T211917Z
UID:ed7cbd4dc12f6dfd1315e47a35ba74dc2b00527852a5ca3c660622cd
CATEGORIES:Conferences - Seminars
DESCRIPTION:Dr Antonio Orvieto (ELLIS Institute Tübingen)\nTalk title: Ac
 curate and Efficient Processing of Long Sequences and Large Graphs without
  Attention\n\nAbstract: When applied to sequential data\, transformers hav
 e an inherent challenge: their attention mechanism leads to quadratic comp
 lexity with respect to sequence length. This issue extends to graph transf
 ormers\, where complexity scales quadratically with the number of nodes in
  the network. Today\, we'll explore theoretically grounded alternatives to
  the attention mechanism that hinge on carefully parametrized linear recur
 rent neural networks. Unlike the more commonly known LSTMs and GRUs\, line
 ar RNNs are particularly GPU-efficient. This efficiency enables us to scal
 e up the architecture\, successfully study signal propagation\, and achiev
 e competitive performance. We'll present how\, with a Linear Recurrent Uni
 t (LRU) replacing attention\, we can achieve state-of-the-art results on s
 equence modeling and graph data. This approach offers a promising directio
 n for future research\, especially in genetics\, protein structure predict
 ion\, and audio/video processing and generation.\n \nBio:  Dr Antonio Or
 vieto is a principal investigator at the newly established ELLIS Institute
  Tübingen and independent group leader at the MPI for Intelligent Systems
  in Germany. He studied Robotics and Control Engineering in Italy and Swit
 zerland. He holds a PhD from ETH Zürich and spend time at Deepmind\, Me
 ta\, MILA\, INRIA and HILTI. In his research\, Antonio strives to improve 
 the efficiency of deep learning technologies by pioneering new architectur
 es and training techniques grounded in theoretical knowledge. His work enc
 ompasses two main areas: understanding the intricacies of large-scale opti
 mization dynamics and designing innovative architectures and powerful opti
 mizers capable of handling complex data. Central to his studies is explori
 ng innovative techniques for decoding patterns in sequential data\, with i
 mplications in biology\, neuroscience\, natural language processing\, and 
 music generation.
LOCATION:ELD 020 https://plan.epfl.ch/?room==ELD%20020
STATUS:CONFIRMED
END:VEVENT
END:VCALENDAR