AI meets Source Code: status quo and outlooks

In 2011, Marc Andreessen (A16Z) famously wrote a prescient claim that “software is eating the world”.  In 2017, Jensen Huang (Nvidia CEO) followed up with "software is eating the world, but AI is going to eat software".  Fast forward 5 years, and this tech-driven "banquet" seems to be bound to happen, thanks to the exponential growth of Deep Learning.  Researchers from the ML and PL communities have joined efforts to advance the AI4Code field at a breakneck pace, while several companies have recently released their own flavor of AI pair-programmers (e.g., GitHub Copilot, Amazon Code Whisperer, etc.)
In this talk, I will give an overview of the novel tasks enabled by AI4Code, with a focus on 2 of my recent works:
- Code Transformer (ICLR 2021): a SotA, language-agnostic encoder model for representation learning of source code;
- PaLM: Scaling Language Modeling with Pathways (2022): a 540-billion parameter, dense decoder-only transformer model with breakthrough capabilities on code tasks.
I will discuss the scientific and technical insights that led to the success of Large Language Models of Code (e.g., OpenAI Codex, Google PaLM, etc.), while at the same time highlighting current limitations and future research directions.

Michele (pirroh) Catasta is Head of Applied Research at X, the moonshot factory (formerly Google[x]), where he is focusing on AI applied to Source Code. Previously, he worked at Stanford University as a Research Scientist and Instructor, with affiliations to the Statistical Machine Learning group and SNAP (Prof. Jure Leskovec). Michele graduated at EPFL with a Ph.D. in Computer Science and worked also for MIT Media Lab, Google, and Yahoo Labs. His research expertise encompasses Machine Learning applied to different domains--from Information Retrieval to Recommender Systems--and data types--from source code to graph data. In the past few years, he has been covering the role of Advisor and Early Investor for several AI startups.

Not on campus? Follow this talk on ZOOM.