IC Colloquium : Capturing and Editing Models of the Real World in Motion

 By : Christian Theobalt - Max Planck Institute for Informatics

Abstract :
New methods for capturing highly detailed models of moving real world scenes with cameras, i.e., models of detailed deforming geometry, appearance or even material properties, become more and more important in many application areas. They are needed in visual content creation, for instance in visual effects, where they are needed to build highly realistic models of virtual human actors. Further on, efficient, reliable and highly accurate dynamic scene reconstruction is nowadays an important prerequisite for many other application domains, such as:
human-computer and human-robot interaction, autonomous robotics and autonomous driving, virtual and augmented reality, 3D and free-viewpoint TV, immersive telepresence, and even video editing. The development of dynamic scene reconstruction methods has been a long standing challenge in computer graphics and computer vision. Recently, the field has seen important progress. New methods were developed that capture - without markers or scene instrumentation - rather detailed models of individual moving humans or general deforming surfaces from video recordings, and capture even simple models of appearance and lighting. However, despite this recent progress, the field is still at an early stage, and current technology is still starkly constrained in many ways. Many of today's state-of-the-art methods are still niche solutions that are designed to work under very constrained conditions, for instance: only in controlled studios, with many cameras, for very specific object types, for very simple types of motion and deformation, or at processing speeds far from real-time. In this talk, I will present some of our recent works on detailed marker-less dynamic scene reconstruction and performance capture in which we advanced the capabilities of existing approaches in several ways. I will show new methods for marker-less skeletal motion and performance capture in less constrained environments, even in outdoor scenes, and with a low number of cameras. I will also discuss new state-of-the-art methods for marker-less motion capture of hands, also when in interaction with objects and background clutter, from a single RGB-D camera. Finally, I will show new approaches for high-quality capture of deforming surfaces and the human face from monocular color and depth cameras, even in real-time. If time allows, I will also briefly show how dynamic scene reconstruction methods may be used for advanced video editing effects and how joining forces of machine learning and model-based methods may open up new possibilities.

Bio :
Christian Theobalt is a Professor of Computer Science and the head of the research group "Graphics, Vision, and Video" at the Max-Planck-Institute (MPI) for Informatics, Saarbruecken, Germany. Most of his research deals with algorithmic problems at the intersection of Computer Graphics, Computer Vision, HCI and Machine Learning, such as:
static and dynamic 3D scene reconstruction, marker-less motion and performance capture, computer animation, virtual and augmented reality, appearance and reflectance modeling, machine learning for graphics and vision, new sensors for 3D acquisition, advanced video processing, as well as image- and physically-based rendering. For his work, he received several awards, including the Otto Hahn Medal of the Max-Planck Society in 2007, the EUROGRAPHICS Young Researcher Award in 2009, and the German Pattern Recognition Award 2012. In 2015 he was elected one of the top 40 innovation leaders under the age of 40 in Germany by the magazine Capital. Further, in 2013 he was awarded an ERC Starting Grant by the European Union, the most prestigious and most competitive grant for individual researchers. He is a co-founder of a spin-off company from his group - www.thecaptury.com - that sells the most advanced marker-less motion and performance capture solution commercially available today.

More information