From Point Clouds to High-Fidelity Models - Advanced Methods for Image-Based 3D Reconstruction

Abstract

Capturing automatically a virtual 3D model of an object or a scene from a collection of images is a useful capability with a wide range of applications, including virtual/augmented reality, heritage preservation, consumer digital entertainment, autonomous robotics, navigation, industrial vision or metrology, and many more. Since the early days of photogrammetry and computer vision, it has been a topic of intensive research but has eluded a general solution for it. 3D modeling requires more than reconstructing a cloud of 3D points from images; it requires a high-fidelity representation whose form is often dependent on individual objects. This thesis guides you in the journey of image-based 3D reconstruction through several advanced methods that aims to push its boundaries, from precise and complete geometry to detailed appearance, using both theory with elegant mathematics and more recent breakthroughs in deep learning. To evaluate these methods, thorough experiments are conducted at scene level (and large-scale) where efficiency is of key importance, and at object level where accuracy, completeness and photorealism can be better appreciated. To show the individual potential of each of these methods, as well as the possible wide coverage in terms of applications, different scenarios are considered and serve as a proof-of-concept. Thereby, the journey starts with large-scale city modeling using aerial photography from the cities of Zürich (Switzerland), Enschede (Netherlands) and Dortmund (Germany), followed by single object completion using the synthetic dataset ShapeNet, that includes objects like cars, benches or planes that can be found in every city, to finish with the embellishment of these digital models via high-resolution texture mapping using a multi-view 3D dataset of real and synthetic objects, like for example statues and fountains that also dress the landscape of cities. Combining them together into an incremental pipeline dedicated to a specific application would require further tailoring but is quite possible.