On the Reconstruction, Understanding and Editing of 3D Scenes for Augmented Reality
Open access
Author
Date
2023Type
- Doctoral Thesis
ETH Bibliography
yes
Altmetrics
Abstract
We stand on the cusp of a new technological era, where technology seamlessly integrates into our daily lives. Entering this brave new world requires the convergence of always-on artificial intelligence and augmented reality. However, we still must overcome numerous challenges to realize this vision. This thesis addresses three pivotal challenges that still remain: 3D reconstruction, 3D scene understanding, and 3D scene editing. Augmented reality applications demand a reconstruction of the world that is continuously updated with new information. Therefore, we start with tackling the challenge of incrementally fusing noisy and outlier contaminated data in an online system. We approach the challenge from a data-driven perspective, utilizing a learned scene representation, to enhance existing methods’ efficiency with the power of machine learning. However, spatial awareness alone is not sufficient. Hence, we move on to 3D scene understanding, where we confront the high costs of annotating datasets for 3D semantic segmentation models. We introduce an automated semantic annotation pipeline that matches human annotation quality, unifying the predictions of state-of-the-art models into a shared label space that are further improved through 3D lifting. Additionally, we extend the online reconstruction pipeline to semantic mapping, overcoming limited receptive fields with a spatio-temporal attention mechanism that efficiently combines information from 2D and 3D with past information. In the final part, we explore the use of neural radiance fields for 3D scene editing. Thus, we propose a method that leverages priors encoded in powerful 2D inpainting method for removing objects from scenes. This requires the design of a confidence-based view-selection mechanism during the optimization stage that enforces multi-view consistency in the final reconstruction. Show more
Permanent link
https://doi.org/10.3929/ethz-b-000658587Publication status
publishedExternal links
Search print copy at ETH Library
Publisher
ETH ZurichOrganisational unit
03766 - Pollefeys, Marc / Pollefeys, Marc
More
Show all metadata
ETH Bibliography
yes
Altmetrics