MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References
OPEN ACCESS
Loading...
Author / Producer
Date
2024
Publication Type
Master Thesis
ETH Bibliography
yes
Citations
Altmetric
OPEN ACCESS
Data
Rights / License
Abstract
Rendering realistic images from 3D reconstructions is an essential task of many Computer Vision and Robotics pipelines, notably for mixed-reality applications as well as for training autonomous agents in simulated environments. However, the quality of novel views heavily depends on the source reconstruction which is often imperfect due to noisy or missing geometry and appearance. Inspired by the recent success of reference-based super-resolution networks, we propose MaRINeR, a refinement method that leverages information of a nearby mapping image to improve the rendering of a target viewpoint. We first establish matches between the raw rendered image of the scene geometry from the target viewpoint and the nearby reference based on deep features, followed by hierarchical detail transfer. We show improved renderings in quantitative metrics and qualitative examples from both explicit and implicit scene representations. We further employ our method on the downstream tasks of pseudo-ground-truth validation, synthetic data enhancement and detail recovery for renderings of reduced 3D reconstructions.
Permanent link
Publication status
published
External links
Editor
Contributors
Examiner: Bauer, Zuria
Examiner : Dusmanu, Mihai
Book title
Journal / series
Volume
Pages / Article No.
Publisher
ETH Zurich
Event
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Organisational unit
03766 - Pollefeys, Marc / Pollefeys, Marc