Volumetric Instance-Level Semantic Mapping Via Multi-View 2D-to-3D Label Diffusion
OPEN ACCESS
Loading...
Author / Producer
Date
2022-04
Publication Type
Journal Article
ETH Bibliography
yes
Citations
Altmetric
OPEN ACCESS
Data
Rights / License
Abstract
Robots operating in real-world settings often need to plan interactions with surrounding scene elements and therefore, it is crucial for them to understand their workspace at the level of individual objects. In this spirit, this work presents a novel approach to progressively build instance-level, dense 3D maps from color and depth cues acquired by either a moving RGB-D sensor or a camera-LiDAR setup, whose pose is being tracked. The proposed framework processes each input RGB image with a semantic instance segmentation neural network and uses depth information to extract a set of per-frame, semantically labeled 3D instance segments, which then get matched to object instances already identified in previous views. Following integration of these newly detected instance segments in a global volumetric map, an efficient label diffusion scheme that considers multi-view instance predictions together with the reconstructed scene geometry is used to refine 3D segmentation boundaries. Experiments on indoor benchmarking RGB-D sequences show that the proposed system achieves state-of-the-art performance in terms of 3D segmentation accuracy, while reducing the computational processing cost required at each frame. Furthermore, the applicability of the system to challenging domains outside the traditional office scenes is demonstrated by testing it on a robotic excavator equipped with a calibrated camera-LiDAR setup, with the goal of segmenting individual boulders in a highly cluttered construction scenario.
Permanent link
Publication status
published
External links
Editor
Book title
Journal / series
Volume
7 (2)
Pages / Article No.
3531 - 3538
Publisher
IEEE
Event
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Object Detection and Segmentation; RGB-D Perception; Mapping
Organisational unit
09559 - Chli, Margarita (ehemalig) / Chli, Margarita (former)
02284 - NFS Digitale Fabrikation / NCCR Digital Fabrication
Notes
Funding
Related publications and datasets
Is part of: 10.3929/ethz-b-000646462