Object Segmentation by Mining Cross-Modal Semantics
METADATA ONLY
Loading...
Author / Producer
Date
2023-10
Publication Type
Conference Paper
ETH Bibliography
yes
Citations
Altmetric
METADATA ONLY
Data
Rights / License
Abstract
Multi-sensor clues have shown promise for object segmentation, but inherent noise in each sensor, as well as the calibration error in practice, may bias the segmentation accuracy. In this paper, we propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features, with the aim of controlling the modal contribution based on relative entropy. We explore semantics among the multimodal inputs in two aspects: the modality-shared consistency and the modality-specific variation. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision. On the one hand, the AF block explicitly dissociates the shared and specific representation and learns to weight the modal contribution by adjusting the proportion, region, and pattern, depending upon the quality. On the other hand, our CFD initially decodes the shared feature and then refines the output through specificity-aware querying. Further, we enforce semantic consistency across the decoding layers to enable interaction across network hierarchies, improving feature discriminability. Exhaustive comparison on eleven datasets with depth or thermal clues, and on two challenging tasks, namely salient and camouflage object segmentation, validate our effectiveness in terms of both performance and robustness. The source code is publicly available at https://github.com/Zongwei97/XMSNet.
Permanent link
Publication status
published
External links
Editor
Book title
MM '23: Proceedings of the 31st ACM International Conference on Multimedia
Journal / series
Volume
Pages / Article No.
3455 - 3464
Publisher
Association for Computing Machinery
Event
31st ACM International Conference on Multimedia (MM 2023)
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
RGB-X Object Segmentation; Cross-Modal Semantics; Robustness