Space3D-Bench: Spatial 3D Question Answering Benchmark
METADATA ONLY
Loading...
Author / Producer
Date
2025
Publication Type
Conference Paper
ETH Bibliography
yes
Citations
Altmetric
METADATA ONLY
Data
Rights / License
Abstract
Answering questions about the spatial properties of the environment poses challenges for existing language and vision foundation models due to a lack of understanding of the 3D world notably in terms of relationships between objects. To push the field forward, multiple 3D Q&A datasets were proposed which, overall, provide a variety of questions, but they individually focus on particular aspects of 3D reasoning or are limited in terms of data modalities. To address this, we present Space3D-Bench – a collection of 1000 general spatial questions and answers related to scenes of the Replica dataset which offers a variety of data modalities: point clouds, posed RGB-D images, navigation meshes and 3D object detections. To ensure that the questions cover a wide range of 3D objectives, we propose an indoor spatial questions taxonomy inspired by geographic information systems and use it to balance the dataset. Moreover, we provide an assessment system that grades natural language responses based on predefined ground-truth answers by leveraging a Vision Language Model’s comprehension of both text and images to compare the responses with ground-truth textual information or relevant visual data. Finally, we introduce a baseline called RAG3D-Chat integrating the world understanding of foundation models with rich context retrieval, achieving an accuracy of 67% on the proposed dataset.
Permanent link
Publication status
published
External links
Book title
Computer Vision – ECCV 2024 Workshops
Journal / series
Volume
15645
Pages / Article No.
68 - 85
Publisher
Springer
Event
3rd Workshop on Open-Vocabulary 3D Scene Understanding (OpenSUN3D @ ECCV 2024)
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Spatial Q&A Benchmark; Retrieval-augmented generation; Vision language model
Organisational unit
03766 - Pollefeys, Marc / Pollefeys, Marc
Notes
Funding
Related publications and datasets
Is new version of: 10.48550/arXiv.2408.16662