Metadata only
Date
2024-01-15Type
- Conference Paper
ETH Bibliography
yes
Altmetrics
Abstract
VLN-CE is a recently released embodied task, where AI agents need to navigate a freely traversable environment to reach a distant target location, given language instructions. It poses great challenges due to the huge space of possible strategies. Driven by the belief that the ability to anticipate the consequences of future actions is crucial for the emergence of intelligent and interpretable planning behavior, we propose DREAMWALKER - a world model based VLN-CE agent. The world model is built to summarize the visual, topological, and dynamic properties of the complicated continuous environment into a discrete, structured, and compact representation. DREAMWALKER can simulate and evaluate possible plans entirely in such internal abstract world, before executing costly actions. As opposed to existing modelfree VLN-CE agents simply making greedy decisions in the real world, which easily results in shortsighted behaviors, DREAMWALKER is able to make strategic planning through large amounts of "mental experiments." Moreover, the imagined future scenarios reflect our agent's intention, making its decision-making process more transparent. Extensive experiments and ablation studies on VLN-CE dataset confirm the effectiveness of the proposed approach and outline fruitful directions for future work. Show more
Publication status
publishedExternal links
Book title
2023 IEEE/CVF International Conference on Computer Vision (ICCV)Pages / Article No.
Publisher
IEEEEvent
Organisational unit
02652 - Institut für Bildverarbeitung / Computer Vision Laboratory
Notes
Conference lecture held on October 5, 2023.More
Show all metadata
ETH Bibliography
yes
Altmetrics