Time Lens++: Event-based Frame Interpolation with Parametric Nonlinear Flow and Multi-scale Fusion
METADATA ONLY
Loading...
Author / Producer
Date
2022
Publication Type
Conference Paper
ETH Bibliography
yes
Citations
Altmetric
METADATA ONLY
Data
Rights / License
Abstract
Recently, video frame interpolation using a combination of frame- and event-based cameras has surpassed traditional image-based methods both in terms of performance and memory efficiency. However, current methods still suffer from (i) brittle image-level fusion of complementary interpolation results, that fails in the presence of artifacts in the fused image, (ii) potentially temporally inconsistent and inefficient motion estimation procedures, that run for every inserted frame and (iii) low contrast regions that do not trigger events, and thus cause events-only motion estimation to generate artifacts. Moreover, previous methods were only tested on datasets consisting of planar and faraway scenes, which do not capture the full complexity of the real world. In this work, we address the above problems by introducing multi-scale feature-level fusion and computing one-shot non-linear inter-frame motion-which can be efficiently sampled for image warping-from events and images. We also collect the first large-scale events and frames dataset consisting of more than 100 challenging scenes with depth variations, captured with a new experimental setup based on a beamsplitter. We show that our method improves the reconstruction quality by up to 0.2 dB in terms of PSNR and up to 15% in LPIPS score.
Permanent link
Publication status
published
External links
Editor
Book title
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Journal / series
Volume
Pages / Article No.
17734 - 17743
Publisher
IEEE
Event
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022)
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Computational photography; Image and video synthesis and generation; Interpolation; Computer vision; Motion estimation; Memory management; Dynamics; Cameras; Pattern recognition