Event-Based Fusion for Motion Deblurring with Cross-modal Attention
OPEN ACCESS
Loading...
Author / Producer
Date
2022
Publication Type
Conference Paper
ETH Bibliography
yes
Citations
Altmetric
OPEN ACCESS
Data
Rights / License
Abstract
Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution, providing valid image degradation information within the exposure time. In this paper, we rethink the event-based image deblurring problem and unfold it into an end-to-end two-stage image restoration network. To effectively fuse event and image features, we design an event-image cross-modal attention module applied at multiple levels of our network, which allows to focus on relevant features from the event branch and filter out noise. We also introduce a novel symmetric cumulative event representation specifically for image deblurring as well as an event mask gated connection between the two stages of our network which helps avoid information loss. At the dataset level, to foster event-based motion deblurring and to facilitate evaluation on challenging real-world images, we introduce the Real Event Blur (REBlur) dataset, captured with an event camera in an illumination-controlled optical laboratory. Our Event Fusion Network (EFNet) sets the new state of the art in motion deblurring, surpassing both the prior best-performing image-based method and all event-based methods with public implementations on the GoPro dataset (by up to 2.47 dB) and on our REBlur dataset, even in extreme blurry conditions. The code and our REBlur dataset are available at https://ahupujr.github.io/EFNet/.
Permanent link
Publication status
published
External links
Book title
Computer Vision - ECCV 2022
Journal / series
Volume
13678
Pages / Article No.
412 - 428
Publisher
Springer
Event
17th European Conference on Computer Vision (ECCV 2022)
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Computer Vision; Event-based vision; Event cameras; Deblurring; Sensor fusion
Organisational unit
03514 - Van Gool, Luc (emeritus) / Van Gool, Luc (emeritus)