Kavli Affiliate: Xiang Zhang
| First 5 Authors: Yangguang Wang, Xiang Zhang, Mingyuan Lin, Lei Yu, Boxin Shi
| Summary:
Scene Dynamic Recovery (SDR) by inverting distorted Rolling Shutter (RS)
images to an undistorted high frame-rate Global Shutter (GS) video is a
severely ill-posed problem, particularly when prior knowledge about
camera/object motions is unavailable. Commonly used artificial assumptions on
motion linearity and data-specific characteristics, regarding the temporal
dynamics information embedded in the RS scanlines, are prone to producing
sub-optimal solutions in real-world scenarios. To address this challenge, we
propose an event-based RS2GS framework within a self-supervised learning
paradigm that leverages the extremely high temporal resolution of event cameras
to provide accurate inter/intra-frame information. % In this paper, we propose
to leverage the event camera to provide inter/intra-frame information as the
emitted events have an extremely high temporal resolution and learn an
event-based RS2GS network within a self-supervised learning framework, where
real-world events and RS images can be exploited to alleviate the performance
degradation caused by the domain gap between the synthesized and real data.
Specifically, an Event-based Inter/intra-frame Compensator (E-IC) is proposed
to predict the per-pixel dynamic between arbitrary time intervals, including
the temporal transition and spatial translation. Exploring connections in terms
of RS-RS, RS-GS, and GS-RS, we explicitly formulate mutual constraints with the
proposed E-IC, resulting in supervisions without ground-truth GS images.
Extensive evaluations over synthetic and real datasets demonstrate that the
proposed method achieves state-of-the-art and shows remarkable performance for
event-based RS2GS inversion in real-world scenarios. The dataset and code are
available at https://w3un.github.io/selfunroll/.
| Search Query: [#feed_custom_title]