Motion Blur Decomposition with Cross-shutter Guidance

Xiang Ji      Haiyang Jiang      Yinqiang Zheng
The University of Tokyo
Paper arXiv Code Data

Retrieving a sequence from degradation is highly ambiguous for either blur or RS inputs. The Cross-Shutter strategy provides a way to simultaneously deal with the ambiguities of blur decomposition and RS interpolation.

Abstract

Motion blur is a frequently observed image artifact, especially under insufficient illumination where exposure time has to be prolonged so as to collect more photons for a bright enough image. Rather than simply removing such blurring effects, recent researches have aimed at decomposing a blurry image into multiple sharp images with spatial and temporal coherence. Since motion blur decomposition itself is highly ambiguous, priors from neighbouring frames or human annotation are usually needed for motion disambiguation. In this paper, inspired by the complementary exposure characteristics of a global shutter (GS) camera and a rolling shutter (RS) camera, we propose to utilize the ordered scanline-wise delay in a rolling shutter image to robustify motion decomposition of a single blurry image. To evaluate this novel dual imaging setting, we construct a triaxial system to collect realistic data, as well as a deep network architecture that explicitly addresses temporal and contextual information through reciprocal branches for cross-shutter motion blur decomposition. Experiment results have verified the effectiveness of our proposed algorithm, as well as the validity of our dual imaging setting.

Method Overview

The overall architecture contains two stages: motion interpretation and blur decomposition. Blur decomposition is implemented through a GenNet. Motion interpretation takes as input a blur image and an RS image along with its temporal positional encoding. It consists of three blocks and one of them is unfolded in (b). (c) presents specific details of shutter alignment and aggregation (SAA). Feature extracted by encoder block (EB) will be converted using spatial transformer network (STN), and then enhanced through a Conv. block to accurately predict displacement field between shutters.


Optical System and RealBR Dataset



Rather than a biaxial system for image-to-image deblurring, we develop a triaxial imaging system that simultaneously captures Blur-RS pairs along with high-speed ground truth, and collect a real dataset named RealBR (samples shown as below).

Input (Blur)
Input (RS)
GT (HS)

Comparisons on RealBR

Quantitative comparisons of reconstructed latent frame sequence with lengths of 3, 5 and 9 on RealBR.


Comparisons on Synthetic Data

We conduct comparisons with more competitive settings on synthetic data based on GOPRO.


Challenging Scenarios

Low-lit Scenes: We further explore effects of noisy RS observations to our method.

RS (noisy)
Blur
Ours
GT

Misaligned Views: We randomly shift RS view in image space and provide comparisons with strictly-aligned views.

BibTeX

@article{ji2024motion,
  title={Motion Blur Decomposition with Cross-shutter Guidance},
  author={Ji, Xiang and Jiang, Haiyang and Zheng, Yinqiang},
  journal={arXiv preprint arXiv:2404.01120},
  year={2024}
}