CAIN

Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, Kyoung Mu Lee

2nd place in [AIM 2019 ICCV Workshop] - Video Temporal Super-Resolution Challenge

Abstract

Prevailing video frame interpolation techniques rely heavily on optical flow estimation and require additional model complexity and computational cost; it is also susceptible to error propagation in challenging scenarios with large motion and heavy occlusion. To alleviate the limitation, we propose a simple but effective deep neural network for video frame interpolation, which is end-to-end trainable and is free from a motion estimation network component. Our algorithm employs a special feature reshaping operation, referred to as PixelShuffle, with a channel attention, which replaces the optical flow computation module. The main idea behind the design is to distribute the information in a feature map into multiple channels and extract motion information by attending the channels for pixel-level frame synthesis. The model given by this principle turns out to be effective in the presence of challenging motion and occlusion. We construct a comprehensive evaluation benchmark and demonstrate that the proposed approach achieves outstanding performance compared to the existing models with a component for optical flow computation.

Model

• Download pretrained CAIN model from [Here]

Dataset

• [ Vimeo90K Triplet dataset ]
• [ SNU-FILM benchmark ] : SNU Frame Interpolation with Large Motion evaluation benchmark
  • Our benchmark consists of test split of GOPRO dataset + manually collected video sequences from YouTube.
  • The evaluation is 4 different settings: Easy, Medium, Hard, Extreme
    • The average motion magnitude increases from Easy to Extreme
  • Sample image (from GOPRO) and visualizations of its motion magnitude w.r.t. each evaluation setting

Download

• Selected frame triplets for evaluation: [Download link]
• All RGB frames: [Download link]

Results

Video

Citation

If you find this code useful for your research, please consider citing the following paper:

@inproceedings{choi2020cain,
    author = {Choi, Myungsub and Kim, Heewon and Han, Bohyung and Xu, Ning and Lee, Kyoung Mu},
    title = {Channel Attention Is All You Need for Video Frame Interpolation},
    booktitle = {AAAI},
    year = {2020}
}

Acknowledgement

Many parts of this code is adapted from:

• EDSR-Pytorch
• RCAN

We thank the authors for sharing codes for their great works.

For further questions, please contact @myungsub)