Myungsub Choi, Janghoon Choi, Sungyong Baik, Tae Hyun Kim, Kyoung Mu Lee

Project Paper-CVF Paper-ArXiv Supp

Paper

Abstract

Video frame interpolation is a challenging problem be-cause there are different scenarios for each video depending on the variety of foreground and background motion,frame rate, and occlusion. It is therefore difficult for a single network with fixed parameters to generalize across different videos. Ideally, one could have a different network for each scenario, but this is computationally infeasible for practical applications. In this work, we propose to adapt the model to each video by making use of additional information that is readily available at test time and yet has not been exploited in previous works. We first show the benefits of ‘test-time adaptation’ through simple fine-tuning of a network, then we greatly improve its efficiency by incorporating meta-learning. We obtain significant performance gains with only a single gradient update without any additional parameters. Finally, we show that our meta-learning framework can be easily employed to any video frame interpolation network and can consistently improve its performance on multiple benchmark datasets.

Model

Dataset Preparation

Results

Additional Results Video

Video

Citation

If you find this code useful for your research, please consider citing the following paper:

@inproceedings{choi2020meta,
    author = {Choi, Myungsub and Choi, Janghoon and Baik, Sungyong and Kim, Tae Hyun and Lee, Kyoung Mu},
    title = {Scene-Adaptive Video Frame Interpolation via Meta-Learning},
    booktitle = {CVPR},
    year = {2020}
}

Acknowledgement

The main structure of this code is based on MAML++. Training scripts for each of the frame interpolation method is adopted from: [DVF], [SuperSloMo], [SepConv], [DAIN], [CAIN] We thank the authors for sharing the codes for their great works.

For further questions, please contact @myungsub