Video Frame Interpolation and Editing with Implicit Motion Estimation

Abstract

The amount of video data captured is steadily increasing not only in terms of quantity but also in quality due to higher spatial and temporal resolutions of cameras. This poses new challenges in processing visual data efficiently. In this thesis we focus on applications for frame interpolation and modification propagation in videos. Traditional approaches usually require some accurate pixel correspondences between the images, which is an ill-posed problem. Thus they suffer from the inherent ambiguities in correspondence estimation and are particularly sensitive to occlusion/disocclusion and changes in color or brightness. In this thesis, we present efficient and novel methods which reduce and even remove the need for computing explicit correspondences. To achieve this, we build upon recent advances in phase-based methods as well as neural networks which estimate the motion between images implicitly. First, we present a purely phase-based method for edit propagation in videos. We propose a novel algorithm to combine and adapt the phase information of the pixels in order to propagate image edits. We evaluate the flexibility by applying it to various edit applications. Second, we develop a data driven approach for the application of color propagation in grayscale videos. By combining appearance and semantics we are able to extend the temporal range to which colors can be propagated. The extended comparisons with recent methods show the superiority of our method. Finally, we propose a method for frame interpolation combining the advantages of a phase-based approach with a data driven strategy. We implement a convolutional neural network which reasons on the phase-based representation of the images. As a consequence, we are able to produce visually preferable results over optical flow for challenging scenarios containing motion blur and brightness changes. To conclude, we believe that, methods using implicit motion estimation provide an interesting and efficient alternative to traditional approaches and bear potential for many more interesting research and applications. We hope that our work provides an important step in such a direction.