Analysing the quality of interframe interpolation with optical flow

Abstract

The generation of an intermediate frame between two consecutive frames in a video is an ongoing challenge in computer vision. This concept is known as motion interpolation, or interframe interpolation, and usually works by following the path of an optical flow. To obtain accurate interpolated images, a good optical flow estimation is needed. Typically, it is assumed that the better a flow estimation is, the better the resulting interpolated images are. To test how the underlying optical flow influences the interpolation results, we look at two different optical flow estimations, RAFT [1] and the baseline version of ProFlow [2], and perform various interpolation experiments with them. In the experiments, we first examine the accuracy of the underlying flows estimated with RAFT and the ProFlow baseline by comparing their EPE and flow visualizations. Following this, we conduct several interpolation experiments with these flows by using different interpolation methods. Here, the focus lies on two methods called forward warping and forward forward warping, which are based on the concept of image warping. For a wide variety of datasets, we use three different benchmarks, Middlebury [3], KITTI [4], and Sintel [5]. In an oracle experiment, we then compare how the interpolation performs when ground truth flows are used and investigate different approaches to change these results. From these experiments, we find that a RAFT flow does not always produce better interpolation results than a ProFlow baseline flow, despite being more accurate. Furthermore, the oracle experiments show that using a more accurate flow estimation often creates more doubling artifacts in the interpolated image and thus a better flow does not always result in a better interpolation quality.