Multi-frame approaches for learning optical flow predictions

Abstract

In recent years, optical flow estimation techniques have shifted from classical variational approaches to learning-based methods, which show promising results with respect to both accuracy and speed. However, only few attempts have been made at using more than two frames of the input sequence for optical flow prediction. ProFlow by Maurer and Bruhn [MB18] is a self-supervised learning approach that uses three consecutive frames in order to estimate forward motion from the corresponding backward motion. By capitalizing on the temporal information contained in the sequence, ProFlow manages to improve performance in occluded regions, where a good estimation is particularly difficult. In this work, ProFlow is extended such that it uses backward motions over multiple frames to predict the corresponding forward motion, thus leveraging additional information from the past. Four different multi-frame modifications are introduced that each enable a forward flow estimation from multiple backward flows. The best performing approach employs several convolutional neural networks (CNNs) where each network is trained on a separate backward flow. By explicitly combining the respective predictions, the model achieves an improvement of between 10% and 15% on the Sintel datasets, proving that additional temporal information can be leveraged for optical flow prediction.