Learning the loss in optical flow estimation based on the end-point-error

Abstract

Currently, deep neural networks are penetrating every corner of computer vision tasks, including optical flow estimation. Although supervised methods of optical flow estimation have facilitated more accurate motion prediction, the lack of labeled training data has hindered this progress. As a result, unsupervised and semi-supervised methods have attracted enormous interest in optical flow estimation. Contrastive learning serves as one of the primary bases of some semi-supervised methods. Despite the advances of combining contrastive learning and flow estimation by using the loss between the positive and negative prediction as the semi-supervised loss term, there are neither no full investigations on whether such a loss can be estimated nor a clear indication of the extent of the penalties on the negative samples. The aim of this work is to investigate whether such a flow error can be estimated in a supervised fashion given two consecutive images and the corresponding estimated optical flow only. We proposed three architectures of error estimation networks and performed experiments on them, in which the update of parameters is supervised by the end-point error to the ground truth flow error during training. The ground truth flow error is the difference between the ground truth flow and the estimated flow. The evaluation results indicate that with a proper combination of error and flow estimation networks, the flow error can be estimated to some extent, especially on the synthetic dataset FlyingChairs2. Furthermore, we fine-tune the RAFT flow estimation networks with the validation samples of FlyingChairs2 by means of the error-based semi-supervised methods and improve the accuracy by approximately 4.7% in AEPE on the FlyingChairs2 dataset. In summary, we would conclude that the RAFT-like and GMA-like error estimation networks are able to predict flow errors. Moreover, the estimated flow error can be utilized as a potential direction to improve the optical flow estimation in a semi-supervised manner.