Advanced data augmentation for the RAFT optical flow approach

Abstract

We add several new augmentation methods to RAFT, a deep learning architecture that is used to calculate the optical flow between two sequential images. Because RAFT is trained using supervised learning, it requires annotated training data that not only contains image sequences but also the corresponding ground truth optical flow. Since the optical flow cannot be automatically generated from arbitrary image sequences, synthetic data sets are created to train these networks. One drawback of these data sets is their small size and low variety of optical flows they contain. To increase this variety, one option is to use data augmentation techniques to modify the training samples before feeding them to the network. These augmentations can change the images of a sample on the pixel level, but also modify the geometry of these images and hence the optical flow as well. We conduct experiments during each training phase to find out which kind of augmentation at which intensity is able to increase the accuracy of the trained model when estimating the optical flow of MPI-Sintel. Furthermore we compare this accuracy to that achieved by the original RAFT implementation. We find out that it depends on the specific training phase which kind of augmentation and which intensity is beneficial for the model’s performance. The model that uses our augmentations is able to beat the original RAFT implementation after both are trained on FlyingChairs and after both are trained FlyingChairs and FlyingThings3D afterwards. When using these models to estimate the optical flow of KITTI-15, these models then perform worse, which shows that ideal augmentation settings are dependent on the target data set. The results after training on MPI-Sintel in the third phase show that adding these augmentations does not necessarily improve the model’s performance, as the model that uses advanced augmentations doesn’t manage to beat the original RAFT implementation.