Improved RAFT architectures for optical flow estimation

Abstract

The estimation of optical flow, that is computing the displacement field between two images, is a useful tool in computer vision that has many applications as part of larger frameworks. RAFT [70], a recent method for optical flow estimation, has significantly improved the quality of results on realistic benchmarks over previous approaches, while simultaneously reducing model complexity and training cost. Despite these advancements, RAFT still has several shortcomings including its flow upsampling that can only capture high-resolution details to a limited extent, simple cost volume without normalization, and limited incorporation of multiple frames in sequences. Due to its novelty, the method has also not been applied to related tasks, such as unsupervised optical flow estimation. To address this, we propose several remedies to these mentioned shortcomings of RAFT, including different cost volume normalization strategies and alternative matching cost functions, as well as different flow upsampling strategies that can capture more high-resolution details. We also extend the method to unsupervised training as well as online training, which involves multiple frames of sequences. In the context of unsupervised training, we introduce learned losses that can be applied to arbitrary model architectures and improve results over traditional photometric and smoothness losses. Our online learning approaches yield an improvement over RAFT’s warm start and use multi-frame consistency to improve performance on video sequences. We evaluate our approaches on optical flow benchmarks and find that our modifications represent improvements over RAFT when working within a limited computational budget. We also argue that these result should scale for training configurations without such limitations.