Filter dictionaries for optical flow prediction with RAFT

Abstract

In the field of optical flow estimation, a dense vector field must be generated describing the apparent two-dimensional displacement of objects in consecutive images of a sequence. Although state of the art predictions are currently produced by deep convolutional neural networks, one major issue is that they are strongly susceptible to adversarial attacks, such as the Perturbation Constrained Flow Attack, which create small, noisy perturbations pursuing maximal change in the optical flow estimate. To improve adversarial robustness, this thesis includes receptive field convolutional layers into the optical flow predicting neural network RAFT. These receptive field layers use filter dictionaries to impose specific (geometric) priors onto convolutional kernels and improve results in image classification and reconstruction tasks. Each kernel in these RFCNNs can be written as a weighted sum over a fixed subset of filters taken from the dictionary. Besides the existing Gaussian derivative and Parseval completed sparse directional dictionaries, a novel PCA dictionary is proposed which consists of the principal components of the previously trained network’s kernels. All types of dictionaries are compared against each other at multiple positions in the network. Results show that receptive fields in individual layers mostly do not affect and in RAFT’s feature encoder even degrade performance, while Parseval completed dictionaries do not benefit the neural network in this context of optical flow. However, filter dictionaries with geometric motivations in RAFT’s update block, namely the Gaussian derivatives and sparse directional FDs, make the network up to 20% more robust against the PCFA in exchange for a worse fit in quality.