Improved descriptor Learning for correspondence problems

Abstract

Solving correspondence problems is a fundamental task in computer vision. In the past decades, many approaches tried to find the matches between images. One way to solve this task is to use feature-based methods, such as SIFT, SURF and DAISY. The mentioned methods are based on engineered features.

Learned features are another type of descriptors that are basically learned and computed via a convolutional neural network (CNN). This type of descriptors has gained a lot of attention in the past years. In this thesis, we design and train many CNN to find an architecture and a set of parameters, so that it computes suitable descriptors for the optical flow estimation task.

We implement a fast way of computing the descriptors from images, based on a strategy suggested by Bailer et al. After computing the non-dense set of matches by the Coarse-to-Fine PatchMatch (CPM) algorithm, we use the interpolation technique which is introduced and used in Edge-Preserving Interpolation of Correspondences for Optical Flow (Epicflow), to compute the dense flow field.

We use the approach of CPM, which is a popular method to estimate optical flow for large displacements. We embed our trained CNN model into CPM in such a way that the algorithm uses the learned descriptors to find the matches in a coarse-to-fine manner.

We use the recent benchmarks of KITTI 2015 and MPI-Sintel to train and evaluate our CNN. To this end, we compare the estimated optical flow to the ground truth optical flow provided in the mentioned benchmarks. By computing the Average End-Point Error (AEE) of the obtained optical flow, we thus have a measure to assess the learned descriptors and we can use it as a feedback to change the network so that it leads to more suitable descriptors. In this way, we were able to design and train a network that computes descriptors which perform better than DAISY and SIFT for the MPI-Sintel dataset.