Intraoperative localization and scene reconstruction using differentiable rendering and graph-based landmark registration with application to cystoscopy

Abstract

Minimally invasive procedures, such as bladder endoscopy, reduce trauma but present navigation challenges due to restricted visibility. Traditional mapping methods are often insufficient, and comprehensive solutions in the literature are scarce. This work introduces a novel approach for intraoperative navigation and scene reconstruction, focusing on deformable environments, such as those encountered in cystoscopy. The proposed reconstruction concept relies on a monocular camera image but can be flexibly extended to include additional sensor data. The fundamental reconstruction strategy employed in this work follows the question: How does the model representation and camera perspective need to be adjusted such that the rendering of the model matches the current observation? The reconstruction process requires the formulation of several optimization objectives, necessitating a fully differentiable rendering approach. To this end, a novel formulation of an inverse rendering process is proposed, where 2D image data is projected from the camera's perspective onto a 3D mesh model. Vascular structures, which remain consistent despite deformations, serve as dependable landmarks. Combining graph-based landmark recognition with rendering-based reconstruction techniques offers a comprehensive solution for determining the intraoperative location and scene, especially considering the complexity of deformable surgical environments.