Real-time diminished reality through spatial 2D planar inpainting

Abstract

Diminished Reality (DR) aims to visually remove selected real-world content while preserving a plausible view of the surrounding environment. As three-dimensional (3D) scanning and spatial computing become more prevalent, DR has emerged as a critical tool for real-time privacy anonymization-allowing users to stream or capture 3D environments while automatically masking sensitive two-dimensional (2D) data like text, documents, or personal photos. Previous method have mostly focused on diminishing the 3D spatial object by inpainting occluded region. However, the special case of diminishing 2D visual content attached to planar physical surfaces-such as text, drawings, posters, and signs-requires an entirely different paradigm. Although modern image inpainting methods can produce convincing results for individual frames, applying them repeatedly to a video requires per-frame masks and may introduce flickering, texture drift, or inconsistent appearance over time. This computational and visual instability poses a significant challenge for real-time applications, such as privacy anonymization during live 3D capture. This thesis presents a geometry-aware diminished reality approach for this 2D paradigm. Instead of generating a new inpainting result for every frame, the proposed method registers each target region once using a single reference mask and a inpainted reference image. Sparse 3D reconstruction is used to associate the target with its underlying physical plane, while online camera pose estimation enables the mask and inpainted reference appearance to be propagated through the video using plane-induced warping. This allows the replacement to remain attached to the same physical surface under camera motion. Experiments on self-recorded indoor video sequences show that the proposed method can remove planar visual content while maintaining spatial alignment and temporal stability under moderate viewpoint changes, partial visibility, and multiple target surfaces. Compared with frame-wise inpainting-based methods, the proposed approach eliminates the need for continuous object detection or region segmentation and our method achieves a processing speed of nearly 20 FPS. The results demonstrate that combining sparse geometric anchoring with inpainted reference propagation is a practical alternative for stable diminished reality on planar physical surfaces, making it a highly viable solution for real-time privacy masking during 3D capture.