Interactive remote-visualisation for large displays

Abstract

While visualisation often strives for abstraction, the interactive exploration of large scientific data sets, such as densely sampled 3D fields, massive particle data sets or molecular visualisations still benefits from rendering their graphical representation in large detail on high-resolution displays such as Powerwalls or tiled display walls. With the ever-growing size of data, and the increased availability of the aforementioned displays, collaboration becomes desirable in the sense of sharing this type of a visualisation running on one site in real time with another high-resolution display on a remote site. While most desktop computers - and in turn the visualisation software running on them - are alike, large high-resolution display setups are often unique, making use of multiple GPUs, a GPU cluster or only CPUs to drive the display. Therefore, particularly if the goal is the interactive scientific visualisation of large data sets, unique software might have to be written for a unique display and compute system. Molecular visualisations are one application domain in which users would clearly benefit from being able to collaborate remotely, combining video and audio conference setups with the possibility of sharing high-resolution interactive visualisations. However, for large - often tiled - displays and image resolutions beyond 4K no obvious generic, let alone commercial, solution exists. While there are specialized solutions that support sharing the output of these displays, based on hardware-accelerated video encoding, these make compromises between quality and bandwidth. They either deliver a high quality image and therefore induce bandwidth requirements that cannot generally be met, or they uniformly decrease the quality to maintain adequate frame rates. However, in visualisation in particular, details are crucial in areas that are currently being investigated. Hence, an interactive remote-visualisation for high-resolution displays requires new methods that can run on different hardware setups and offer a high image quality while reducing the required bandwidth as much as possible. In this dissertation, an innovative technique for rendering and comparing molecular surfaces as well as a novel system that supports interactive remote-visualisation, for molecular surfaces and other scientific visualisations, between different high-resolution displays is introduced and discussed. The rendering technique solves the view-dependency and occlusion of the three dimensional representation of the molecular surfaces by showing the topography and the physico-chemical properties of the surface in one single image. This also allows analysts to compare and cluster the images in order to understand the relationship structures, based on the idea that a visually similar surface implies a similarity in the function of the protein. The system presented in this dissertation uses a low latency pixel streaming approach, leveraging GPU-based video encoding and decoding to solve the aforementioned problems and to allow for interactive remote visualisations on large high-resolution displays. In addition to remote-visualisation the system offers collaboration capabilities via bidirectional video and audio simultaneously. The system is based on the fact that, regardless of the underlying hardware setup, large displays share one property: they have a large (distributed or not) frame buffer to display coloured pixels. Consequently, this allows the users to collaborate between two sites that use different display walls with only a minimal delay. To address the bandwidth limitations, several methods have been developed and introduced which aim to reduce the required bandwidth and the end-to-end latency while still offering high image quality. The aim of these methods is to reduce the image quality and therefore the required bandwidth in regions that are not currently of interest to the users, while those that are of interest remain at a high quality. These methods can be categorised into algorithmic and user-driven optimisations to the remote visualisation pipeline. The user-driven optimisations make use of gaze tracking to adapt the quality of the encoding locally while the algorithmic optimisations use the content of the frames. Algorithmic optimisations include the usage of a convolutional neural network to detect regions of interest and adapt the encoding quality accordingly and a temporal downsampling prior to the encoding. These methods can also be combined, for example, foveated encoding may be combined with temporal downsampling to further reduce the required bandwidth and the latency. Overall, this dissertation advances the state of the art by enabling the collaborative analysis of molecular and other scientific visualisations remotely at interactive frame rates without imposing bandwidth requirements that cannot generally be met.