Evaluation of different image representations for reinforcement learning agents

Abstract

Though Deep Reinforcement Learning (DRL) has emerged as a powerful paradigm for training agents to perform complex tasks, it encounters challenges when confronted with raw sensory inputs. Despite using the deep neural network’s prowess to generate meaningful internal representations, DRL approaches suffer from a high sample complexity. The effectiveness and scalability of DRL techniques are frequently hindered by the high-dimensional nature of input data, especially in methods utilizing image-based observations. To overcome this challenge, a promising approach is to start with improved input representations to enhance learning performance significantly. This work addresses this challenge using novel techniques to enhance DRL agents’ training efficiency and performance. We propose using compact and structured image representations, namely object-centric and scene graph-based state representations, as intermediate state representations for training lightweight DRL agents. These representations facilitate extracting important features from raw observations, effectively reducing input space dimensionality. To assess the effectiveness of our proposed approaches, we conduct experiments on three Atari 2600 games: Space Invaders, Frostbite, and Freeway. Our findings reveal that models trained with intermediate state representations, while showing slightly lower performance than those trained from raw image pixels, achieved a notable performance by surpassing Human Normalized Score (HNS) in one game environment with fewer model parameters. Furthermore, we investigate alternative loss functions for value function estimation and explore strategies to mitigate the issue of diminishing entropy during training. Finally, through a systematic analysis of experimental findings, we provide valuable insights into the efficacy and drawbacks of these approaches, shedding light on promising avenues for future research in formulating suitable state spaces for training agents using DRL.