Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. A promising approach is to learn a latent representation together with the control policy. However, fitting a high-capacity encoder using a scarce reward signal is sample inefficient and leads to poor performance. Prior work has shown that auxiliary losses, such as image reconstruction, can aid efficient representation learning. However, incorporating reconstruction loss into an off-policy learning algorithm often leads to training instability. We explore the underlying reasons and identify variational autoencoders, used by previous investigations, as the cause of the divergence. Following these findings, we propose effective techniques to improve training stability. This results in a simple approach capable of matching state-of-the-art model-free and model-based algorithms on MuJoCo control tasks. Furthermore, our approach demonstrates robustness to observational noise, surpassing existing approaches in this setting.
The challenge is to efficiently learn a mapping from pixels to an appropriate representation for control using only a sparse reward signal. Although deep convolutional encoders can learn good representations (upon which a policy can be trained), they require large amounts of training data. As existing reinforcement learning approaches already have poor sample complexity, this makes direct use of pixel-based inputs prohibitively slow. For example, model-free methods on Atari and DeepMind Control take tens of millions of steps, which is impractical in many applications, especially robotics.
Some natural solutions to improve sample efficiency are i) to use off-policy methods and ii) add an auxiliary task with an unsupervised objective. Off-policy methods enable more efficient sample re-use, while the simplest auxiliary task is an autoencoder with a pixel reconstruction objective. These model-based reinforcement learning methods often show improved sample efficiency, but with the additional complexity of balancing various auxiliary losses, such as a dynamics loss, reward loss, and decoder loss in addition to the original policy and value optimizations. These proposed methods correspondingly brittle to hyperparameter settings, and difficult to reproduce, as they balance multiple training objectives.
Cameras are a convenient and inexpensive way to acquire state information, especially in complex, unstructured environments, where effective control requires access to the proprioceptive state of the underlying dynamics. Thus, having effective RL approaches that can utilize pixels as input would potentially enable solutions for a wide range of real world applications, for example robotics.
The core objective of standard RL is to learn a policy that can maximise the agent’s expected cumulative reward. An important modification augments this objective with an entropy term to encourage exploration and robustness to noise.
Methodology and Key Findings
The research identifies variational autoencoders as a primary cause of training instability when combined with off-policy RL algorithms. Variational autoencoders introduce stochasticity and specific loss functions that can conflict with the stability requirements of off-policy learning, leading to divergence. By moving away from this architecture, the proposed method achieves greater stability.
The resulting approach, which is simple in its design, demonstrates the ability to match the performance of both state-of-the-art model-free and model-based algorithms on MuJoCo control tasks. MuJoCo is a widely used physics simulation environment for robotics and control research. This indicates that the method is competitive with more complex, often harder-to-reproduce, model-based approaches that typically require balancing multiple auxiliary losses.
Furthermore, a significant finding is the robustness of this method to observational noise. In real-world applications, camera inputs can be noisy, and traditional pixel-based RL methods often degrade in performance under such conditions. The proposed approach shows superior performance in this setting compared to existing methods, making it more practical for deployment in imperfect environments.
Applications and Broader Context
The implications of improving sample efficiency in model-free RL from images extend to several domains. In robotics, where collecting real-world data is time-consuming and costly, methods that learn directly from visual inputs with fewer samples are highly valuable. The ability to use inexpensive cameras as state sensors can lower the barrier to entry for deploying RL in complex, unstructured environments.
The work also contributes to the broader field of visual reinforcement learning. By addressing the fundamental issue of sample inefficiency and training instability, it provides a more reliable foundation for future research. The identification of the variational autoencoder as a source of instability is a critical insight that can guide the development of more stable and efficient learning algorithms.
Other related research directions mentioned in the source data include methods that store embeddings for efficient reinforcement learning (SEER), modifications to the RL problem that maximise total correlation within trajectories, and novel algorithms like Langevin Soft Actor-Critic (LSAC) that prioritise uncertainty-driven critic learning. There is also work on using intrinsically motivated stimuli, such as novelty and surprise, to improve exploration in sparsely rewarded environments, and the application of time reversal symmetry to enhance sample efficiency. These diverse approaches highlight the active and multifaceted effort within the machine learning community to overcome the sample efficiency challenge in RL, particularly when learning from high-dimensional visual inputs.
Conclusion
Improving sample efficiency in model-free reinforcement learning from images is a critical challenge for enabling practical applications in robotics and other real-world domains. The research highlights the instability caused by variational autoencoders when used as auxiliary losses in off-policy RL and proposes a stable, simple alternative. This approach matches state-of-the-art performance on benchmark tasks and demonstrates superior robustness to observational noise. The findings provide a valuable step forward in making pixel-based RL more efficient, stable, and applicable to complex, real-world control problems.