Towards Reinforcement Learning for Non-Stationary Environments

In the Reinforcement Learning paradigm, environments that change between (or during) episodes are known as non-stationary. This property poses a challenge for traditional Reinforcement Learning approaches due to the fact that such methods typically rely on information that has been learned in previous episodes. However, in a non-stationary task, such information is usually an obstacle to learning that can lead to worse-than-random performance because it misleads the agent. Although an active area of research, most existing RL approaches also suffer from poor sample efficiency and struggle when the size of the state space increases. This work introduces a novel Reinforcement Learning approach that performs well in non-stationary environments, irrespective of the size of the state space. The approach proposed is termed Idea-based Reinforcement Learning, a symbolic method that can be applied to any problem that is a fully observable Markov Decision Process. IbRL is proven to perform statistically significantly above chance-level in all experiments and performs better than PPO2 in non-stationary problems where the state space is large.