Rainbow (https://arxiv.org/pdf/1710.02298.pdf) is an off-policy deep reinforcement learning algorithm based on DQN. We looked at and implemented deep Q-learning (DQN) and some of the extensions to DQN in Chapter 6, Implementing an Intelligent Agent for Optimal Discrete Control Using Deep Q-Learning. There have been several more extensions and improvements to the DQN algorithm. Rainbow combines six of those extensions and shows that the combination works much better. Rainbow is a state-of-the art algorithm that currently holds the record for the highest score on all Atari games. If you are wondering why the algorithm is named Rainbow, it is most probably due to the fact that it combines seven (the number of colors in a rainbow) extensions to the Q-learning algorithm, namely:

• DQN
• Double Q-Learning
• Prioritized experience replay
• Dueling networks
• Multi-step learning/n-step learning
• Distributional RL
• Noisy nets