In this section, we will discuss how we can scale up our shallow Q-learner to a more sophisticated and powerful deep Q-learner-based agent that can learn to act based on raw visual image inputs, which we will use towards the end of this chapter to train agents that play Atari games well. Note that you can train this deep Q-learning agent in any learning environments with a discrete action space. The Atari game environments are one such interesting class of environments that we will use in this book.

We will start with a deep convolutional Q-network implementation and incorporate it into our Q-learner. Then, we will see how we can use the technique of target Q-networks to improve the stability of the deep Q-learner. We will then combine all the techniques we have discussed so far to put together the full implementation of our deep Q learning-based agent.