Ideally, the agent must associate with each action at the respective reward r in order to then choose the most rewarded behavior for achieving the goal. This approach is therefore impracticable for complex problems, in which the number of states is particularly high and consequently the possible associations increase exponentially.

This problem is called the exploration-exploitation dilemma. Ideally, the agent must explore all possible actions for each state, finding the one that is actually most rewarded for exploiting it in achieving its goal.

Thus, decision-making involves a fundamental choice:

• Exploitation: Make the best decision given current information
• Exploration: Collect more information

In this process, the best long-term strategy can lead to considerable sacrifices in the short term. Therefore, it is necessary to gather enough information to make the best decisions.

Here are some examples of adopting this technique for real-life cases:

Selection of a store:

• Exploitation: Go to your favorite store
• Exploration: Try a new store

Choice of a route:

• Exploitation: Choose the best route so far
• Exploration: Try a new route

In practice, in very complex problems, convergence to a very good strategy would be too slow.

A good solution to the problem is to find a balance between exploration and exploitation:

• An agent who limits himself to exploring will always act in a casual way in every state, and it is evident that convergence to an optimal strategy is impossible
• If an agent explores little, it will always use the usual actions, which may not be optimal ones