7.1.1Strategic classification of the agent learning method

Recently, many agent learning strategies have been developed, such as Rote Learning, Learning from Instructions and by Advice Taking, Learning from Examples, Inductive Learning, Learning by Analogy, Learning by Observation and Discovery, Case-Based Learning, Explanation-Based Learning, Completing Learning, Cooperating Learning, Game-theoretic Learning, Adaptive Learning (through the adaptive adjustment of all kinds of objects to stabilize a target neural network, which is then combined with a Genetic Algorithm), and Reinforcement Learning.

7.1.2Characteristics of agent learning

The agent is an intellectual body that has the capacity for self-learning, as well as the following characteristics:

Self-control: during the learning procedure, the agent does not need external guidance and has the ability to control its own learning actions.

Reactivity: during the learning procedure, the agent will respond to changes in the outside world.

Inferential: according to its own current knowledge, which means the current information included in some repository, the agent will implement a continuing rational inquiry and update the repository.

Learnability: from unknown to known, the agent learns from everything around, much like a conscious human. After obtaining the learning results, the agent will store them in the repository. Thus, the agent can directly recall knowledge from the repository when faced with a similar situation.

According to these characteristics, the agent techniques have been applied in multiple areas, e.g. game development [1–2], mail routing [3], spoken dialogue [4], and robot soccer [5].

7.1.3Related work

A typical case of single-agent learning is OpenML, which is a software development tool published by Intel Corporation in 2004. This tool enables systems built by software engineers to “learn” from the application. The key to this learning process is to use previous data to improve the accuracy and durability of the system. Moreover, this tool can predict the probability of events that have happened. OpenML is based on Bayes’ theorem. Its core philosophy is to study the frequency of an event that has taken place and predict the probability of this event happening in the future. An interesting application of OpenML is when Intel researchers used it to create an audio/video speech recognition system. This recognition system employs video cameras to detect a speaker’s facial expressions and mouth movements. This “read lips” method helps to improve the accuracy of speech recognition in noisy environments such as airports or supermarkets [6].

The ADE project at the University of Southern California is an adaptive learning system geared for education. The core of this project is to solve problems. In Stanford University, an adaptive system called “Fab” provides a web page recommendation service. All of these applications are representations of agent learning technology, collectively referred to as the interface agent technique.

A typical case of a learning agent has been developed by Chen and Sycara. Their WebMate can automatically extract resources online according to the user’s interests and can satisfy demands for the retrieval of related domain knowledge. Yang and Ning conducted a deep analysis of reinforcement learning. They proposed an ensemble method based on AODE that employs the BDI model and reinforcement learning to guide the learning process of the agent. Moreover, they studied the deviation of motivated learning. Using planning rules, they were able to overcome the shortcomings of reinforcement learning to some extent and improve the learning efficiency. The machine learning research group led by Professor Fanzhang at Soochow University has conducted a lot of theoretical work in the field of agent learning. For example, they have studied an agent self-learning model based on DFL [7] and an automated reasoning platform [8].

The ILS system of GTE [9] integrates heterogeneous agents through a centralized controller. In this way, the agents are able to judge one another. The MALE system enables different agents to cooperate on an interactive board, similar to a blackboard. Guorui proposed an emotional body system that employs the features of the genetic system, nervous system, and endocrine system in its construction. In this system, the researchers proposed a novel action learning method using emotions. The machine learning research group led by Professor Li has also reported great achievements in this area, such as a multi-agent problem-solving system based on DFL and a combinatorial mathematics learning system.

7.2.1Model structure

Definition 7.1 The agent mental model based on DFL consists of an 11-tuple (B, C, I, A, S, Sty, R, Brf, Acf, Irf, Stf), where B (Belief) represents the understanding of information and basic views of the environmental world within the range of cognitive ability of the agent model. $B(P_i,(\overleftarrow{x_i},\overrightarrow{x_i}))$ represents the agent belief of event $(\overleftarrow{x_i},\overrightarrow{x_i})$ at some point. B is the set of current beliefs, and $(\overleftarrow{B},\overrightarrow{B})=\{B(P_1,$$(\overleftarrow{x_1},\overrightarrow{x_1})),B(P_2,(\overleftarrow{x_2},\overrightarrow{x_2})),\mathrm{...},B(P_m,(\overleftarrow{x_m},\overrightarrow{x_m}))\}$ form current events.

From the definition of Belief, we know that an event is actually a pseudo-proposition; therefore, the implication of $B(P_i,(\overleftarrow{x_i},\overrightarrow{x_i}))$ is the extent to which the agent believes that proposition Pi is true, which we represent using $(\overleftarrow{x_i},\overrightarrow{x_i})$ The structure and updating algorithm of our belief database are shown in Tab. 7.1 and Algorithm 7.1. In this model structure, we will update the belief database when the agent conducts operations relevant to the belief database as follows: first, contrast the belief of the current learning process to query whether there is any information about the belief. If the belief already exists and its value is invariant, do nothing. If the value of the belief has changed, make the corresponding modifications. If the belief does not already exist, we will create and insert it at the appropriate location.

Tab. 7.1: Structure of belief database.

C (Capability) represents prerequisites for performing a task, i.e. the capacity of the current agent. $C(M_i,(\overleftarrow{x_i},\overrightarrow{x_i}))$ represents the agent’s capacity to process problem Mi at time $(\overleftarrow{x_i},\overrightarrow{x_i}).$ C is the set of capacities after the learning process, and $(\overleftarrow{C},\overrightarrow{C})=\{C(M_1,(\overleftarrow{x_1},\overrightarrow{x_1})),C(M_2,(\overleftarrow{x_2},\overrightarrow{x_2})),\mathrm{...},C(M_m,(\overleftarrow{x_m},\overrightarrow{x_m}))\},$ which assumes the current number of events is m.

Different problems result from the learning process of the agent. We need to solve these problems. First, the capacity of the agent to solve one problem may be zero. However, as the learning proceeds, the capacity of the agent is enhanced. When the agent accomplishes one step of the learning process, one of the following cases exists:

1.For a certain problem, the agent’s capacity to solve it has not changed.

2.For a certain problem, the agent’s capacity to solve it has changed.

3.For a certain problem, the agent’s capacity to solve it does not exist in the database.

The structure of the capacity database is analogous to that of the belief database, and we present the updating strategy in Algorithm 7.2.

Tab. 7.2: Structure of intention database.

I (Intention) represents the prearrangements of ones behaviour in future events and represents the action orientation of the agent for future events. In the i th step of the learning process, we assume the number of next-time executable strategies of the agent is n, which means the strategy set is $\{st{y}_1,st{y}_2,\mathrm{...},st{y}_n\}.I_i(st{y}_i,(\overleftarrow{x_i^j},\overleftarrow{x_i^j}))$ represents the intention of the executable strategy Styj in the i th learning step with respect to $(\overleftarrow{x_i^j},\overleftarrow{x_i^j}).$ Ii is the intention set of optional strategies of the agent in the i th learning step, and the number of current optional strategies is n, or $({\stackrel{\leftharpoondown }{I}}_i,\stackrel{\rightharpoondown }{I_i})=$$\{I_i(st{y}_1,(\overleftarrow{x_i^1},\overrightarrow{x_i^1}),I_i(st{y}_2,\mathrm{...}(\overleftarrow{x_i^2},\overrightarrow{x_i^2})),\mathrm{...},I_i(st{y}_n,(\overleftarrow{x_i^n},\overrightarrow{x_i^n}))\}.$ Moreover, the whole intention set can be represented by $(\overleftarrow{I},\overrightarrow{I})=\{(\overleftarrow{I_1},\overrightarrow{I_1}),(\overleftarrow{I_2},\overrightarrow{I_2}),\mathrm{...},(\overleftarrow{I_m},\overrightarrow{I_m})\}.$

According to the definition of intention, we know that if the agent learns according to the specific strategies and eliminates the possibility of backtracking, the intention database will be waiting for new intentions. However, if backtracking occurs, the intention database will be modified. The structure and updating algorithm for the intention database are shown in Tab. 7.2 and Algorithm 7.3.

A (Action) saves the information of all actions in every step of the learning process. If we set the agent to move into the i th step of the learning process, the current optional action sequence could be represented by $A_i=\{A_i^1,A_i^2,\mathrm{...},A_i^m\},$ where the number of current optional actions is m. When the agent moves into the n th step of the learning process, the set of optional actions can be defined by A = {A1, A2, ..., An}. The structure and updating algorithm of the action database are shown in Tab. 7.3 and Algorithm 7.4.

Tab. 7.3: Structure of action database.

S (State) is the set of current states of the agent. $({\overleftarrow{S}}_i,\overrightarrow{S_i})$ represents the state in the i th learning step. $(\overleftarrow{S},\overrightarrow{S})$ represents the set of states that have happened after n learning steps, or $(\overleftarrow{S},\overrightarrow{S})=\{(\overleftarrow{S_1},\overrightarrow{S_1}),(\overleftarrow{S_2},\overrightarrow{S_2}),\mathrm{...},(\overleftarrow{S_1},\overrightarrow{S_n})\}.$ The structure and updating algorithm of the state database are shown in Tab. 7.4 and Algorithm 7.5.

Sty (Strategy) records the learning path defined by the learning rules. These learning paths are recorded during the process of learning, and this set is actually a subset of the action set.

Tab. 7.4: Structure of state database.

Tab. 7.5: Action database of Example 7.1.

Tab. 7.6: Strategy database of Example 7.1.

Example 7.1 After the agent has been through three steps of the learning process, the action database is as shown in Tab. 7.5

Based on the target, capability database, and intention database of the agent, fused with the rewards and punishments provided by the ambient environment, we choose the path Left → Down → Right: A1 = A11 = Left, A2 = A12 = Down, A3 = A23 = Right. Therefore, we obtain the strategy database shown in Tab. 7.6.

In the learning process, we find that the current strategy database is not optimal for deeper study. Thus, we trace back to the last step, delete the outdated strategies, and insert the new strategies.

Theorem 7.1 The current strategy is $St{y}_i=\{A_1^{i_1},A_2^{i_2},\mathrm{...},A_n^{i_n}\}.$ During the learning process, a better strategy $St{y}_i=\{A_1^{j_1},A_2^{j_2},\mathrm{...},A_n^{j_n}\}$ has been found. Deleting Styi will not affect the optimal learning process of the agent.

Proof: The choice of strategy during the learning process follows certain rules. As the superiority of the new strategy has been demonstrated, the outdated strategy Styi, is obviously not the best, and deleting it will not cause an error.

R (Reward): After the agent takes actions on the environment, the environment will produce a value, named the reward. If the environment returns a positive value when the agent takes an action, the possibility of the action happened in the next time will increase, and vice versa. By the definition of the reward value, each action corresponds to a value. Thus, the reward value in the i th learning step of the agent is $(\overleftarrow{R_i},\overrightarrow{R_i})=\{(\overleftarrow{R_i^1},\overrightarrow{R_i^1}),(\overleftarrow{R_i^2},\overrightarrow{R_i^2})),\mathrm{...},(\overleftarrow{R_i^m},\overrightarrow{R_i^m}))\}.$ When the learning process moves into the n th step, the set of reward values is defined by $(\overleftarrow{R},\overrightarrow{R})=$$\{(\overleftarrow{R_1},\overrightarrow{R_1}),(\overleftarrow{R_2},\overrightarrow{R_2}),\mathrm{...},(\overleftarrow{R_n},\overrightarrow{R_n})\}.$

Tab. 7.7: Structure of reward database.

The structure and updating algorithm of the reward database are shown in Tab. 7.7 and Algorithm 7.6.

Brf (Belief Revision Function): As the environment changes, the state of the agent also changes. This kind change results in the beliefs of the agent being constantly modified, and the belief revision function is $Brf:(\overleftarrow{b_i},\overrightarrow{b_i})\times (\overleftarrow{s_i},\overrightarrow{s_i})\to (\overleftarrow{b_{i+1}},\overrightarrow{b_{i+1}}).$

Acf (Action Choice Function): Each action performed by the agent depends on the agent’s belief, capacity, intention, and its own external environment. Thus, the action choice function is $Acf:(\overleftarrow{b_i},\overrightarrow{b_i})\times (\overleftarrow{c_i},\overrightarrow{c_i})\times (\overleftarrow{i_i},\overrightarrow{i_i})\times (\overleftarrow{s_i},\overrightarrow{s_i})\to a_i.$

Irf (Intention Revision Function): As the environment changes, the state of the agent also constantly changes. This results in the intentions of the agent being constantly modified, and the intention revision function is $Irf:(\overleftarrow{i_i},\overrightarrow{i_i})\times (\overleftarrow{s_i},\overrightarrow{s_i})\to$$(\overleftarrow{i_{i+1}},\overrightarrow{i_{i+1}}).$

Stf (State Transition Function): This function maps the Cartesian product between action sets A and B to the state set. Thus, the state transition function is $Stf:a_i\times (\overleftarrow{s_i},$$\overrightarrow{s_i})\to ({\overleftarrow{s}}_{i+1},{\overrightarrow{s}}_{i+1}).$

7.2.2Related axioms

There are some relationships between the mental state of each agent that we call the mutual generation and restriction principle. The mutual generation between mental states refers to the ability of mental states to promote one another, and the mutual restriction refers to the ability of mental states to restrict one another.

a) Belief:

Axiom 7.1 $B(P_i,\overline{(\overleftarrow{x_i},\overrightarrow{x_i}}))=\overline{B(P_i,(\overleftarrow{x_i},\overrightarrow{x_i}))}$

Axiom 7.2 $B(P_i,(\overleftarrow{x_{i1}},\overrightarrow{x_{i1}}))\Lambda B(P_i,(\overleftarrow{x_{i2}},\overrightarrow{x_{i2}}))=B(P_i,(\overleftarrow{x_{i1}},\overrightarrow{x_{i1}})\Lambda (\overleftarrow{x_{i2}},\overrightarrow{x_{i2}}))$

Axiom 7.3 $B(P_i,(\overleftarrow{x_{i1}},\overrightarrow{x_{i1}}))\vee B(P_i,(\overleftarrow{x_{i2}},\overrightarrow{x_{i2}}))=B(P_i,(\overleftarrow{x_{i1}},\overrightarrow{x_{i1}})\vee (\overleftarrow{x_{i2}},\overrightarrow{x_{i2}}))$

b) Capacity:

Axiom 7.4 $C(M_i,\overline{(\overleftarrow{x_i},\overrightarrow{x_i}}))=\overline{C(M_i,(\overleftarrow{x_i},\overrightarrow{x_i}))}$

Axiom 7.5 $C(M_i,(\overleftarrow{x_{i1}},\overrightarrow{x_{i1}}))\Lambda C(M_i,(\overleftarrow{x_{i2}},\overrightarrow{x_{i2}}))=C(M_i,(\overleftarrow{x_{i1}},\overrightarrow{x_{i1}})\Lambda (\overleftarrow{x_{i2}},\overrightarrow{x_{i2}}))$

Axiom 7.6 $C(M_i,(\overleftarrow{x_{i1}},\overrightarrow{x_{i1}}))\vee C(M_i,(\overleftarrow{x_{i2}},\overrightarrow{x_{i2}}))=C(M_i,(\overleftarrow{x_{i1}},\overrightarrow{x_{i1}})\vee (\overleftarrow{x_{i2}},\overrightarrow{x_{i2}}))$

c) Intention:

Axiom 7.7 $I_i(st{y}_j,\overline{(\overleftarrow{x_i^j},\overrightarrow{x_i^j})}=\overline{I_i(st{y}_j,(\overleftarrow{x_i^j},\overrightarrow{x_i^j}))}$

Axiom 7.8 $I_i(st{y}_j,(\overleftarrow{x_i^{j1}},\overrightarrow{x_i^{j1}})\wedge I_i(st{y}_j,(\overleftarrow{x_i^{j2}},\overrightarrow{x_i^{j2}}))=I_i(st{y}_j,(\overleftarrow{x_i^{j1}},\overrightarrow{x_i^{j1}})\wedge (\overleftarrow{x_i^{j2}},\overrightarrow{x_i^{j2}}))$

Axiom 7.9 $I_i(st{y}_j,(\overleftarrow{x_i^{j1}},\overrightarrow{x_i^{j1}})\vee I_i(st{y}_j,(\overleftarrow{x_i^{j2}},\overrightarrow{x_i^{j2}}))=I_i(st{y}_j,(\overleftarrow{x_i^{j1}},\overrightarrow{x_i^{j1}})\vee (\overleftarrow{x_i^{j2}},\overrightarrow{x_i^{j2}}))$

The mental states of the agent are not independent but are correlated and mutually restrictive. From the two mapping relations $Acf:(\overleftarrow{b_i},\overrightarrow{b_i})\times (\overleftarrow{c_i},\overrightarrow{c_i})\times (\overleftarrow{i_i},\overrightarrow{i_i})\times$$(\overleftarrow{s_i},\overrightarrow{s_i})\to a_i\text{and}Stf:a_i\times (\overleftarrow{s_i},\overrightarrow{s_i})\to (\overleftarrow{s_{1+i}},\overrightarrow{s_{i+i}}),$ we can see that there are some relationships between action ai and state $\begin{array}{l}(\overleftarrow{s_i},\overrightarrow{s_i}).\\ \end{array}$ If the action makes the state move toward the target state, the relationship between them is one of facilitation. Conversely, the relationship may be restrictive. Obviously, under a certain state $\begin{array}{l}(\overleftarrow{s_i},\overrightarrow{s_i}),\\ \end{array}$ the action of the agent is determined by belief $(\overleftarrow{b_i},\overrightarrow{b_i}),$ capacity $(\overleftarrow{c_i},\overrightarrow{c_i}),$ and intention $(\overleftarrow{i_i},\overrightarrow{i_i}).$ Consequently, there are certain relationships between each kind of attribute.

7.2.3Working mechanism

Based on the above model, we now introduce the main working mechanism. Based on DFL, the agent can be divided into a responding layer, learning layer, and thinking layer. The responding layer mainly realizes the information exchange between the agent and the external environment. The learning layer handles the learning process of the agent. The thinking layer includes two parts, one to design the overall learning process of the agent and another to consider collaboration with other agents. The concrete working mechanism is shown in Fig. 7.1.

In a constantly changing environment, the agent is an entity with the following basic working mechanism. The agent perceives changes in the environment and transmits this information to the perception layer.

The perception layer conducts new perceptual behaviour based on belief and the result from the previous behaviour layer and transmits the new perceptual result to the current behaviour layer. Then, it updates the belief database.

The action taken by the behaviour layer is dependent on the current capacity, state, and intention of the agent and the result of the learning layer. When the behaviour layer takes a new action, it transmits the result of the new action to the learning layer and the perception layer. Then, it updates the corresponding capacity database, state database, and intention database.

The learning layer conducts new learning steps based on the new action transmitted from the behaviour layer and the new management plans transmitted from the management layer.

The management layer makes plans for the next learning step based on the result from the learning layer, cooperation information from the cooperation layer, and the strategies in the strategy database. If the management layer has made a new plan, it will transmit the new plan to the learning layer and cooperation layer at the same time and update the corresponding strategy database.

The cooperation layer decides how to collaborate with other agents based on the plan from the management layer and the current intention from the intention database and updates the intention database.

7.3.1Learning task

The learning task of the agent is defined as follows: when learning a strategy π : S → A, the agent will select the state for the next action based on the reward value r.

Definition 7.2 Starting with any state si, Vπ(si) is acquired with any strategy π as

$V^{\pi }\left(s_i\right)=r_i+\gamma r_{i+1}+{\gamma }^2{r}_{i+2}+\cdots ={\displaystyle \sum _{k=0}^{\infty }{y}^k{r}_{l+k}},\left(7.1\right)$

where γ is a constant in the range {0, 1}. If γ = 0, we obtain an immediate return from the agent. If γ = 1, we will obtain the result that the future return is more important than the immediate return. This section starts by considering the simplest case of an immediate return. Thus, we first present the immediate return single-agent learning algorithm.

7.3.2Immediate return single-agent learning algorithm based on DFL

Definition 7.3 $(\overleftarrow{S},\overrightarrow{S})$ is the set of state $({\overleftarrow{S}}_i,\overrightarrow{S_i})$ of the agent. A is the action set of the agent, named Ai, and $(\overleftarrow{R},\overrightarrow{R})$ is the reward set of the agent, named $(\overleftarrow{R_i},\overrightarrow{R_i}).$ At a particular time, if the agent is in a certain state $({\overleftarrow{S}}_t,\overrightarrow{S_t})$ and the actions that can be taken by the agent are represented by $a_t=\{a_t^1,a_t^2,a_t^m\},m\ge 1,$ the learning process of the agent will be defined as follows: