- When the gradients of all tasks are in the same direction, then it is called gradient agreement, and when the gradient of some tasks differ greatly from others, then it is called gradient disagreement.
- The update equation in gradient agreement can be expressed as
. - Weights are proportional to the inner product of the gradients of a task and the average of gradients of all of the tasks in the sampled batch of tasks.
-
The weights are calculated as follows:
- The normalization factor is proportional to the inner product of
and 
. - If the gradient of a task is in the same direction as the average gradient of all tasks in a sampled batch of tasks, then we can increase its weights so that it'll contribute more when updating our model parameter. Similarly, if the gradient of a task is in the direction that's greatly different from the average gradient of all tasks in a sampled batch of tasks, then we can decrease its weights so that it'll contribute less when updating our model parameter.
Chapter 8: Gradient Agreement as an Optimization Objective
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.