When multiplying matrices, the order in which we multiply them is relevant. Say, for instance, we have two 4x4 matrices. Let R be the first matrix and let's assume that this matrix encodes pure rotation; let T be the second matrix and let's assume that T encodes pure translation. Now:

In other words, the order of the operations affects the result. It is not the same to rotate around the origin and then translate away from it (orbiting camera), as compared to translating the origin and then rotating around it (tracking camera)! Your success depends on understanding this critical difference.

In order to set the location of the camera as the center for rotations, we need to invert the order in which operations are called. This is equivalent to converting from an orbiting camera to a tracking camera.