We need to take into account the order in which we put the patrol points in the array, so that they work as intended. Also, we could work on this a little bit by creating a function for deciding what the next patrol point will be. This will come in handy if we decide to keep improving the patrolling behavior according to our game's requirements:

private int CalculateNextPoint()
{
  return (currentPoint + 1) % patrolPoints.Length;
}

private void Update()
{
  if (!agent.pathPending && agent.remainingDistance < pointDistance)
    GoToPoint(CalculateNextPoint());
}