In this section, we will implement a class that will be responsible for capturing frames from the HoloLens color camera and another class that will make use of Microsoft's Cognitive Vision API service to tag each of these frames to help us understand what the user sees. 

Let's start by creating a data object that will be used to store the details of each frame. This data object, Sighting, will be added to a collection of the Node class so that we can find the most appropriate node and construct a path when required. 

Within Visual Studio, create a new class called Sighting in the Content folder and make the following amendments:

internal class Sighting
{
public string Description { get; private set; }
public HashSet<string> Tokens = new HashSet<string>();

public Vector3 Position { get; private set; }
public Vector3 Forward { get; private set; }
public Vector3 Up { get; private set; }

public long Timestamp { get; private set; }

public Sighting(string description, Vector3 position, Vector3 forward, Vector3 up, params string[] tokens)
{
Description = description;

Position = position;
Forward = forward;
Up = up;

if (tokens != null)
{
foreach (var token in tokens)
{
 Tokens.Add(token);
}
}
Timestamp = Utils.GetCurrentUnixTimestampMillis();
}

public Sighting AddToken(string token)
{
Tokens.Add(token);
return this;
}
}

In addition, add the namespace System.Numerics. As mentioned earlier, the Sighting class simply stores metadata associated with each frame captured from the HoloLens color camera. As we saw before, Microsoft's Cognitive Vision API returns a list of tags and a caption for a given photo. We store these in the tokens collection and description along with the position and gaze direction relative to the node's anchors coordinate system. This allows us to determine the general direction of an item (tag) as well as avoid unnecessary work by analyzing frames that are relatively in the same position and direction. Let's now make some amendments to our Node class to be able to contain a list of sightings. Open up your Node class in Visual Studio and make the following amendments:

internal class Node
{
// ...

public List<Sighting> Sightings = new List<Sighting>();

public void AddSighting(Sighting sighting)
{
Sightings.Add(sighting);
}

// ...
}  

We will now move onto implementing FrameGrabber; as in the last chapter, this class will be responsible for intersecting frames from the HoloLen's color camera and making them available for analysis. The only significant difference here is what metadata we associate with it. In this example, we want to have each frame associated with the node and position and pose of the user. Using this metadata, we can create Sighting, as we saw before. Let's start by creating a class that will be used to store the details of a captured frame, including the required metadata. Within Visual Studio, create a new class called Frame in the Content folder and make the following amendments:

using System.Numerics;

internal class Frame
{
 public byte[] frameData;
 public Node node;
 public long timestamp;

 public Vector3 position;
 public Vector3 forward;
 public Vector3 up;

 public bool IsSimilar(Frame frame)
 {
  if (frame.node != node)
 {
  return false; 
 }
 var dot = Vector3.Dot(forward, frame.forward);
 return dot >= 0.5f;
 }
}

Like the Sighting class, its role is to store data; in this case, we are storing the captured frame as a byte array (frameData) along with the associated node and the user's spatial details. We have also added the method IsSimilar. As mentioned previously, we want to avoid analyzing frames that are too similar--here, what we define as similar are those frames captured that share the same node (frame.node != node) and face in the same direction.

We determine this by calculating the dot product, which gives us a scalar of how close the forward directions of both the frames are; if the dot product of the two frames forward vectors were 1, then they would be facing in exactly the same direction. If the dot product returned was 0, then the forward directions of the frames would be perpendicular to each other.