Let's now turn our attention to the class that will be responsible for capturing the frames, most of which should look familiar to you (if you have read the previous chapter).  There is a bit of code, most of which is boilerplate, so let's build it up bit by bit. 

Create a new C# class in the Content folder, calling it FrameGrabber, and add the following namespaces:

using Windows.Graphics.Imaging;
using Windows.Media;
using System.IO;
using Windows.Storage.Streams;
using System.Numerics;
using Windows.Media.Capture.Frames;
using Windows.Media.Capture;

As we saw in the preceding code snippet, FrameGrabber is responsible for capturing frames from the color camera of the HoloLens and attaching metadata, such as the user's current position and pose. We will achieve this by passing a datasource to FrameGrabber, which can be called upon when creating a new frame. Create the interface for this datasource that will act as a contract between FrameGrabber and any object that wants to act as its data source, as in the following snippet:

public interface IFrameGrabberDataSource
{
Node CurrentNode { get; }
Vector3 NodePosition { get; }
Vector3 GazeForward { get; }
Vector3 GazeUp { get; }
}

The datasource will make available all the additional data we require for our frame (beyond the image itself). Next, let's declare all the variables and properties for our FrameGrabber class: 

IFrameGrabberDataSource datasource;

MediaCapture mediaCapture;
MediaFrameSource mediaFrameSource;
MediaFrameReader mediaFrameReader;

private bool _analyzingFrame = false;
public bool IsAnalyzingFrame
{
get{
lock (this){ return _analyzingFrame; }
}
set{
lock (this){ _analyzingFrame = value; }
}
}

private bool _isNewFrameAvailable = false;
public bool IsNewFrameAvailable
{
get{
lock (this){ return _isNewFrameAvailable; }
}
set{
lock (this){ _isNewFrameAvailable = value; }
}
}

private bool _currentFrame;
public bool CurrentFrame
{
get{
lock (this){ IsNewFrameAvailable = false; return _currentFrame; }
}
set{
lock (this){ IsNewFrameAvailable = true; _currentFrame = value; }
}
}

Most should look familiar from the last chapter; broadly, our variables can be grouped into those related specifically to the media (mediaCapture, mediaFrameSource, and so on) and those related to managing the process pipeline (IsNewFrameAvailable, CurrentFrame, and so on). Next, we will implement the construction for FrameGrabber. Add the following constructor to the FrameGrabber class: 

private FrameGrabber(IFrameGrabberDataSource datasource = null, MediaCapture mediaCapture = null, MediaFrameSource mediaFrameSource = null, MediaFrameReader mediaFrameReader = null)
{
this.datasource = datasource;
this.mediaCapture = mediaCapture;
this.mediaFrameSource = mediaFrameSource;
this.mediaFrameReader = mediaFrameReader;

if (this.mediaFrameReader != null)
{
this.mediaFrameReader.FrameArrived += MediaFrameReader_FrameArrived;
}
}

As we have done in the previous chapter, our construction is private, requiring FrameGrabber to be instantiated indirectly via a Factory method. The purpose of this Factory method is to hide some of the details required to construct the associated media classes, which we will see next. Add the following static method to the FrameGrabber class:

public static async Task<FrameGrabber> CreateAsync(IFrameGrabberDataSource datasource)
{
MediaCapture mediaCapture = null;
MediaFrameReader mediaFrameReader = null;

MediaFrameSourceGroup selectedGroup = null;
MediaFrameSourceInfo selectedSourceInfo = null;

var groups = await MediaFrameSourceGroup.FindAllAsync();
foreach (MediaFrameSourceGroup sourceGroup in groups)
{
foreach (MediaFrameSourceInfo sourceInfo in sourceGroup.SourceInfos)
{
if (sourceInfo.SourceKind == MediaFrameSourceKind.Color)
{
selectedSourceInfo = sourceInfo;
break; 
}
}

if (selectedSourceInfo != null)
{
selectedGroup = sourceGroup;
break;
}
}

if (selectedGroup == null || selectedSourceInfo == null)
{
return new FrameGrabber();
}

var settings = new MediaCaptureInitializationSettings
{
SourceGroup = selectedGroup,
SharingMode = MediaCaptureSharingMode.SharedReadOnly,
StreamingCaptureMode = StreamingCaptureMode.Video,
MemoryPreference = MediaCaptureMemoryPreference.Cpu,
}

mediaCapture = new MediaCapture();

try{
await mediaCapture.InitializeAsync(settings);
} 
catch(Exception e){
return new FrameGrabber();
}

MediaFrameSource selectedSource = mediaCapture.FrameSources[selectedSourceInfo.Id];
mediaFrameReader = await mediaCapture.CreateFrameReaderAsync(selectedSource);

MediaFrameReaderStartStatus status = await mediaFrameReader.StartAsync();

if (status == MediaFrameReaderStartStatus.Success)
{
return new FrameGrabber(datasource, mediaCapture, selectedSource, mediaFrameReader);
}
else{
return new FrameGrabber();
}
}

The details of this method are described in the previous chapter, so we don't delve into them here. At a high level, we search for a suitable media source and, once found, create and initialize the MediaCapture instance and, then, create MediaFrameReader, which can be used to intercept the captured frames. 

We will next implement the delegate responsible for handling frames received from MediaFrameReader. Add the following method to the FrameGrabber class:

void MediaFrameReader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
{
if (!IsAnalyzingFrame && !IsNewFrameAvailable)
{
MediaFrameReference frame = sender.TryAcquireLatestFrame();

if (frame != null)
{
new Task(() => SetFrame(frame)).Start();
}
}
}

Before processing the received frame, we check whether we are not currently analyzing a frame and whether the previous frame we processed has been consumed (!IsAnalyzingFrame && !IsNewFrameAvailable). If these conditions are satisfied, then we proceed with trying to acquire the last frame of MediaFrameReader and, if successful, passing it to the SetFrame method on a separate thread. The SetFrame method makes up the bulk of this classes' logic. It is responsible for getting the raw data from MediaFrameReference and pulling through the user's current position and orientation to create a new Frame instance. Let's implement this method now. Add the following to the FrameGrabber class:

async void SetFrame(MediaFrameReference frame)
{
IsAnalyzingFrame = true; 

var node = datasource.CurrentNode;
var position = datasource.NodePosition; 
var forward = datasource.GazeForward;
var up = datasource.GazeUp;

var timestamp = Utils.GetCurrentUnixTimestampMillis();

byte[] frameData = null; 
try{ frameData = await GetFrameData(frame); } catch{} 
if(frameData == null)
{
IsAnalyzingFrame = false;
return; 
}

CurrentFrame = new Frame
{
frameData = frameData,
node = node,
position = position, 
forward = forward,
up = up,
timestamp = timestamp
}
IsAnalyzingFrame = false;
}

As mentioned before, Frame consists of the raw data of the captured frame and the current state of the user in terms of their spatial information, which we obtain via assigned datasource. After obtaining the necessary parts, we will create a new Frame instance and assign it to the CurrentFrame property, which will flag IsNewFrameAvailable. 

One final method to implement is, GetFrameData. This method takes MediaFrameReference and returns its video data as a byte array. Let's implement this method now; because there is a lot going on in this method, we will deliver in chunks to make it more consumable. Start by adding the following method to the FrameGrabber class:

async Task<byte[]> GetFrameData(MediaFrameReference frame)
{
byte[] bytes = null; 

VideoMediaFrame videoMediaFrame = frame.VideoMediaFrame;

VideoFrame videoFrame = videoMediaFrame.GetVideoFrame();
SoftwareBitmap softwareBitmap = videoFrame.SoftwareBitmap;

SoftwareBitmap bitmapBGRA8 = SoftwareBitmap.Convert(softwareBitmap, BitmapPixelFormat.Bgra8, BitmapAlphaMode.Ignore);

}

In the preceding snippet, we obtained the raw bitmap of the video frame associated with the media frame. We then convert this into a pixel format of BGRA channels with 8 bytes (Bgra8) for each. We next create a JPEG encoder and write it into an in-memory stream before reading it back out to a byte array and then, finally, returning it; amend the GetFrameData method with the following code:

using (InMemoryRandomAccessStream stream = new InMemoryRandomAccessStream())
{
BitmapEncoder encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.JpegEncoderId, stream);
encoder.SetSoftwareBitmap(bitmapBGRA8);
encoder.IsThumbnailGenerated = false;

try{
await encoder.FlushAsync();
bytes = new byte[stream.Size];
await stream.AsStream().ReadAsync(bytes, 0, bytes.Length);
} 
catch{}
}

return bytes; 

This now concludes our FrameGrabber class. Let's put it to use. In the next section, we will return to our AssistantItemFinderMain class and start capturing frames using FrameGrabber.