Getting assistance 

The last part of the functionality we need to add is giving the user the ability to get assistance finding an item. For this, we will be using--voice. In this example, we will be using a prebuilt class bundled with the starter project, but will go into the details later on in this book, when we explore different modes of interaction. The class we will be making use of is the appropriately named SpeechManager. This class provides a very minimalist implementation for speech recognition; for each tag we identify, we add to SpeechManager via the asynchronous method AddTagsAsync, which will create phrases using the prefixes:

  • find tag
  • where is tag
  • locate tag

When a phrase is recognized, SpeechManager will broadcast the OnPhraseRecognized event, passing the status, text, and tag. We will only process events that have a status equal to 1, indicating that the confidence is medium or higher on the recognized phrase. We then split the tag, delimited by the character "_", where the first part indicates the intent and the second the tag (or item) we want to find. 

Let's jump back into the AssistantItemFinderMain class to hook this all up. As we have done previously, start by adding the variable to the holder reference to an instance of SpeechManager

private SpeechManager speechManager;

Next, initialise it in the InitServices method and handle stopping it in the Dispose method. Amend each of these methods with the following code:

async void InitServices()
{
// ...

speechManager = await SpeechManager.CreateAndStartAsync();
speechManager.OnPhraseRecognized += SpeechManager_OnPhraseRecognized;
}

In the preceding snippet, we are creating a new instance of it and, once instantiated, assigning a delegate to the OnPhraseRecognized event:

public void Dispose()
{
// ...

if(speechManager != null)
{
speechManager.Stop();
speechManager = null;
}
}

There is only one more method left: SpeechManager_OnPhraseRecognized to handle events from SpeechManager. As mentioned in the preceding code snippet, we ignore events with a status less than one, and split the tag parameter to determine the intent and item to search for. Let's add this method to the AssistantItemFinderMain class:

After some experimentation, it was discovered that Microsoft's Cognitive Vision API worked satisfactory with large salient objects in the scene (such as a bike) but wasn't picking up smaller objects (such as keys). To prototype the concept, an additional intent was added and can be used manually to place items in the environment. The following is a list of phrases you can use to manually place items: remember keys, remember wallet, remember hat, remember umbrella, remember coat, remember jacket, and remember hammer.
void SpeechManager_OnPhraseRecognized(int status, string text, string tag)
{
if (status < 0) return;

var tagSplit = tag.Split('_');
var intent = tagSplit[0];
var item = tagSplit[1];

if (intent.Equals("findLocation", StringComparison.OrdinalIgnoreCase))
{
requestedSightingTerm = item;
}
else{
var node = CurrentNode;
var position = NodePosition;
var forward = GazeForward;
var up = GazeUp;

node.AddSighting(new Sighting(text, position, forward, up, item));
}
}

Once we have extracted the intent and item from the tag parameter, we test to see if we are finding the item or adding it. If finding, then we simply set the item value to the variable requestedSightingTerm, where the path construction of the navigation path will be handled in the Update method. If we are adding an item, then we instantiate a new Sighting and add it to currentNode.

This concludes this example and it's time to take it for a test run. Turn ON your HoloLens, if OFF, and click on the green arrow just in front of the Remote Machine button in the toolbar, as we did before, to build and deploy to the device. Once deployed and running, walk around your environment and manually place some items to test the path construction:

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.147.89.85