Installing Mosquitto

On the command line, you already have two commands available to you: one to publish messages on given topics and one to subscribe to messages on given topics.

Comparison of Other MQTT Brokers

For your reference, you can find a comparison of a few other MQTT brokers here: https://github.com/mqtt/mqtt.github.io/wiki/server-support.

RabbitMQ is a strong open source contender, but while clustering support is definitely robust, the setup is not quite easy.

MQTT Messages on the Command Line

Usually, my favorite graphical way to see messages is to use MQTT Explorer, a graphical client to MQTT, as shown in Figure 5-2.

It should also be available through your favorite package manager.

Let’s get back to sending the full equivalent of an intent message. You can use the command line to send a JSON file directly to the target intent queue, which as we have just seen is as follows: hermes/intent/hellonico:highlight.

where onlycats.json is the JSON file with the content we have just seen in Listing 5-1.

If you still have MQTT Explorer running, you will see the message in the corresponding queue, as shown in Figure 5-3.

We have seen how to interact, publish, and subscribe to messages with MQTT from the command line. Let’s do the same using Java now.

Dependencies Setup

Either using the project from previous chapters or creating a new one based on the project template, we are going to add some Java libraries to interact with the MQTT messaging queue and to parse JSON content.

Sending a Basic MQTT Message

Since I’m writing part of this book on a plane flying over Russia, we will send a quick “hello” message to our fellow Russian spies using the MQTT protocol.

To check that things are in place, verify that the message is showing in your running MQTT Explorer instance, as shown in Figure 5-4.

Note that you should of course be listening to the hello topic first.

Simulating a Rhasspy Message

Our never-ending love of cats is again showing in Figure 5-5 in the MQTT Explorer window.

JSON Fun

Before being able to handle Rhasspy messages from the MQTT topic, we need to be able to do a bit of JSON parsing.

Obviously, there are many ways to do parsing in Java; we can parse manually (hint, don’t do this) or use different libraries. Here we are going to use the org.json library because it’s pretty fast at handling incoming MQTT messages.

Listening to MQTT Basic Messages

If you’re running everything from your Raspberry Pi, then the following Visual Studio Code setup will do everything properly and display the Russian message, as shown in Figure 5-6.

Listening to MQTT JSON Messages

Now that you know everything about keys and brokers, it is time to get and install Rhasspy.

Preparing the Speaker

You do need a working microphone setup before executing any of the voice commands.

Standard USB microphones should be plug-and-play, and I have been using conference room microphones with pretty good results.

You ReSpeaker/Raspberry Pi setup should look like the one in Figure 5-7.

Installing Docker

Installing Rhasspy with Docker

The whole Rhasspy suite can be started by running the prepared container, and this is done by executing the following command on the Raspberry Pi, provided you have installed Docker properly.

The Rhasspy server and its console are ready, so let’s go back to the main computer and access the Raspberry Pi via the browser-based IDE.

Starting the Rhasspy Console

To access the console, go to http://192.168.1.104:12101/, where 192.168.1.104 is the IP address of the Raspberry Pi.

You’re then presented with the lively screen shown in Figure 5-8.

The console makes it quite obvious for you on first start that it should go and fetch some remote files, so provided your Raspberry Pi is connected to the Internet, just click the Download Now button.

The operation takes a few minutes to complete, so don’t lose your patience here. Wait for the dialog box shown in Figure 5-9.

The console refreshes by itself, and we’re now being greeted with Figure 5-10.

But we seem to still have some markers telling us something is missing, as shown in Figure 5-11 and Figure 5-12.

Fortunately, and for now, clicking the top-right green Train button makes our console shiny and free of red markers, as in Figure 5-13. We are now ready to use the Rhasspy console.

The Rhasspy Console

Let’s take a look at the Rhasspy menu in more detail. Figure 5-14 shows the different sections for configuring the home assistant.

We know how to navigate the console now, so let’s make our vocal assistant recognize something for us.

First Voice Command

To create a voice command that generates an intent, we need to make some edits in the Sentences section. As just explained, the Sentences section is where we define the sentences our engine can recognize.

First Command, Full Sentence

For once, the grammar used to define intents is quite simple and is inserted via an .ini file (which comes from the good old Windows 95 days).

The pattern is as follows:

[IntentName]

Sentence1

So, if it is Monday morning and you are really in need of coffee, you could write something like this:

[Coffee]

I need coffee

Let’s try it. Let’s start by deleting the content of that .init file and adding just the previous coffee intent, so the Sentences section now looks like Figure 5-15.

If you click the nicely visible Save Sentences button, you’ll get the training reminder shown in Figure 5-16.

Speech Section and Trying Your Intent

The Rhasspy engine is now trained, and you can try it by heading to the Speech section, shown again in Figure 5-17.

The Speech section is where you can either speak or write a sentence and see whether and what intent is recognized.

Supposed we type “I need coffee” in the Sentence field; the engine should recognize and generate the “Coffee” response that we just defined, as in Figure 5-18.

The voice command “Coffee” is recognized, and an associated generated JSON document is attached to it, with various inputs on confidence and raw values.

Since we’re focusing on voice commands here, you can also just tap and hold the “Hold to record” button, say “I need coffee,” and see the spoken sentence being turned into text first, then displayed in the Sentence field, and then analyzed as was just done.

This is probably the time to plug in the ReSpeaker or your USB microphone.

If you have installed eSpeak (http://espeak.sourceforge.net/), which is a text-to-speech command-line tool, you can also generate a WAV file and feed it to Rhasspy.

espeak "I need coffee" --stdout >> ineedcoffee.wav

Upload that ineedcoffee.wav file and click the Get Intent button.

Fine-Tuned Intents

It is of course possible to just pop in optional words, variables, and placeholders in the commands defined in the Sentences section. Let’s improve our game here and add details to our intent so as to create better voice commands.

Optional Words

It is of course possible to make words optional in the sentence to make the commands more natural to users.

In the previous coffee example, you could have an optional urgency embedded into the intent, as in you may really need coffee at times. This is done by putting the optional words inside square brackets, as shown here:

[Coffee]

I [really] need coffee

After training, you see in Figure 5-19 and Figure 5-20 how the same Coffee intent is being properly recognized by Rhasspy.

Adding Alternatives

But what about when it’s a day where you feel really good about yourself and you don’t really need that addicting black stuff to wake you up? Let’s rewrite our coffee intent so it can recognize both when you need coffee and when you don’t.

[Coffee]

I (need | don't need) coffee

Training and using this new intent, you can now tell your coffee assistant that you do not need coffee. Back on the Speech tab, you would get the intent shown in Figure 5-21.

You probably already noticed one problem with this generated intent. Yes, it’s not very useful as is, because the same intent is generated whether we want coffee or not.

We could of course parse the received string or write two different intents, but what if we simply give a name to that list of possibilities? This is done with round brackets, as shown here:

[Coffee]

I ((need | don't need) {need}) coffee

Now, our Coffee intent will add a slot named “need” to the JSON document, and its value will be either “need” or “don’t need.”

Let’s try again; see the result in Figure 5-22.

You should notice several things here. First, the summary of the intent is now showing more details, the intent is marked in red, and each slot is nicely shown in a list with its associated value (here, need = don’t need).

Second, the full JSON document of the intent contains an entities section, and each entity has an entity name and associated value.

As a side note, the JSON document also contains a slots section, with a simple list of all the slots of the intent.

What’s next?

Making Intents with Slots More Readable

You probably noticed that if we add one too many slots, then the whole sentence becomes messy quite quickly.

We can move the slot definition right below the intent name and assign possible values with =.

For example, the same coffee intent can be rewritten as shown here:

[Coffee]

need = ((need | don't need) {need})

I <need> coffee

Nothing has changed, and the intent behaves the same as before, but it is now vastly more readable.

That’s great, but what’s next?

Defining Reusable Slots

Yes, defining slots inline also brings extra complexity to our sentence definitions, so it is possible to move them out and define them on the Slots tab.

Slots are defined in a simple JSON file, and to move our coffee need to the Slots tab, you would write the JSON file shown here:

{

    "need": [

        "need",

        "don't need"

    ]

}

This shows up in the browser as in Figure 5-23.

To use that in the sentence file, you would now have to write the intent, as shown here:

[Coffee]

myneed = $need {need}

I <myneed> coffee

Here is a breakdown of the code:

• myneed: This is the placeholder in the sentence, meaning its definition can be seen as being inlined at training time.

• $need: The dollar sign specifies that this is coming from the slots JSON file.

• {need}: This is the name of the slot within that intent, meaning you can reuse the same list in different intents, each time with a different slot name.

Great. Now we have highly configurable intents, with a complex list of options on hand.

Let’s move to the section where the external system can make use of the generated JSON file when an intent is recognized.

Settings: Get That Intent in the Queue

The generated JSON file looks like it could be useful, but we want to have it external to Rhasspy and in the MQTT queue so that we can do more processing.

Let’s head to the Settings section of the console, as shown in Figure 5-24.

Let’s look at the MQTT interaction first. If you click the MQTT link, you’ll be taken to the corresponding section, where you can enter the details of your running MQTT daemon, as shown in Figure 5-25.

Make sure to fill in the hostname or the IP address of your Raspberry Pi here. Restart Rhasspy when asked, and that’s it. Now let’s try to get some coffee on MQTT.

Back on the Speech tab, we can now trigger a new coffee intent, as in Figure 5-26.

Let’s now take a look at the MQTT Explorer window of Figure 5-27.

What do you see? We see a message being sent to the hermes/intent/coffee topic, which is something similar to what you saw in the previous sections of this chapter.

Yes, you can feel it too—this is going to work.

You can also see that there is a simple MQTT message sent to Rhasspy/intent/coffee, with just the slot’s name and value in the message body. If you don’t need to play with the confidence and other details, this is the best alternative. Working with this message is left as an exercise for you.

You can get some instant gratification by trying a few messages specifying whether you want or don’t want coffee; just make sure to not drink too much of it.

Settings: Wake-Up Word

It’s nice to go back to speech every now and then, but it would be nice to just leave the Raspberry Pi by itself, and when saying a given word, or set of words, get Rhasspy to listen for intents.

This is done by using wake-up words , which are defined in the Settings section.

I have been having some luck with enabling the Snowboy kit (https://snowboy.kitt.ai/) in the configuration, as shown in Figure 5-28.

Now, when you say “Snowboy,” Rhasspy will wake up, as if you were holding the Record button.

We’re getting there, aren’t we?

Creating the Highlight Intent

Now, we have to create a small intent that we will use to tell our main object detection application which object we want to focus on.

You should definitely try this on your own before reading ahead…

to have the intent JSON be published in the hermes/intent/highlight queue, with the content as shown in Figure 5-29.

Voilà, we’re mostly done with the Rhasspy and voice command section, so we can now go back to detecting objects, cats, and people in real time.

Simple Setup: Origami + Voice

The intent used here is the highlight intent that was defined earlier, with a slot to recognize the names taken from the object categories of the COCO dataset.

If things are all good and you say “Show me only cats,” the annotate filter will update the upper-left text, as shown in Figure 5-30.

Now let’s connect this setup with object recognition and with the real-time video streaming part.

Origami Real-Time Video Analysis Setup

From the previous chapter, you will remember that we were using an origami filter, called Yolo, to perform object detection. Just like the annotate filter, the Yolo filter is included in the origami-filters library, and in a very Java way, you just have to extend the base filter to highlight the detected objects.

Available in the origami-filter library, the Yolo filter allows you to download and retrieve networks on demand using a network specification.So, for example, networks.yolo:yolov3-tiny:1.0.0 will download yolov3-tiny and cache for further usage, depending on which device you’re on. This is a feature of the origami framework.

Each one is either faster or more precise than the one before it.