© Nicolas Modrzyk 2020
N. ModrzykReal-Time IoT Imaging with Deep Neural Networkshttps://doi.org/10.1007/978-1-4842-5722-7_3

3. Vision on Raspberry Pi 4

Nicolas Modrzyk1 
(1)
Tokyo, Tokyo, Japan
 

Imagine a world in which your entire possession is one raspberry, and you give it to your friend.

—Gerda Weissmann Klein, All But My Life (Hill and Wang, 1995)

In Chapters 1 and 2, you saw how to get up and running with computer vision using Java on a regular desktop computer. The setup was a bit manual, but all the different building blocks were freely available, so it was just a matter of putting those blocks together.

In this chapter, you’ll add one more block to the mix, the Raspberry Pi 4. The first goal of the chapter is to perform the same kinds of tasks that you did on a computer but run as much as possible on the IoT device, while typing code and controlling the code execution from your local desktop. In other words, you’ll write on your computer but execute on the Raspberry.

For the code execution part, you need to have some specific hardware and cabling, so you should prepare yourself to go shopping. We’ll provide a shopping list and then move on to the setup.

Specifically, we will go through creating the SD card and setting up the Raspberry Pi, but it is assumed that most of those steps are already properly covered in other books and blogs, so you can go and read those if you need more than the basics presented here.

At the end of this chapter, you will have good grasp of the Raspberry, its speed, and its energy consumption while running the different vision operation and detection algorithms. You will also be ready to integrate this functionality on a larger scale into a voice-controlled home assistant of your design and thus expand your personal home or work office automation.

Bringing the Raspberry to Life

“Man,” I cried, “how ignorant art thou in thy pride of wisdom!”

—Mary Shelley, Frankenstein (1818)

We’re no Doctor Frankenstein, but to bring the Pi to life, we will have to work as if we are designing one: plugging in the cables, assembling, and turning on the power of the little device. Some shopping will be needed, and we have to gather the physical hardware before doing anything useful. Luckily, most of the things to buy are cheap and can be reused from other people’s old computers or from your grandma’s attic. She won’t notice.

Once the shopping is done and the Raspberry Pi and all the essential parts are assembled, we will move to putting in place the same vision setup on Visual Studio Code as in Chapter 1 and learn how to program OpenCV in Java again.

Shopping

While the Raspberry Pi is a solid computer on its own, most of the time you will not be able to do anything with it without a few other extra pieces. In this chapter, you will need a few accessories.

Table 3-1 provides a shopping list, but you can decide what type and where to buy things.

Careful

There are a wide range of cables and power adapters these days. Be sure to buy a power adapter that gets around 4.8A to 5A output if you want to do overclocking. Your chosen USB cable should also be able to carry the electrical current, but these days most of them do.

Table 3-1

Shopping List

Name

What

Approximate Price (USD)

Required?

Always Needed?

 

Raspberry Pi 4 4GB

Main computer

$55

Yes

Yes

USB-C to USB-A cable

Connect the computer to power

$6

Yes

Yes

Power adapter with USB-A (at least 4.8A or even 5A for overclocking)

Electric power

$10

Yes

Yes

HDMI cable

Connect to screen

$7

Yes

No

SD card 128GB

Hard disk

$10

Yes

Yes

ReSpeaker microphone

Recommended microphone for snips (Chapter 3)

$25

No

If bought, yes

Display with HDMI port

Display for initial debugging

$25

Yes

No

USB webcam

Record video

$20

Yes

Yes

Keyboard and mouse

 

$20

Maybe

No

Total

 

$180

   
Once you have gathered all the parts (Figure 3-1), it’s time to get the software running on the Pi.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig1_HTML.jpg
Figure 3-1

Raspberry Pi 4, power adapter, webcam, screen, etc

Downloading the OS

The Raspberry Pi will refuse to even look at a beautiful cat like Marcel without an operating system. There are a few operating systems readily available on the Raspberry web site.

Here’s a short list of the main ones:
  • Ubuntu Mate

  • Ubuntu Core

  • Windows 10 IoT Core

  • Noobs

  • Raspbian (the official operating system)

Ubuntu Mate and Ubuntu Core are new and good alternatives, but we will focus on the official operating system, Raspbian, because most of the documentation available for the Raspberry is based on it. It’s also quite easy to get started with it.

Raspbian is a Debian-based operating system, so most Linux aficionados will be at ease with it and its included aptitude package manager. Even though packages have to be compiled specifically for the ARM-based CPU of the Raspberry, the package list is quite up-to-date and mostly in sync with the main Debian line.

To be able to use the operating system as a bootable image that the Raspberry can use, we first need to download the image file onto a local computer and create a bootable disk specifically for the Raspberry Pi.

To download the very big OS file (don’t try this while tethering on your phone), head to https://www.raspberrypi.org/downloads/ and look for the Raspbian icon shown in Figure 3-2.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig2_HTML.jpg
Figure 3-2

Raspbian icon

Click the icon. You will then land on the Raspbian-specific download page, as shown in Figure 3-3, where you can find the link to the zip file to retrieve.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig3_HTML.jpg
Figure 3-3

Torrent or zip file for Raspbian

We don’t need a full desktop install and all the proposed software, but we will aim for simplify so “Raspbian Buster with desktop” will be sufficient for us now.

Creating the Bootable SD Card

With the OS file on your computer, you are ready to create the bootable device. As noted on the Raspberry web site, the easiest way to get the OS on the Pi is by using Etcher, which is software to create the bootable SD card from a zip file or an .img file.

Etcher is available from https://www.balena.io/etcher/, and it can run on any platform, such as macOS, Windows, and Linux. The screenshots in this book were taken on Windows.

The first step after starting the application is to select the downloaded zip file, as shown in Figure 3-4.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig4_HTML.jpg
Figure 3-4

Selecting the image file

With the image file selected and the SD card inserted into an SD slot of your computer, let’s select the SD card you want to write the OS onto, as shown in Figure 3-5.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig5_HTML.jpg
Figure 3-5

Selecting the micro SD card

The card used here was specifically chosen because it’s so big, so you ignore the message shown about its size, as shown in Figure 3-6.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig6_HTML.jpg
Figure 3-6

The SD card is big

Finally, let’s go ahead and “flash” the card, in other words, write the files onto the card. The progress is shown and should take no more than two to three minutes, as shown in Figure 3-7.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig7_HTML.jpg
Figure 3-7

Flashing

In the last step, Etcher will check the integrity of the copied data, as shown in Figure 3-8.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig8_HTML.jpg
Figure 3-8

Validating

Once the flashing and the validation are done, Windows will try to use the drive as before, but the format of the SD card is now not readable without a specific driver, as shown in Figure 3-9. You can safely ignore the message and click Cancel.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig9_HTML.jpg
Figure 3-9

Don’t be scared…

The card is here, so let’s now plug in the cables of the Pi.

Connecting the Cables

Electricity can be dangerous. My nephew tried to stick a penny into a plug. Whoever said a penny doesn’t go far didn’t see him shoot across that floor. I told him he was grounded.

—Tim Allen

I have found plugging things together to be quite fun and relaxing and could actually be used in personal therapy for the tough world we live in.

Table 3-2 provides a nonexhaustive list of the connections that have to be made.
Table 3-2

What to Plug Where Checklist

From

To

Why

 

Power AC

Power adapter

Get some power

USB-A

Power adapter

Power to the Pi

Webcam

Pi USB port

Get some video stream

Mouse

Pi USB port

Use a pointer

Keyboard

Pi USB port

Use a keyboard

Screen HDMI

Pi Micro HDMI

Display

USB-C

Raspberry Pi

Power to the Pi

Note

Make sure to plug either the power adapter or the USB-C to Raspberry adapter in last; otherwise, the power will turn the board on without anything to do.

Things usually are easier to understand with a picture, so make sure to refer to Figure 3-10 to see where things fit together on the actual board.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig10_HTML.jpg
Figure 3-10

Original blueprints of the Raspberry Pi 4 and what to connect to

Finally, with the micro SD card inserted into the Raspberry Pi 4, you are ready to power things on (or have you already?).

First Boot

The first boot is quite fast. The boot process includes a step where the file system needs to be resized, which is now handled automatically by Raspbian on first boot. You should get a welcome screen like the one shown in Figure 3-11.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig11_HTML.jpg
Figure 3-11

First boot

With the keyboard, mouse, and screen plugged in, you can do quite a bit, but for most of the tasks we are going to do in this chapter, we want to work over SSH, which requires a network connection.

You can plug an Ethernet cable into the Ethernet port of the Pi, but WiFi works quite well. On the top right, click the WiFi icon and enter your WiFi settings in the dialog box, as shown in Figure 3-12.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig12_HTML.jpg
Figure 3-12

Setting up the WiFi

One last step is required before we can connect via SSH, which is to enable the Raspberry Pi–embedded OpenSSH server, which can be enabled via the following menu:
Preferences ➤ Raspberry Pi Configuration ➤ Interfaces
Click to enable the SSH server, as shown in Figure 3-13.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig13_HTML.jpg
Figure 3-13

Enabling the installed SSH server

Now let’s connect to the Raspberry via SSH.

Finding Your Raspberry Using nmap

As you probably know, to remotely connect to the Raspberry Pi, you need its IP address, which you can find from within the Raspberry Pi by opening a terminal and typing the following:
ip addr

This will display the IP address you need to use to connect to the Raspberry Pi.

If you’re out of luck and without a keyboard, a screen, a mouse, or all of these, you can revert to some simple commando countermeasures and use nmap to find all the Raspberry Pis on the network (and all the computers and Chinese smartphones for that matter).

This is done by running a command similar to the following one from the main computer:
sudo nmap -O -p 22 192.168.1.*

Here, 192.168.1.∗ is the start of your own desktop IP address and where the Raspberry would be. Note that for this to work, we’re assuming that your computer and the Raspberry are on the same network.

What does the command actually do? In short, nmap simply scans the local network for available machines and open ports.

The -p switch scans for a given port, and the -O switch says to gather as much information as possible about each found host.

Running the previous nmap command on the local network gives us something like Figure 3-14.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig14_HTML.jpg
Figure 3-14

Looking for your Raspberry Pi

A Raspberry (hopefully yours!) has been found on the network. You can now SSH through it using the default username/password, which should still be pi / raspberry, with the following command:
and then get connected, as shown in Figure 3-15.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig15_HTML.jpg
Figure 3-15

Starting an SSH connection

We made one connection, but we’re going to have to repeat those connection steps over and over, so let’s save a few SSH settings for later by modifying a configuration file both locally and on the remote Raspberry.

Setting Up SSH Easily

Usually when connecting via SSH , you disable password authentication, especially the default authentication of the Raspberry Pi, and set up a key from your local machine to connect to the remote machine and then set up that remote machine to accept only one key, or a limited number of them, so as to prevent random security attacks.

This works like this:
  1. 1.

    Create your personal key pair to use with SSH. One private key stays on your computer, and one public key is added as a known entity on the remote machine. Until quantum computing comes in, that key pair is said to be secure, and it’s close to impossible to guess one’s private key by just knowing the public key. For more information on this, read Joshua Davies’ article at https://commandlinefanatic.com/cgi-bin/showarticle.cgi?article=art054.

     
  2. 2.

    Register the public key that got generated on the remote device, here the Raspberry Pi. Go ahead and give the public key to as many people as you want; by default, this is the file ending with the .pub extension.

     
  3. 3.

    Add some configuration for SSH to use that generated private key when connecting to the Raspberry through a shortcut. That private key, though, is yours, so make sure you don’t give it to anyone!

    On Amazon Web Services (AWS), you usually get a .pem file to connect through a new running instance. That .pem file is automatically registered for you on the new VM, so only you can access that newly created VM, thus allowing a secure access to your in-the-cloud machines.

     
To create your key on your own local machine, you use ssh-keygen with the -t switch to specify the algorithm and the -b switch to specify the number of bits (nowadays, 4,096 is considered very secure).
% ssh-keygen -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (~/.ssh/id_rsa):

Once the command has run successfully, two files are generated: an id_rsa.pub file and an id_rsa file. pub is your public key that you can give to anyone, and id_rsa is your private key and should be handled with the utmost care.

To tell the Raspberry Pi about your public key and recognize you, you add it as a one-liner to the /home/pi/.ssh/authorized_keys file, creating the file and the .ssh parent folder if needed.

Once this is done, on your computer, it’s time to add details of the SSH connection settings to the $HOME/.ssh/config file, again creating the file if you don’t have it.
Host pi4
      User pi
      IdentityFile ~/.ssh/id_rsa
      HostName 192.168.1.17
      ForwardX11 yes
      ForwardX11Trusted yes
Here is what the connection settings mean :
  • pi4 is the shortcut that we will use to connect to the Raspberry from now on.

  • User is the default user on the Raspberry Pi and usually is pi.

  • IdentifyFile is the path to the private key of your computer that will be used to authenticate you; this is the file you created a few moments ago.

  • HostName is the IP address of the Raspberry. Use nmap again if you forgot it.

  • ForwardX11 and ForwardX11Trusted are required because we will redirect the video stream to the main computer in a few seconds.

With all these settings, you may be wondering if this was all worth it. Let’s get a bit of instant gratification to prove that it was. Via an SSH connection, we will now open an application with a UI, for example the included Chromium browser.

You can switch the display of the windows on the Raspberry screen by changing the setting of the DISPLAY variable .
  • To display the Raspberry Pi screen, use export DISPLAY=:0.

  • To display the local computer screen, use export DISPLAY=:10.

Here’s the list of commands:
ssh pi4
export DISPLAY=:10
chromium-browser
Figure 3-16 shows how Chromium is executing on the Pi but opening its window on the computer.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig16_HTML.jpg
Figure 3-16

Opening a remote application

Setting Up Visual Code Studio for Remote Use

One of the main concepts of this book is that you will be able to write code on a machine with a big screen and then get it to run on a remote device, redirecting the visual output to the big screen.

Up to now, you have been writing code in Visual Studio Code and running it locally. From now on, you’ll write code locally and run it remotely.

All those SSH setup efforts earlier in the chapter were made because Visual Studio Code has a plugin named Remote-SSH that, once you have the SSH setup done, does all the rest for you and fully integrates with the editor’s ecosystem.

Let’s first install the Remote-SSH plugin, as shown in Figure 3-17.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig17_HTML.jpg
Figure 3-17

Installing the Remote-SSH plugin

Once the plugin is installed and after a quick refresh of the editor, you can start the Connect to Host command available from the Command launcher, as shown in Figure 3-18.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig18_HTML.jpg
Figure 3-18

Connecting to the host

Caution

The Remote-SSH plugin may not work properly if the current project has too many files. If that seems to be the case, make sure to open a new blank window in Visual Studio Code before opening a new Remote-SSH session.

Figure 3-19, Figure 3-20, and Figure 3-21 show you the steps necessary to get connected to your Raspberry Pi. Note that the pi4 shortcut is taken from the one you just set up in the $HOME/.ssh/config file. This will be different if you used another shortcut.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig19_HTML.jpg
Figure 3-19

Choosing pi4

../images/490964_1_En_3_Chapter/490964_1_En_3_Fig20_HTML.jpg
Figure 3-20

Setting up the SSH host

../images/490964_1_En_3_Chapter/490964_1_En_3_Fig21_HTML.jpg
Figure 3-21

Connected to the Raspberry Pi from Visual Studio Code

At this stage, from inside Visual Studio Code, you have a terminal remotely connected to your Raspberry Pi. That means you can launch remote commands at will, but you can also unplug the keyboard and the mouse from your Raspberry Pi; the rest will pretty much be done from your local computer.

As a first check, we can run a command to check the IP address of the Pi, as shown here:
ip addr
The output will be shown on the Terminal tab of Visual Studio Code, as shown in Figure 3-22.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig22_HTML.jpg
Figure 3-22

Confirming the IP address on the Terminal tab

Outputting the IP address is obviously not the main thing we can do with that terminal, so let’s move on and install the required Java Development Kit now.

Setting Up the Java OpenJDK

All the code in this book is Java code, so we’ll need to install the Java Development Kit and runtime on the Raspberry Pi.

As an optional dependency, you can also install Maven. Maven is not really required for most of the examples of the book, but this will stop Visual Studio Code from displaying annoying messages that it cannot find the (unnecessary) Maven executable.

So, on the Terminal tab of Visual Studio Code, let’s get the software package installer, apt, to install Java for us. This is done via the small snippet shown here:
sudo apt update
sudo apt install openjdk-11-jdk
Optionally, also run the following command to install Maven:
sudo apt install maven

sudo at the start of each command line is used to get the required escalated privileges to install new software with apt.

The output of the previous commands should be similar to Figure 3-23.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig23_HTML.jpg
Figure 3-23

Installing Java

Alternative to Setting Up the Java SDK

The current selection of JDK on Raspberry Pi 4 running Buster (the latest Raspbian version) is quite impressive. Sometimes, however, especially on older versions of Raspbian, you could be stuck with an outdated JDK for quite some time.

It’s worth noting that companies like Azul and Bellsoft have released up-to-date versions of the JDK specifically compiled for the ARM CPU, the processor running on the Raspberry Pi.

Visit these two sites for direct links to the JDK packages:
https://bell-sw.com/pages/java-11.0.5%20for%20Embedded/
https://www.azul.com/downloads/zulu-community/
The following short snippet will download and install the JDK 11 from Bellsoft:
sudo wget https://download.bell-sw.com/java/11.0.5+11/bellsoft-jdk11.0.5+11-linux-arm32-vfp-hflt.deb
sudo dpkg -i bellsoft-jdk11.0.5+11-linux-arm32-vfp-hflt.deb
The available version is indeed Java 11, as shown in the following output:
pi@raspberrypi:~ $ java -version
openjdk version "11.0.5-BellSoft" 2019-10-15
OpenJDK Runtime Environment (build 11.0.5-BellSoft+11)
OpenJDK 32-Bit Server VM (build 11.0.5-BellSoft+11, mixed mode)

With the Java Development Kit now installed on your Raspberry Pi, let’s move on to the meat of this chapter, running OpenCV code in Java and on the Pi.

Checking Out the OpenCV/Java Template

We’ll start by checking out the OpenCV/Java project template directly on the Raspberry. There are a few ways to do this; the three main ones are as follows:
  • Directly perform a Git clone of the template

  • Download the zip file from the Git project template

  • Generate the templated project using Maven

Let’s quickly review these three one by one.

Performing a Git Clone

It is possible and convenient to simply check out code from Visual Studio Code straight to your Pi. This is done via the command palette and typing anything starting with git.

You will get a few options, one of them being Git: Clone , as shown in Figure 3-24.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig24_HTML.jpg
Figure 3-24

Git:Clone option in Visual Studio Code

From here, enter the location of the template repository as follows, as shown in Figure 3-25:
https://github.com/hellonico/opencv-java-template.git
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig25_HTML.jpg
Figure 3-25

Entering the template URL

Finally, confirm by clicking Open in the dialog box (Figure 3-26).
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig26_HTML.jpg
Figure 3-26

Of course, open the cloned repository

The project is now checked out on the Raspberry and ready to be executed.

Downloading the Zip File

Alternatively, you can download the zip file from the project template. This can be done using the wget command in the terminal in case you do not want to use Git at all.
wget https://github.com/hellonico/opencv-java-template/archive/master.zip
unzip master.zip
Figure 3-27 shows the Terminal tab of Visual Code Studio.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig27_HTML.jpg
Figure 3-27

Downloading the zip file from the Pi

This has the advantage of not being reliant on Git but has the consequence that it is harder to share updates between different Raspberry Pis.

Using Maven

A third way to set up the project is the same technique that was used to originally create the Git template repository: using the build tool Maven. In the Java ecosystem, quite a few people are a bit afraid of going the Maven way, but for a set of given tasks, Maven is a pretty solid choice.

Maven , once installed, can create a project for you with all the required files through the Maven archetypes.

The following command, once copied and pasted onto the Terminal tab, will generate the project structure for you, as shown here:
mvn org.apache.maven.plugins:maven-archetype-plugin:2.4:generate
   -DarchetypeArtifactId=maven-archetype
   -DarchetypeGroupId=origami
   -DarchetypeVersion=1.0
   -DarchetypeCatalog=https://repository.hellonico.info/repository/hellonico/
   -Dversion=1.0-SNAPSHOT
   -DgroupId=hello
   -DartifactId=opencv-java-template
The important parameters of the command have been highlighted in the previous snippet and are explained here:
  • archetypeVersion: This is the version of the project structure, here 1.0. There may be a few updates soon, but don’t expected too many. It’s better to create your own.

  • version: This is the version of your project; it usually starts at 1.0-SNAPSHOT.

  • groupId: This is the Maven group of your project, like com.google, etc.

  • artifactId: This is the name of the project within the groupId. It must be unique.

    Here we are using an older version of the archetype plugin; with this version, it is easier to retrieve artifacts from the custom Maven repository at repository.hellonico.info.

    Later versions of the archetype plugin require a deeper knowledge of the Maven framework, which is out of the scope of this book.

Whichever technique you use, you can now open the remote folder. You should see something similar to the tree in Figure 3-28.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig28_HTML.jpg
Figure 3-28

Project files on the Raspberry Pi

Again, the files are located on the Raspberry Pi, the Java code will be executed on the Raspberry Pi, but the editor is effectively running on your local computer.

Installing the Visual Code Java Extension Pack Remotely

If you remember from Chapter 1, we installed plugins to get Visual Studio Code to understand and run Java. We’ll do the same here, but this time on the Raspberry Pi.

Figure 3-29 shows the Marketplace tab, where you can search for the same plugin that was installed before, the Java Extension Pack. Note in Figure 3-30 how Visual Studio Code nicely asks you if you want to install the plugin on the Raspberry.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig29_HTML.jpg
Figure 3-29

“SSH. PI4” has no extension yet

../images/490964_1_En_3_Chapter/490964_1_En_3_Fig30_HTML.jpg
Figure 3-30

Looking for the Java Extension Pack again

Figure 3-31 and Figure 3-32 show the last two steps of installing the Visual Code Java plugin on the Raspberry Pi.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig31_HTML.jpg
Figure 3-31

Finding the Java Extension Pack again

../images/490964_1_En_3_Chapter/490964_1_En_3_Fig32_HTML.jpg
Figure 3-32

Editor reload required again

You should now have your editor showing the familiar Run and Debug links straight in the editor, as shown in Figure 3-33.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig33_HTML.jpg
Figure 3-33

Familiar Run and Debug buttons showing

All the steps were quite long the first time around, but now you’re ready for some good times. You are all set up for running some OpenCV/Java on the Raspberry Pi.

Running the First OpenCV Example

Without further ado, let’s click the Run button and get the program to execute.

Clicking the Run button runs the same program as in Chapter 1, but the program is executed on the Raspberry Pi this time. The output shows the familiar 3×3 matrix, as in Figure 3-34.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig34_HTML.jpg
Figure 3-34

The code has executed on the Raspberry Pi

Note that various downloads are done in the background on the first run, so the command takes a bit of time. On the second run, things are quicker.

As a reminder, debugging works here in the same way as in Chapter 1. As an exercise, try adding a breakpoint to pause the program execution after running Debug and then add a watch expression on System.getProperty() to show the java.vm.vendor value. Since we are now running on the Raspberry Pi, the value should be Raspbian, as shown in Figure 3-35.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig35_HTML.jpg
Figure 3-35

Debugging and displaying java.vm.vendor

Running on Linux or a VM with AWS Instead

So, what if you were actually not running on a Raspberry Pi but on a remote Linux server or a cloud virtual machine ?

All the setup you have done up to now was actually very generic, so you could have been connecting to a standard box running Linux, provided you have SSH access to it.

For example, in Figure 3-36, you can see the same setup connecting to a remote Linux box and running my favorite desktop distribution, Manjaro Linux.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig36_HTML.jpg
Figure 3-36

Connecting remotely to a running Linux box

To give you some more creative ideas, I am also using that same setup to connect to remote machines on Amazon Web Services.

Capturing a Video Live Stream

If your webcam is plugged in properly via the USB port, you are now ready to write some code to capture the video stream straight from your IoT device.

We use two building blocks here to get us started.
  • OpenCV’s VideoCapture Java class to get the hardware to the software (or more precisely, from the video stream to the OpenCV Mat objects). In this example, the VideoCapture object takes an int as the ID of the video capture device to use.

  • The ImShow class, which is not part of the core OpenCV Java package. It’s a small class similar to OpenCV’s HighGui class (https://docs.opencv.org/master/javadoc/org/opencv/highgui/HighGui.html), but it makes it easier to just plug into video streams and show them on the screen.

We also use init OpenCV with a convenient method called Origami.init(), which searches for and loads the proper binary OpenCV library for us.

Listing 3-1 shows the full code for this first streaming example.
package hello;
import org.opencv.core.Mat;
import org.opencv.videoio.VideoCapture;
import origami.ImShow;
import origami.Origami;
public class Webcam {
    public static void main(String[] args) {
        Origami.init();
        VideoCapture cap = new VideoCapture(0);
        Mat matFrame = new Mat();
        ImShow ims = new ImShow("Camera", 800, 600);
        while (cap.grab()) {
            cap.retrieve(matFrame);
            ims.showImage(matFrame);
        }
        cap.release();
    }
}
Listing 3-1

Accessing and Displaying the Video Stream

The code is pretty standard OpenCV code. Running the code should give a picture like in Figure 3-37 (without the frame rate showing).
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig37_HTML.jpg
Figure 3-37

Video stream without frame rate

You’ll notice the speed is very close to real-time. The code in Listing 3-2 can be added to the frame loop to show the frame rate, as in Figure 3-38.
long now = System.currentTimeMillis();
frame++;
Imgproc.putText(matFrame, "FPS " + (frame / (1 + (now - start) / 1000)), new Point(50, 50),
Imgproc.FONT_HERSHEY_COMPLEX, 2.0, new Scalar(255, 255, 255));
Listing 3-2

Adding the Frame per Second on the Frame

../images/490964_1_En_3_Chapter/490964_1_En_3_Fig38_HTML.jpg
Figure 3-38

Video stream with the frame rate

Usually between 15 and 20 frames per second is an expected value for this exercise.

You may have run into problems while running the previous example. Sometimes the program dies with the stack trace shown in Listing 3-3.
Exception in thread "main" java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
        at java.desktop/java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:208)
        at java.desktop/java.awt.Window.<init>(Window.java:548)
        at java.desktop/java.awt.Frame.<init>(Frame.java:423)
        at java.desktop/java.awt.Frame.<init>(Frame.java:388)
        at java.desktop/javax.swing.JFrame.<init>(JFrame.java:180)
        at origami.ImShow.<init>(ImShow.java:52)
        at hello.Webcam.main(Webcam.java:22)
Listing 3-3

No X11 DISPLAY Set

To run a graphic program via SSH, like you are doing right now, Java requires a system environment variable to be set. The name of this variable, as you can see, is DISPLAY. You can check its value on the Terminal tab with the following:
echo $DISPLAY
There are three main states you could be in depending on the value of this variable, as listed here:
  • <empty>, which is not so good. You need to set the value yourself.

  • :0, or localhost:0, which means any visual frame will be shown in the Raspberry screen. This is OK for debugging the setup, but it’s not the one you usually want when working from your computer.

  • :10, or localhost:10, which means the frames will be shown on your computer. On macOS, this is done via the XQuartz application, which runs a graphical environment compatible with the one from the Raspberry Pi. On Windows, the equivalent is Xming: https://sourceforge.net/projects/xming/.

If you set up the variable properly, then the stream will display on your computer screen, as in Figure 3-39.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig39_HTML.jpg
Figure 3-39

Remote Raspberry Pi stream straight to your computer

You know everything about setting up the Pi for remote development. Let’s move on to some recipes to analyze the content of video streams.

Playing a Video

You may not always have access to a video stream and may have to revert to playing recorded footage. That works too with the VideoCapture class. You can give it the path to the video file you want to play, as shown here:
public class PlayVideo {
    public static void main(String[] args) {
        Origami.init();
        VideoCapture cap = new VideoCapture("marcel_1.mp4");
        Mat matFrame = new Mat();
        ImShow ims = new ImShow("Camera", 400, 300);
        long start = System.currentTimeMillis();
        long frame = 0;
        while (cap.grab()) {
            cap.retrieve(matFrame);
            long now = System.currentTimeMillis();
            frame++;
            Imgproc.putText(matFrame, "FPS " + (frame / (1 + (now - start) / 1000)), new Point(50, 50),
                    Imgproc.FONT_HERSHEY_COMPLEX, 2.0, new Scalar(255, 255, 255));
            ims.showImage(matFrame);
        }
           cap.release();
    }
}

This code is the same as playing a stream taken from a webcam, except the file name is a parameter to the VideoCapture constructor.

Note that you can also find some free sample videos on the following web site:
https://sample-videos.com/index.php
Figure 3-40 shows some of Marcel’s morning activities.
../images/490964_1_En_3_Chapter/490964_1_En_3_Fig40_HTML.jpg
Figure 3-40

Marcel in the morning

If you try to enter an http:// URL as a path to VideoCapture, you will see that things do not work as you probably expected. The remote video you wanted to play will not load.

A URL in the path is used to play streams from a network webcam, not static files. If you have one on hand, it’s a good exercise to plug the web URL of the webcam directly into the VideoCapture object.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.223.108.105