5

MAKE: PROJECTS

Communicating in (Near) Real Time

So far, most of the networked communications you’ve seen worked through a web browser. Your device made a request to a remote server, the server ran a program, and then it sent a response. The client made a connection to the web server, exchanging some information, and then the connection was broken. In this chapter, you’ll learn how to keep that connection open. You’ll write two different server programs that allow you to maintain the connection in order to facilitate a faster and more consistent exchange between the server and client.

Supplies for Chapter 5

PROJECT 8 & 9: Video Controller

• 1 Arduino MKR1000 AF 3156, RS 124-0657, A ABX00004, GBX00011 (Africa and EU), D 1659-1005-ND
  Alternatively, an Uno-layout board with the WiFi101 shield or WINC1500-based board. AF 3033 with 2891
• 1 WiFi connection to the internet If you access the internet at home with WiFi, you should be all set. Make sure you know the network name and password.
• 1 solderless breadboard D 438-1045-ND, J 20723 or 20601, SF PRT-12615 or PRT-12002, F 4692810, AF 64, SS 319030002 or 319030001
• 5 220-ohm resistors D 220QBK-ND, J 690700, F 9339299, R 707-7612
• 1 perforated printed circuit board AF 1609, D V2018-ND, J 616673, F 4903213, RS 159-5420
• 4 3/4" hex standoffs, 4-40 threads female-female You may need different part numbers to match your enclosure’s particular height, but these will get you started on a search. D 36-2204-ND, RS 123-6835, F 2301244
• 8 1/4" 4-40 thread pan head machine screws D 36-9300-ND, RS 274-5086, F 2500400
• 3-5 LEDs D 160-1144-ND or 160-1665-ND, J 34761 or 94511, F 1855510, RS 228-5972 or 826-830, SF COM-09592 or COM-09590
• 1 rotary encoder with pushbutton AF 377, SF COM-10982 with BOB-11722, SS 311130001
• 1 pushbutton D GH1344-ND or SW400-ND, J 2231822 or 119011, SF COM-09337, F 1634684, RS 718-2213
• Male headers D A26509-20-ND, J 103377, SF PRT-00116, F 1593411
• Female headers D ED7102-ND, F 1122344, SF PRT-00115
• 1 project enclosure See text.

Interactive Systems and Feedback Loops

In every interactive system, there’s a feedback loop: you take action, the system responds, you see the response—or a notification of it—and you take another action. In some systems, the timing of that loop can be very loose, with substantial delays between action and response. In other applications, the timing must be tight.

For example, in the cat bed application in Chapter 3, there’s no need for the system to respond in less than a few seconds, because your reaction is not very time-sensitive. As long as you get to see the cat while he’s on the bed (which may be true for several minutes or hours), you’re happy. Monski Pong in Chapter 2 relies on a reasonably tight feedback loop in order to be fun. If it took a half-second or longer for the paddles to move when you move Monski’s arms, it would be no fun. The timing of the feedback loop depends on the shortest time that matters to the participant.

Any system that requires coordination between action and reaction needs a tight feedback loop. Consider remote control systems, for example, such as a robot operated over a network. In that case, you’d need not only a fast network for the control system, but also a fast response from the camera or sensors on the robot (or in its environment) that are giving you information about what’s happening. You need to be able to both control it quickly and see the results quickly. Networked action games also need a fast network. It’s no fun if your game console reacts slowly, allowing other players with a faster network connection to get the jump on you. For applications like this, an exchange protocol that’s constantly opening and closing connections (like HTTP does) wouldn’t be very effective.

When there’s a one-to-one connection between two networked objects, it’s easy to establish a tight feedback loop. When multiple objects are involved, though, it gets harder. To begin with, you have to consider how the network of connections between all the objects will be configured. Will it be a star network, with all the participants connected through a central server? Will it be a ring network? Will it be a many-to-many network, where every object has a direct connection to every other object? Each of these configurations has different effects on the feedback loop timing. In a star network, the objects on the edge of the network aren’t very busy, but the central one is. In a ring network, every object shares the load more or less equally, but it can take a long time for a message to reach objects on opposite sides of the ring. In a direct many-to-many network, the load is distributed equally, but each object needs to maintain a lot of connections.

In most cases where you have a limited number of objects in conversation, it’s easiest to manage the exchange using a central server. The most common program example of this is a text-based chat server like the text-based chat in videoconferencing applications like Skype or Google Hangout. Server programs that accept incoming clients and manage text messages between them in real time are often referred to as chat servers. The server programs you’ll write in this chapter are variations on the chat server model. These servers will listen for new connections and exchange messages with all the clients that connect. Because there’s no guarantee how long messages take to pass through the internet, the exchange of messages can’t be instantaneous. As long as you’ve got a fast network connection for both clients and server, though, the feedback loop will be faster than human reaction time.
X

Transmission Control Protocol: Sockets & Sessions

Each time a client connects to a web server, the connection that’s opened uses a protocol called Transmission Control Protocol, or TCP. TCP specifies how objects on the internet open, maintain, and close a connection that will involve multiple exchanges of messages. The connection made between any two objects using TCP is called a socket. A socket is like a pipe joining the two objects. It allows data to flow back and forth between them as long as the connection is maintained. Both sides need to keep the connection open in order for it to work.

For example, think about the exchanges between a web client and server that you saw in the previous two chapters. The pipe is opened when the server acknowledges the client’s contact, and it remains open until the server has finished sending the file. If there are multiple files needed for a web page, such as images and style sheets, then multiple socket connections are opened and closed.

There’s a lot going on behind the scenes of a socket connection. The exchange of data over a TCP connection can range in size anywhere from a few bytes to a few terabytes or more. All that data is sent in discrete packets, and the packets are sent by the best route from one end to the other.

The period between the opening of a socket and the successful close of the socket is called a session. During the session, the program that maintains the socket tracks the status of the connection (open or closed) and the port number; counts the number of packets sent and received; notes the order of the packets and sees to it that packets are presented in the right order, even if the later packets arrive first; and accounts for any missing packets by requesting that they be resent. All of that is taken care of for you when you use a TCP/IP stack like the Net library in Processing or the firmware on the WiFi modules that you first saw in Chapter 4.

The complexity of TCP is worthwhile when you’re exchanging critical data. For example, in an email, every byte is a character in the message. If you drop a couple of bytes, you could lose crucial information. The error-checking of TCP does slow things down a little, though, and if you want to send messages to multiple receivers, you have to open a separate socket connection to each one.

There’s a simpler type of transmission protocol that’s also common on the internet: User Datagram Protocol (UDP). Whereas TCP communication is based on sockets and sessions, UDP is based only on the exchange of packets. You’ll learn more about it in Chapter 7.

TCP Sockets vs. WebSockets

When you used telnet to connect to a web server in Chapter 3, you were opening a TCP socket connection to the server. HTTP works over TCP sockets. If you’re familiar with web development at all, you may have heard of a new protocol introduced with HTML5 called webSockets. The webSocket protocol allows HTTP clients and servers to establish longer sessions for sustained exchanges, such as you might see in a game or other application that requires a tight interactive loop.

The second server program you’ll write in this chapter is a webSocket server. The first is a TCP socket server. Understanding the first will make the second easier to comprehend.
X

Project 8

A Video Control Application

Media control is a great way to learn about real-time connections. This project is a networked video application that you can control from a physical controller. The server will be a Processing program, and the client will be a physical device that is using a WiFi-connected microcontroller. The clients and the server’s screen have to be physically close so that the operator can see the screen. In this case, you’re using a network for its flexibility in handling multiple connections, not for its ability to connect remote places.

There are a few well-known controls on any video controller: a play/pause button, fast-forward and rewind buttons, and perhaps a knob to allow you to shuttle frame-by-frame through the video. The controller you make for this video server will be even simpler. It will have a play/pause button, a shuttle knob, and a button to connect or disconnect from the server.

Here’s how the system will work:

• The video server shows a video.
• The controller clients connect to the server through a TCP connection.
• When a player connects, the server replies with a greeting string.
• The client can send the following commands:
  • start or stop
  • scroll forward or back n frames
  • disconnect
• The server doesn’t send any messages to the client other than the hello message.

The communications protocol doesn’t define anything about the physical form of the client object. As long as the client can make a TCP connection to the server and can send and receive the appropriate ASCII messages, it can work with the server. You can attach any type of physical inputs to the client, or you can write a client that sends all these messages automatically, with no physical input from the world at all (though that would be boring). Your client doesn’t have to use pushbuttons and a scroll knob if you prefer to use other sensors, as long as you send the messages.

You’ll need to define the specific bytes you want to use for these messages, however, and the format in which you want to send them. Since you’ve learned about key-value pairs, let’s use them, with colons to separate the key and the value. By doing so, it will be easy to use standard programming tools that can parse that format. The messages will look like this:

• Server greeting: connect: n where n is client’s number on the server
• Client video play: playing: n where n is 1 or 0 (for true or false)
• Client scroll: position: n where n is a positive or negative frame count. Positive numbers scroll forward, negative numbers scroll back.
• Client disconnect: exit:n if n=1, this 1 tells the server to disconnect the client
• All messages will be terminated with a newline character ( ).

A Test Chat Server

You need a server to get started. You don’t need code to control the video right now; you just want to confirm that the clients can connect and send messages. Following is a server written in Processing with all the elements to handle network communications. It will let you listen for new clients, and then send them messages by typing in the applet window that appears when you run the program.
X

/*
  Test Server Program
 Context: Processing
 
 Creates a server that listens for clients and prints
 what they say.  It also sends the last client anything that's
 typed on the keyboard.
 */
  
// include the net library:
import processing.net.*;
  
int port = 8080;                         // the port the server listens on
Server myServer = new Server(this, port); // the server object
ArrayList clients = new ArrayList();       // list of clients 

void setup() {
  size(640, 360);                    // set the window size
}

void draw() {
  // listen for clients:
  Client currentClient = myServer.available();
  
  // if a client sends a message, read it:
  if (currentClient != null ) {
    readMessage(currentClient);
  }
}

void readMessage(Client thisClient) {
  // read available text from client as a String and print it:
  String message = thisClient.readStringUntil('
');
  // if there's no message, skip the rest of this function:
  if (message == null) return;
  // print the message and who it's from
  println(thisClient.ip() + ": " + message);  
  
  if (message.contains("exit")) {    // if it's a disconnect message, 
    myServer.disconnect(thisClient); // disconnect client  
    clients.remove(thisClient);      // delete client from the clientList
  }
}

// ServerEvent occurs when a new client connects to the server:
void serverEvent(Server myServer, Client thisClient) {
  println("New client: " + thisClient.ip()); // print client's IP
  clients.add(thisClient);                   // add it to the clientList
  thisClient.write("client:" + clients.size() + "
");     // say hello
}

void keyReleased() {
    myServer.write(key);
}

Run the server and open a telnet connection to it. Remember, it’s listening on port 8080, so if your computer’s IP address is, say, 192.168.1.45, you’d connect like so: telnet 192.168.1.45 8080. If you’re telnetting in from the same machine, you can use: telnet localhost 8080 or telnet 127.0.0.1 8080. (Windows 10 users: telnet may ask for password permission to connect). Whatever you type in the telnet window will show up in Processing’s console pane when you press Enter, and whatever you type in the server’s applet window will show up at the client’s command line. Try sending the messages as described previously: playing, position, and exit. Sending exit:1 will cause the server to close the connection to the client.

Once you understand the server, it’s time to make a microcontroller-based client to connect to it.
X

The Controller Client

The video controller client listens to local input and remote input. The local input is from you, the user. The remote input is from the server. The client is constantly listening to you, but it only listens to the server when it’s connected.

To listen to you, the client needs:

• An input for instructing it to connect. The same input can be used to send a disconnect message.
• An input for sending a scroll message.
• An input for sending a play/pause message.

To let the user know what the client device is doing, add:

• An output to indicate whether the client is connected to the server.
• An output to indicate whether the video should be playing or not
• Outputs to indicate when the buttons or scroll wheel are active

It’s always a good idea to put outputs on the client to give local feedback when it receives input from the user. This is what the last outputs are for. Even if there is no connection to the server, local feedback lets the user know that the client device is responding to her actions. For example, pressing the connect button doesn’t guarantee a connection, so it’s important to separate the output that acknowledges a button push from the one that indicates successful connection. If there’s a problem, this helps the user determine whether the problem is with the connection to the server, or with the client device itself. The circuit for this project includes LEDs triggered directly by the sensors, so no code is needed to activate them.

In the code for this client, you’ll control two other LEDs, which are used to indicate whether the client is connected or disconnected, and whether the remote video should be playing or stopped. If this client had a more complex interaction, you’d need more status indicators.
X

The Circuit

The circuit for the client uses an MKR1000 as its microcontroller. It uses several of the GPIO pins for the pushbuttons, the scroll wheel, and the indicator LEDs. The ESP8266 boards don’t have enough GPIO pins to build this project, so if you want to use one of them, you’ll need to use a second microcontroller connected serially to it in order to control the physical inputs and outputs. You’d also need to break the code into the network code and the physical I/O code as well.

This client uses a rotary encoder as the scroll wheel. A rotary encoder, also called a quadrature encoder, is a sensor that has two output pins and a knob that can turn a full 360°. When the knob turns, the output pins pulse. Their pulsing is out of phase with each other so that by watching the sequence of pulses, you can tell which direction the encoder is turning. The sequence of pulses produced is called gray code. A program to read an encoder must read and interpret gray code. Different encoders have different resolutions. The ones recommended here both produce 24 pulses per revolution, meaning that you’d see 24 changes of the I/O pins every time you turn the encoder one full revolution.

Rotary encoders often incorporate a pushbutton in the shaft, as the ones recommended here do. The pushbutton is handy, as it allows you to combine the play/pause button and the scroll wheel into one physical control. If you’re using the SparkFun encoder, it also has an RGB LED in it, so you could use the red, green, and blue channels of the LED as three of your indicator LEDs if you wish. I used the Adafruit encoder in the illustrations shown here.

This project uses an illuminated pushbutton as well, for the connect button. It’s a convenient way to show the connection state to the server right on the control that affects it. If you’re not using an illuminated pushbutton, you’ll need a separate LED to replace the one inside it.

The pushbuttons are wired differently from a normal switch. First, they’re wired to ground instead of to voltage. When you push one of the buttons, the input pin to which it’s connected will read as LOW. This is called inverted logic, and allows you to read the buttons without a separate pulldown resistor. In the program, you’ll initialize them using pinMode(pinNumber, INPUT_PULLUP). Most microcontrollers have internal pullup resistors on their GPIO pins so that you can wire switches this way.

The pushbuttons also have LEDs connected to them from voltage. This way, when you push the button, the LED lights up, because its cathode connects to ground through the button. There’s no need to add any code to make this happen—the pushbutton will ground the LED circuit automatically, and you’ll get local feedback on when the button is being pressed. The encoder pins also have LEDs wired similarly.

Figures 5-2 and 5-3 show the circuit on a solderless breadboard as usual. Figures 5-4 and 5–5 show the device laid out on a printed circuit prototyping board (known as a perfboard). The perfboard makes it possible for the circuit to be a little smaller and more physically robust. It’s harder to assemble than a solderless breadboard circuit, though. so you should build the circuit on a breadboard first. Both layouts are shown so you can choose the method you like.

You’ll need an enclosure for the circuit. You can buy a project case from your favorite electronics retailer, but you can also get creative. For example, a pencil box from your friendly neighborhood stationery store will work well. Drill holes in the lid for the controls and the LEDs, add a battery inside, and you’re all set. If you’re skilled with a mat knife, you can also make your own box from cardboard or mat board. Figures 5-6 and 5–7 show the box I made, and Figure 5-8 shows the template from which I made it. If you’re lucky enough to have access to a laser cutter, use it; if not, a mat knife and a steady hand can do the job.
X

The Code

This sketch uses two new libraries, Encoder, by Paul Stoffregen and Button, by Michael Adams. You can find and install both using the Library Manager as you did for previous libraries. The Encoder library simplifies the reading of a rotary encoder, It reads the changes in the I/O pins of the encoder and returns the number of steps and direction that the encoder has moved. Similarly, the Button library simplifies the reading of a pushbutton attached to an I/O pin. It reads the state of its I/O pin and keeps track of whether it’s toggled, pressed, or released.

/*
Video client
context: Arduino
*/
  
// include necessary libraries
// initialize global variables
  
void setup() {
  // connect to WiFi
  // initialize inputs and outputs
  // initialize serial communications
}
  
void draw() {
  // read scroll wheel
  // read buttons
  
  // if connected to server,
  //    listen for incoming data from server
  //    if scroll wheel has changed, send change to server
  //    if connect button changed, toggle connect state
  //    if play button changed, toggle playing state
  
  // update status output LEDs
}

// config.h
char ssid[] = "ssid";     //  your network SSID (name)
char password[] = "s3c3r3+!"; // your network password

// include required libraries and config files
#include <SPI.h>
#include <WiFi101.h>
#include <Encoder.h>
#include <Button.h>
#include "config.h"     

const char serverAddress[] = "192.168.0.12"; // server address

int port = 8080;                              // port number
WiFiClient tcpSocket;                         // server socket
  

Encoder myEncoder(0, 1);         // instance of the encoder library
Button playButton(2);            // instances of the button library
Button connectButton(3);
const int playLED = 4;           // pin numbers for the LEDs
const int connectLED = 5;
  
boolean playing = false;         // what the video state should be
long lastPosition  = 0;          // last read position of the encoder

void setup() {
  Serial.begin(9600);             // initialize serial communication
  pinMode(0, INPUT_PULLUP);       // initialize encoder pins
  pinMode(1, INPUT_PULLUP);
  pinMode(connectLED, OUTPUT);    // initialize LED pins
  pinMode(playLED, OUTPUT);
  connectButton.begin();           // initialize buttons
  playButton.begin();
  
  // while you're not connected to a WiFi AP,
  while ( WiFi.status() != WL_CONNECTED) {
    Serial.print("Attempting to connect to Network named: ");
    Serial.println(ssid);           // print the network name (SSID)
    WiFi.begin(ssid, password);     // try to connect
    delay(2000);
  }
  
  // When you're connected, print out the device's network status:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);
}

void loop() {
  // read the sensors:
  readEncoder();
  readButtons();
  
  // check for incoming data from the server:
  if (tcpSocket.connected()) {    // if connected to the server,
    if (tcpSocket.available()) {  // if there is a response from the server,
      String result = tcpSocket.readString();  // read it
      Serial.print(result);       // and print it (for diagnostics only)
    }
  }
  // update the status LEDs:
  digitalWrite(connectLED, tcpSocket.connected());
  digitalWrite(playLED, playing);
}

void readEncoder() {
  long position = myEncoder.read();       // read the encoder
  long difference = position - lastPosition;  // compare to last position
  if (difference != 0) {                  // if it's changed,
    if (tcpSocket.connected()) {          // if the socket's connected,
      tcpSocket.print("position:");        // send the key
      tcpSocket.println(difference);       // send the value
    }
    lastPosition = position;              // update lastPosition
  }
}
  

void readButtons() {
  if (connectButton.toggled()) {          // if connect button has changed
    if (connectButton.read() == LOW) {    // and it's pressed
      if (!tcpSocket.connected()) {       // if you're not connected,
        connectToServer();                // connect to server
      } else {                           // if you're already connected,
        tcpSocket.println("exit:1");       // disconnect
      }
    }
  }                                       // end of connectButton.toggled
  

  if (playButton.toggled()) {             // if the play button has changed
    if (playButton.read() == LOW) {       // and it's LOW
      playing = !playing;                 // toggle playing state
      if (tcpSocket.connected()) {        // if you're connected,
        tcpSocket.print("playing:");       // send the key
        tcpSocket.println(playing);        // send the value
      }
    }
  }                                       // end of playButton.toggled
}                                         // end of readButtons()

void connectToServer() {
  Serial.println("attempting to connect");
  // attempt to connect to the server on the given port:
  if (tcpSocket.connect(serverAddress, port)) {
    Serial.println("connected");
  } else {
    Serial.println("failed to connect");
  }
}

Upload the client code to the microcontroller, then run the Processing chat server that you wrote previously. Open the Serial Monitor so you can see the status messages. Once your controller’s connected to WiFi, press the Connect button, and you should see it connect to the server. Click the Play button or turn the encoder, and you should see the messages come through in Processing’s console pane. If you type in the Processing sketch window and hit the Enter key, the message should show up in the Arduino Serial Monitor as well. Congratulations, you’ve got a working client!

Send a few messages to the server by triggering the encoder and the buttons to get a sense how both the server and the controller client device work. Once you’re comfortable, it’s time to add video to the server application.
X

Adding Video to the Server

The main difference between the video server application and the chat server you’ve already written is that the video server parses the messages from the client and uses them to control the video. The network connection management is the same as the chat server, so you can start with the previous Processing code and add to it to make the server. The changes will be as follows:

• Global variables: add video library and variables to manage the video
• Setup function: add code to set the window size and initialize the video
• Draw function: add code to display a frame of video and text overlay
• serverEvent function: no changes needed
• readMessage function: add code to parse messages into key-value pairs and take appropriate actions

New functions:

• movieEvent function: called by sketch every time a new video frame is ready to read. Use it to read the video.
• scrub function: jumps to new position in the movie

You’ll also need a video for your sketch to run. The video library for playing movies is the same as the one you used to capture video in Chapter 3. It can run a wide variety of video formats, but the Processing foundation recommends encoding your video using the H.264 format. Keep the size small, 640x480 or thereabouts. If you don’t have a video handy, the video library examples provide a few you can use. Save your video in a subdirectory of your sketch’s directory, called data.
X

/*
  Video server
 context: Processing
 */
  
// include necessary libraries
import processing.net.*;
import processing.video.*;
  
// set global variables:
int port = 8080;                     // the port the server listens on
Server myServer;                     // the server object
ArrayList clients = new ArrayList(); // list of clients
Movie myVideo;                       //  movie player
boolean playing = false;             // state if the video: playing/paused
String lastMessage = "";             // last message received from client

void setup() { 
  size(640, 360);
  myVideo = new Movie(this, "movie.mov"); // initialize the video
  myServer = new Server(this, port);      // start the server 
  scrub(0.0);                             // set the video to the start
}

void draw() {
  // listen for clients:
  Client currentClient = myServer.available();
  
  // if a client sends a message, read it:
  if (currentClient != null ) {
    readMessage(currentClient);
  }
  // display the latest video frame:
  image(myVideo, 0, 0, width, height);
  
  // display the client that sent the last message:
  fill(15);                          // dark gray fill
  text(lastMessage, 11, height-19);  // drop shadow
  fill(255);                         // white fill
  text(lastMessage, 10, height-20);  // main text
}

void movieEvent(Movie myVideo) {
  myVideo.read();
}
  
void scrub(float newPosition) {
  myVideo.loop();                // play the video
  myVideo.jump(newPosition);       // jump to the calculated position
  if (!playing) myVideo.pause(); // if the movie should be paused, pause it
}

void readMessage(Client thisClient) {
  // read available text from client as a String and print it:
  String message = thisClient.readStringUntil('
');
  // if there's no message, skip the rest of this function:
  if (message == null) return;
  // print the message and who it's from
  println(thisClient.ip() + "	" + message); 
  
  message = message.trim();                   // trim whitespace 
  String[] decodedMsg = split(message, ":");   // split message at the colon
  String property = decodedMsg[0];            // first part is the key
  int value = int(decodedMsg[1]);              // second part is the value
  
  //    if it's a play/pause message, change the video state:  
  if (property.equals("playing")) {  
    playing = boolean(value);      // convert value to a boolean
    if (playing) {                 // if it's true,
      myVideo.loop();              // play the video
    } else {                       // if not,
      myVideo.pause();             // pause it
    }
  }
  
  // if it's a position message, change the video time:
  if (property.equals("position")) {
    float frames = value * 0.033;
    float videoTime = myVideo.time() + frames;
    scrub(videoTime);      // set the video to the appropriate position
  }
  
  // if it's a disconnect message, disconnect the client:
  if (property.equals("exit") && value == 1) {  
    myServer.disconnect(thisClient); // disconnect client
    clients.remove(thisClient);      // delete client from the clientList
  }
  // save the last client message received for printing on the screen:
  lastMessage = thisClient.ip() + ": " + message;
}

From the client’s point of view, this server will respond exactly the same as the chat server. Run the sketch and try connecting to it using your controller client, or from a telnet connection and you’ll see, However, now it will display and control video in reaction to the client’s messages. Figure 5-9 shows what the server displays, using video shot from a moving train.

You can log into the server with as many clients as you wish. The server will respond to the last client’s message. One side effect of this is that the server’s playing state can get out of sync with the clients’ states. To correct this, you could add code to broadcast the server’s playing state whenever it changes. You’d also need to add client code to receive and act on these messages. See if you can work out how to do this.
X

Upgrade: websocket
Connection: Upgrade

Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

The Server and Browser Client

WebSocket servers are built on top of HTTP servers, so your server for this project will start with a node.js express-based HTTP server like the one you saw in Chapter 4. The browser client will be an HTML page that displays and controls the video using the JavaScript library called p5.js.

/*
Express-based webSocket Server
context: node.js
*/
  
// Include libraries:
var express = require('express');           // the express library
var http = require("http");                 // the http library
  
var server = express();                     // the express server
var httpServer = http.createServer(server); // the http server
  
// serve static files from /public:
server.use('/',express.static('public'));
  
// start the server:
httpServer.listen(8080);                    // listen for http connections

Adding Interactive Elements with p5.js

In Chapter 1 you got a brief introduction to p5.js, a JavaScript framework for making interactive web pages in a Processing-like style. This will be your first project with it. Make a new subdirectory in your project directory called public, and copy the files from the p5.js empty-example project into it. The files in it should be as follows:

public/

index.html

sketch.js

libraries/

p5.js

p5.dom.js

p5.sound.js

If you installed the command-line tool p5-manager, you can install the whole p5.js project on the command line like so:

$ p5 g -b public

Then change directories into the public directory and get the latest versions of the p5.js libraries like so:

$ p5 update

You’ll see this pattern frequently in the chapters that follow: a node.js project that’s a server, with a public subdirectory that’s a p5.js project. Together, node.js and p5.js are a handy combination.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <script type="text/javascript" src="p5.js"></script>
    <script type="text/javascript" src="libraries/p5.dom.js">
    <script type="text/javascript" src="sketch.js">
    <title></title>
  </head>

/*
  Video client
  context: p5.js
*/ 
var myVideo;                          // movie player
  
function setup() {
  myVideo = createVideo("movie.mov"); // initialize the video
  myVideo.size(640, 360);             // set its size
  myVideo.position(10, 30);            // position it
  myVideo.loop();                     // loop the video
}

Adding WebSocket Functionality

Now that you’ve got the HTTP server and the HTML client started, you need to give both the capability to make webSocket connections. So far, you’ve worked on the center and the right side of the diagram in Figure 5-10. You’ve got steps 1 and 2 of that side working, in that you can serve the index.html page and the movie. Now you’ll add the third step on the right side. It’s best to get the webSocket functionality working in software alone before you add the microcontroller to the system.

The communications protocol for this server is the same as the one in the previous project. The messages will look like this:

• Server greeting: connect: n where n is client’s number on the server
• Client video play: playing: n where n is 1 or 0 (for true or false)
• Client scroll: position: n where n is a positive or negative frame count. Positive numbers scroll forward, negative numbers scroll back.
• Client disconnect: exit:n if n=1, this 1 tells the server to disconnect the client
• All messages will be terminated with a newline character ( ).

This time, since the server and the browser client are written in JavaScript, you’ll send the messages in JSON, because it’s easy for JavaScript to parse it that way. That means that you need to surround your messages with curly braces, make sure the key names are in quotation marks, and separate the key-value pairs with commas as needed. Then you can use JSON.parse() and JSON.stringify() to convert them from strings to data objects and back again.

$ npm install ws

/*
Express-based webSocket Server
context: node.js
*/ 
var express = require('express');           // the express library
var http = require("http");                 // the http library
var WebSocketServer = require('ws').Server; // ws library's Server class
  
var server = express();                                // express server
var httpServer = http.createServer(server);             // http server
var wss = new WebSocketServer({ server: httpServer }); // websocket server

// start the servers:
httpServer.listen(8080);                // listen for http connections
wss.on('connection', connectClient);    // listen for webSocket messages

// serve static files from /public:
server.use('/',express.static('public'));
  
function connectClient(newClient) {
  // when a webSocket message comes in from this client:
  function readMessage(data) {
    // you'll see more on this function shortly
  }
  
  // if there's a webSocket error:
  function readError(error){
    console.log(error);
  }
  
  // when a client disconnects from a webSocket:
  function disconnect() {
    console.log('Client ' + newClient.clientName + ' disconnected');
  }
  
  // set up event listeners:
  newClient.on('message', readMessage);
  newClient.on('error', readError);
  newClient.on('close', disconnect);
  
  // when a new client connects, send the greeting message:
  var greeting = {"client": wss.clients.size};
  newClient.send(JSON.stringify(greeting));
}

  // when a webSocket message comes in from this client:
  function readMessage(data) {
    var result = JSON.parse(data);              // parse the JSON
    if (result.hasOwnProperty('clientName')) {  // if there's a clientName,
      newClient.clientName = result.clientName; // name the client
    }
    if (result.hasOwnProperty('playing') ||      // if there's a playing prop
       result.hasOwnProperty('position')) {      // or a position property
      broadcast(newClient, result);             // broadcast it    
    }
    if (result.exit === 1) {                    // if it's an exit message,
      console.log("client " + result.clientName + " logging out");
      newClient.close();                        // close the webSocket
    }
    console.log(result);                        // print the message itself
  }

// broadcast data to connected webSocket clients:
function broadcast(thisClient, data) {
  function sendToAll(client) {
    if (client !== thisClient) {
      console.log('broadcasting from:' + client.clientName);
      client.send(JSON.stringify(data));
    }
  }
 // run the sendToAll function on each element of wss.clients:
  wss.clients.forEach(sendToAll);
}

The webSocket server is now different than your Processing server in two ways. First, the webSocket server keeps its own list in a property called clients, unlike Processing, where you had to maintain your own ArrayList of clients. Second, you’re not really doing anything with the incoming messages; you’re just checking that they have useful information and sending them out to the other clients. The web client code for handling incoming messages will look more like the Processing sketch’s readMessage() function, because they both manage the video. In this case, each client controls its own copy of the video. By contrast, the Processing server controlled the video, not the clients.

You’ve added all the code you need for the server to handle webSockets. Make sure you installed the ws library and run this script from the command line. You won’t see any difference when you reload the web page yet, though, since the script in that page doesn’t yet have code to make a websocket connection. Next you’ll add some code to the sketch.js script to make that happen.

/*
  Video client
  context: p5.js
*/
  
var myVideo;                          // movie player
var socket;                           // the websocket
var lastMessage;                      // last server message received
var host = document.location.host;    // address of the server 
var message = {"client": "browser"};  // what you'll send to the server

function setup() {
  myVideo = createVideo("movie.mov"); // initialize the video
  myVideo.size(640, 360);             // set its size
  myVideo.position(10, 30);           // position it
  myVideo.loop();                     // loop the video
  lastMessage = createDiv('');        // create a div for text messages
  lastMessage.position(10, 10);       // position the div
  socket = new WebSocket('ws://' + host); // connect to server
  socket.onopen = sendIntro;          // socket connection listener
  socket.onmessage = readMessage;     // socket message listener
}

function sendIntro() {
  // convert the message object to a string and send it:
  socket.send(JSON.stringify(message));
}
  
function readMessage(event) {
   // read  text from server:
  var msg = event.data;           // read data from the onmessage event
  var videoTime = myVideo.time(); // get the current video time
  var message = JSON.parse(msg);  // convert incoming message to JSON
  
  // if it's a play/pause message, change the video state:
  if (message.playing) {
    myVideo.loop();
  } else {
    myVideo.pause();
  }
  
  // if it's a position message, change the video time:
  var value = parseFloat(message.position);
  if (!isNaN(value)) {           // if it's a numeric value,
    var frames = value * 0.033;  // 1 frame in seconds at ~30fps
    videoTime += frames;        // add the change to the current time
    myVideo.time(videoTime);     // set the video position
  }
  
  // save the last client message received for printing on the screen:
  lastMessage.html(JSON.stringify(message));
}

The WebSocket Controller Client

The video controller client for this project is physically identical to the controller client for the last project. You can use the same circuit; only the firmware on the microcontroller needs to change. The only difference between this microcontroller client and the last one is that this one needs to make webSocket connections instead of TCP socket connections. To do that, you’ll use the ArduinoHttpLibrary that you’ve already been using. That library features a webSocket class that makes it easy to make the connection.

/*
  Video Controller WebSocket Client
  Context: Arduino, with WINC1500 module
*/
#include <SPI.h>
#include <WiFi101.h>
#include <ArduinoHttpClient.h>
#include <Encoder.h>
#include <Button.h>
#include "config.h"     
  
const char serverAddress[] = "192.168.0.12";  // server address

int port = 8080;                              // port number
WiFiClient tcpSocket;                         // server socket
  
// make a websocket instance
WebSocketClient webSocket = WebSocketClient(tcpSocket, serverAddress, port);

void loop() {
  readEncoder();
  readButtons();
  
  // while websocket is connected, listen for incoming messages:
  if (webSocket.connected()) {
    int msgLength = webSocket.parseMessage();  // get message length
    if (msgLength > 0) {                       // if it's > 0,
      String message = webSocket.readString(); // read it
      Serial.println(message);                 // print it
    }
  }
  // update the status LEDs:
  digitalWrite(connectLED, webSocket.connected());
  digitalWrite(playLED, playing);
}

void readEncoder() {
  long position = myEncoder.read();            // read the encoder
  long difference = position - lastPosition;   // compare to last position
  if (difference != 0) {                       // if it's changed,
    if (webSocket.connected()) {               // and webSocket's connected,
      sendJsonMessage("position", difference);  // send a JSON message
    }
    lastPosition = position;                   // update lastPosition
  }
}

void readButtons() {
  if (connectButton.toggled()) {         // if the connect button has changed
    if (connectButton.read() == LOW) {   // and it's LOW
      if (!webSocket.connected()) {       // if you're not connected,
        connectToServer();               // connect to server
      } else {                          // if you're already connected,
        sendJsonMessage("exit", 1);       // disconnect
      }
    }
  }                                     // end of connectButton.toggled
  
  if (playButton.toggled()) {             // if the play button has changed
    if (playButton.read() == LOW) {        // and it's LOW
      playing = !playing;                  // toggle playing state
      if (webSocket.connected()) {         // if you're connected, 
        sendJsonMessage("playing", playing); // send a message
      }
    }
  }                                     // end of playButton.toggled
}                                       // end of readButtons()

void connectToServer() {
  Serial.println("attempting to connect");
  boolean error = webSocket.begin();   // attempt to connect
  if (error) {
    Serial.println("failed to connect");
  } else {
    Serial.println("connected");
    sendJsonMessage("", 0);  // send the client name and nothing else
  }
}

// This function forms a JSON message to send,
// from a key-value pair:
void sendJsonMessage(String key, int val) {
  webSocket.beginMessage(TYPE_TEXT);   // message type: text
  webSocket.print("{"clientName":"MKR1000"");
  if (key != "") {          // if there is no key, just send name
    webSocket.print(","");  // comma, opening quotation mark
    webSocket.print(key);   // key
    webSocket.print("":");  // closing quotation mark, colon
    webSocket.print(val);   // value
  }
  webSocket.print("}");
  webSocket.endMessage();
}

Upload this sketch to your board and open the Serial Monitor. Make sure your server’s still running, then press the connect button. On the server side, you should see a message like this: { clientName: 'MKR1000' }. The connect LED should light and in the Serial Monitor, you should see a greeting message like this: {"client":1}. If you’ve got a browser client logged in, turn the encoder or press the play button and you should be able to control the video now.

With this version of the video server, you can have both multiple controller clients and multiple display clients, and the server won’t see a perceptible change in performance.

This version doesn’t allow you to target specific controller clients to specific display clients, but now that you have the idea, see if you can modify the system to make that happen.
X

Conclusion

The basic structure of the clients and servers in this chapter can be used any time you want to make a system that manages synchronous connections between several devices on the network. The servers’ main jobs are to listen for new clients, keep track of the existing clients, and make sure that the right messages reach the right clients. They must place a priority on listening at all times.

The clients should also place a priority on listening, but they have to balance listening to the server with listening to the physical inputs. They should always give clear and immediate responses to local input, and should indicate the state of the network connection at all times.

When you’re planning the communications between client and server, keep it simple, and leave the system open for more commands, because you never know when you might decide to add a feature. Make sure to build in responses where appropriate, like the “client” and “exit” messages you saw here. Keep the messages unambiguous and, if possible, keep them short as well. Consider using formats that are native to your programming tools, like JSON was used here.

You’ve seen two different approaches to client-server systems with tight interactive loops here. The first used plain TCP sockets connection between the server and its clients, and the second used webSockets. The plain TCP socket is a simpler approach if both your client and your server can make plain TCP socket connections. The webSocket approach, on the other hand, works well with existing HTTP applications and lets you take advantage of the web browser as client.

Now you’ve seen examples of client-server exchanges with loose interactive loops (the HTTP system in Chapter 4) and tight interactive loops, as you saw in this chapter. With those two tools, you can build almost any application in which there’s a central server and a number of clients. For the next chapter, you’ll step away from the internet and take a look at various forms of wireless communication.
X