GetUserMedia()

One of the hardest problems with creating native video communication before WebRTC was not the fact that communication itself was hard to do (though without any peer to peer capabilities, it was very hard to do). No, the most difficult problem was accessing the webcam!

Luckily, we now have navigator.getUserMedia(), which is a method that lets us get both the video and audio from an input device and do something with it using the callback.

navigator.getUserMedia( {audio: true, video: true},
    function(s) {
        vid.src = URL.createObjectURL(s);
        vid.play();
    }, function(error) {throw error;}
);

The first parameter is a configuration object that let’s you choose the type of input that you want to use. Then we have two callbacks, the first for what to do if the input successfully gives you a stream and the second for if it fails.

And that’s that. Simple, right? One of the most annoying problems in web development, now possible using just one method.

Now, while we are talking purely about getUserMedia, I would like to point out that you could do anything with it that you can do with a normal video (since it goes through the video element). In the next few chapters we will be looking at computer vision, a massive topic that is not usually thought of as related to the Web, we will be using getUserMedia as the base for the entire project and it will show you that video is not just for conference calls.

Specifications

As I mentioned at the beginning of the chapter, WebRTC is a set of specifications. It consists of the following three APIs:

• Media Stream API
• PeerConnection API
• DataChannel API

The Media Stream API primarily covers the getUserMedia() method that I explained previously. It requires its own specification because, despite being very easy to use, it is complicated for browser implementers. Peer Connections are also complicated because there has never before been any kind of peer-to-peer networking within the HTML APIs. It provides us with a simple API to use rather than getting bogged down in the messy workings of networking.

Using the API, we create a Peer Connection object to handle all peer-to-peer communication. This object consists of an ICE (Interactive Connectivity Establishment) agent, the state of the connection, and an ICE state. Luckily for us, this is handled using the API too! So instead of working out the correct states and so on, we can simply create the object using new RTCIceCandidate and it handles the states itself.

The Data Channel API is, at the time of writing, the least implemented due to it being much easier to transfer simple data already (using servers) than it is to transfer video and audio in real time. Data channels will probably be more important than the media streams because the data channels allow data to be sent peer to peer, which has countless possibilities such as for making multiplayer games that do not need a server (or at least, the server does not need to take the full load). As we will see in the chapters that follow, the media streams can be extremely powerful but data has a lot more common usages, so it is exciting to be able to transfer data without a server!

Servers

I’m sorry. Yes, WebRTC does not need servers—but it does. If you think about it on very simple terms, you always need a way to find each other. Without a server, unless you manually enter the IP addresses, the Peer Connections simply cannot know where to connect. So we use servers, in our case Node.js, so that messages get to the right clients. This is known as signaling. The complexity of the servers depends on your needs. We will be writing a fairly basic server that handles offers and candidates but will be limited to two clients. It would be fairly simple to extend the server to enable more connections.  I encourage you to attempt this, as it is a great way to understand the intricacies of WebRTC and its networking structure.

Where to Start?

As you can see, we have quite a big task ahead of us. I think we should start with the server because it is the smallest part of the code and acts as a kind of testing suite for us. We are extending the code from Chapter 7, which I have included here for your memory and as a comparison.

var app = require('express')(),
server = require('http').createServer(app),
io = require('socket.io').listen(server);
io.set('log level', 1); // Removes debug logs
server.listen(8080);
 
 
app.get('/:id', function (req, res) {
    res.sendfile(__dirname + '/index.html'),
});
 
io.sockets.on('connection', function (socket) {
    var room = socket.handshake.headers.referer;
    socket.join(room);
    socket.on('message', function (data) {
        io.sockets.in(room).emit('message', data);
    });
    socket.on('leave', function (room) {
        socket.leave(room);
    });
});

We need to keep all this, though some parts will be edited. In applications such as this, where there is potential for it to be quite complicated, I tend to give each socket a unique identifier. To do this I have used node-uuid, so we need to import that at the top. Then within the code that runs when a new socket has connected, we assign the output of uuid.v1() to socket.id. Of course we need to let the client know that it has been given an id, so we emit an 'assigned_id' message that sends the id to the client.

var express = require('express'),
    http = require('http'),
    uuid = require ("node-uuid"),
    app = express(),
    server = http.createServer(app),
    io = require('socket.io').listen(server);
 
 
io.sockets.on('connection', function (socket) {
 
    var room = socket.handshake.headers.referer;
    console.log('JOINED', room);
    socket.join(room);
    socket.id = uuid.v1();
     
    socket.emit('assigned_id', socket.id);
 
    // Everything else goes here
});

The client-side needs to look for the emitted message and to store it within the client-side socket.

socket.on('assigned_id', function(data)  {
  console.log('assigned id' + data);
  socket.id = data;
});

Now we need to create the structure for the signaling. It starts when a client’s webcam is successfully running, it adds its stream to its own peer connection, then creates an offer. Within the callback, the peer connection is given the local description. We get the description from the callback as a parameter. This description holds all the data (as SDP) about the client, such as IP address. I will explain the exact format of the description after we have written the server; it is mostly complicated data for the browser but some of it is useful to know about. The client now needs to let the server know that it is time to send the offer to the other client. To avoid confusion, let’s call the original client c1 and the one it connects to c2. So, the server needs to send any 'received_offer' messages on to the correct socket, c2. It eventually does the same for both 'received_candidate' and 'received_answer' as well; these three are all that are needed for basic switching.

socket.on('received_offer', function(data) {
    console.log("received_offer %j", data);
    io.sockets.in(room).emit('received_offer', data);
});
 
socket.on('received_candidate', function(data) {
  console.log(" received_candidate %j", data);
  io.sockets.in(room).emit('received_candidate', data);
});
 
socket.on('received_answer', function(data) {
  console.log(" received_answer %j", data);
  io.sockets.in(room).emit('received_answer', data);
});

You must remember that the entire code runs on both clients, so c1 will send received_offer to c2 but c2 will also send it to c1. This is how we know when both are ready. When c2 receives the offer it sets its peer connection’s remote description to the one that c1 sent (c1’s local description becomes c2’s remote description and vice versa). It then creates an answer that sends back its local description (again, from the callback); this step may not seem necessary since they both already have each other’s description but it can be considered a confirmation. It also sets a Boolean variable, on the client-side, called connected to be true so that this cannot repeat itself and cause an infinite loop. The peer connection, on each client, registers that an ICE candidate is ready once the local descriptions are set; this triggers an event that is handled by the client using pc.onIceCandidate(), which is then used to send more data (in the form of SDP, which I shall explain soon) to the server by emitting the 'received_candidate' message that was mentioned previously. Of course, c2 now receives the SDP and uses it to add c1 as an ICE candidate to the peer connection.

If all went to according to plan, c1 and c2 will now see each other. Well, they would if we had implemented the client-side. My explanation may have seemed quite confusing but that is because I wanted to explain exactly how it works, not just that the server needs to serve a few messages. It is basically the same concept as we had in Chapter 7 and a pattern that you will often use when writing node.js code. Listing 8-1 is the code for the server, including a message for closing the connection smoothly and another for how many clients are in the room:

Listing 8-1. Server.js

var express = require('express'),
    http = require('http'),
    uuid = require ("node-uuid"),
    app = express(),
    server = http.createServer(app),
    io = require('socket.io').listen(server);
io.set('log level', 1); // Removes debug logs
app.use(express.static('public'));
server.listen(8080);
 
app.get('/:id?', function (req, res) {
    res.sendfile(__dirname + '/index.html'),
});
 
io.sockets.on('connection', function(socket) {
    var room = socket.handshake.headers.referer;
    console.log('JOINED', room);
    socket.join(room);
    socket.id = uuid.v1();
 
    socket.emit('assigned_id', socket.id);
 
    socket.on('debug_clients', function(data) {
        socket.emit('room_count', io.sockets.manager.rooms['/' + room].length);
    });
 
    io.sockets.in(room).emit('room_count', io.sockets.manager.rooms['/' + room].length);
 
    socket.on('received_offer', function(data) {
        console.log(" received_offer %j", data);
        io.sockets.in(room).emit('received_offer', data);
    });
 
    socket.on('received_candidate', function(data) {
        console.log(" received_candidate %j", data);
        io.sockets.in(room).emit('received_candidate', data);
    });
 
    socket.on('received_answer', function(data) {
        console.log(" received_answer %j", data);
        io.sockets.in(room).emit('received_answer', data);
    });
 
    socket.on('close', function() {
        console.log("closed %j", room);
        io.sockets.in(room).emit('closed', room);
    });
 
});

ICE

I’ve mentioned ICE quite a bit. It stands for Interactive Connectivity Establishment, which according to RFC 5245 (http://www.ietf.org/rfc/rfc5245.txt) is “A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols”. The RFC goes into a lot of detail, which some of you may find very interesting, but I think the abstract sums up the purpose quite well. To some of you this will sound like absolute gobbledygook, so hopefully once you have read my brief summaries about ICE and the other technologies then you will understand it.

As I explained in the “Specifications” section of this chapter, the peer connection that every WebRTC client must have is made up of components, including iceConnectionState, iceGatheringState and the ICE Agent. The ICE Agent is the important (yet simple) part. It is the endpoint for WebRTC on each client and it is responsible for communicating with the other endpoints, using a signaling protocol. The reason we use ICE at all is that it is (as the abstract states) a standardized protocol for NAT traversal for UDP communication. This basically means that it is a way of getting through security, such as firewalls, without being blocked. ICE works by checking all possible paths of communication in parallel, which quickly finds the best possible path. The states are used (quite obviously) to find the state of the ICE Agent and they are read-only.

The possible states of iceGatheringState are

• new
• gathering
• complete

The possible states of iceConnectionState are

• starting
• checking
• connected
• completed
• failed
• disconnected
• closed

NAT Traversal

NAT is the technology behind hardware routers (and some commercial switches) that allows many machines behind one router to work using one IP. It takes incoming packets of data, and sends them to the correct device based on where it came from. It translates the internal network addresses (for example, 192.168.1.123) to the outward facing IP and back again. Inbound communication is restricted for many reasons, including security, and usually involves a port whitelist table that the NAT device uses to associate internal addresses to inbound ports (commonly referred to as a firewall).

NAT Traversal is a name for a group of methods of getting around this in a user friendly way. Without NAT Traversal techniques, users would have to manually open ports on their firewall to allow inbound traffic, and give this port information to other connecting parties. Some common NAT Traversal techniques are; Reverse Proxies, Socket Secure (SOCKS), Universal Plug and Play (UPnP), and UDP Hole Punching.

STUN/TURN

STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) are both signaling protocols that are often used as tools by ICE for the NAT traversal. There are open source implementations for each if you wish to run your own server. The two that I used for this chapter (you only need one, but you can have an array of them so that there are backups) are run by Google and Mozilla at stun:stun.l.google.com:19302 and stun:stun.services.mozilla.com. You will notice that both of these servers are STUN. This is mostly because they were the first free servers that I found to be reliable. The main difference between the two is that TURN uses a relay technique, so the data is relayed through to server to get to the client. The upside however is that TURN builds on STUN, so it always tries to use STUN first but if that fails it, then relays. Another advantage of TURN is that it works with Symmetric NATs, which STUN does not.

SDP

If I called the abstract for the ICE protocol gobbledygook, then my vocabulary cannot describe SDP. At first glance (and probably at second and third too), it looks impenetrable. But do not worry, the important bits are quite simple to understand and the rest is documented (primarily in RFC4566). SDP stands for Session Description Protocol and is the form of data that is sent between clients. Below is an example of SDP, in this case running the code for this chapter under localhost and having a connection between two Google Chrome tabs. There is a wide range of possible SDP contents, largely dependent on the stage of communication, so don’t expect it to look identical if you look at your own SDP data. I have only included the first 11 lines because the majority of it is not often useful to us for debugging and it mostly concerned with encryption.

 v=0
 o=- 6888311506875211333 2 IN IP4 127.0.0.1
 s=-
 t=0 0
 a=group:BUNDLE audio video
 a=msid-semantic: WMS YuWoij3TGBXIt0Ud2iqCWMbt7FnkBqSNpP96
 m=audio 1 RTP/SAVPF 111 103 104 0 8 107 106 105 13 126
 c=IN IP4 0.0.0.0
 a=rtcp:1 IN IP4 0.0.0.0
 a=ice-ufrag:qbj6LYoXCSHnUUuq
 a=ice-pwd:C/eziXfWhziBTVpMylTLU2M3

Daunting, I know. Let’s start with the letters before the equal sign. This is called the type and it is always just one character and is case-sensitive. The order of types is predefined, mostly to reduce errors and so that parsers can be implemented more easily. Of course it is also handy for debugging, as we will always know where the data we are looking for should be! I will only explain the first four lines as they are the ones that are most likely to go wrong, the rest are optional and therefore are less likely to break (and if it does, more likely to be a bug in the browser rather than your code). Do note that the attributes are in the form of <type>=<attribute>:<value>. Most attributes are easily searchable online if you are interested in learning about them, and I will probably write an article to accompany this chapter on my site.

Session description
   v=  (protocol version)
   o=  (originator and session identifier)
   s=  (session name)
   i=* (session information)
   u=* (URI of description)
   e=* (email address)
   p=* (phone number)
   c=* (connection information -- not required if included in
        all media)
   b=* (zero or more bandwidth information lines)
   One or more time descriptions ("t=" and "r=" lines; see below)
   z=* (time zone adjustments)
   k=* (encryption key)
   a=* (zero or more session attribute lines)
   Zero or more media descriptions
 
Time description
   t=  (time the session is active)
   r=* (zero or more repeat times)
 
Media description, if present
   m=  (media name and transport address)
   i=* (media title)
   c=* (connection information -- optional if included at
        session level)
   b=* (zero or more bandwidth information lines)
   k=* (encryption key)
   a=* (zero or more media attribute lines)
* means optional

This guide to types was taken from the RFC 4566, so credit goes to The Internet Society. You can see quite easily how the SDP object breaks up into three sections. At the time of writing, all SDP objects start with v=0 because there are no other versions available. In my example, the second line (originator and session identifier) was o=- 6888311506875211333 2 IN IP4 127.0.0.1. The hyphen indicates that there is no username because I was not using a private TURN server. The long number is the unique identifier for the session. Next we find the important part of this line, at least for debugging, the session version; this is the amount of times the session data has changed (each time we send a new message with data, this number increments). You can use the session version to be absolutely sure that the session description was updated. IN stands for Internet, it is there to make the protocol future proof. We then have the type of IP, in this case IP version 4. Last on the second line is the IP address. I was running locally so it reported the IP address back as being 127.0.0.1 (localhost).

You may notice that my example jumps straight to the timing section, this is an implementation decision (in this case by Chrome) as the lines in between are all optional. The first 0 represents start time and the following 0 represents stop time. Since both are 0, it means the session is permanent.