Chapter 13

Integrating with Hardware

In This Chapter

Using the sensors on devices

Collecting contextual information

Getting feedback to the user

WinRT does one thing that is really hard to do from the .NET framework: It gives us access to devices. The webcam, microphone, sensors, near-field communication, and the touch screen itself is really hard to access from .NET, and it’s very easy in WinRT.

Tight integration with the hardware is important for making the app feel alive to the user. When the user is holding a device and she turns around, have the application react to that if it’s appropriate. When the user touches something on the screen, provide some haptic feedback (that little bump that your phone does when you touch a button). Give the user the ability to communicate with your app by voice.

Closely related to the sensors on a device is the media that can be shown on it. The camera and microphone gives a content creator a path to generating content that makes her device hers, and the HTML5 features I discussed in Chapter 5 close the loop, and give her a way to view that content.

WinRT takes all of the cryptic communication structures out of programming for the device’s sensors. The camera, compass, GPS, accelerometer, proximity, and other sensors all have an object-oriented API that is available from C#, C++, or JavaScript. Making the device feel alive is a priority of Microsoft’s — make sure you make it a priority in your app, too.

Using the Camera and Microphone

These days, almost everything that runs Windows comes with a microphone and a camera. Monitors for desktop PCs even have them now. Creating video and audio media is just something that is expected from Windows, and yet it is fairly difficult compared to other operating systems.

Part of that is because the APIs needed to get to the camera are fairly obscure. The awesome video source dialog box at WM_CAP_DLG_VIDEOSOURCE, for some reason, is just not something that everyone knows about or uses.

WinRT changes that with the Media APIs. Now devices like the webcam are much easier to access via C# or JavaScript — no longer do you have to use C++ or depend on a third-party to get to video sources.

On top of that, WinJS makes good use of the HTML5 media tools to bring the video to the user interface with a minimum of fuss. Taking a photo, recording a video, or creating an audio feed is pretty much handled for you thanks to the WinJS HTML5 renderer.

Windows.Media API

The Windows.Media API is a collection of namespaces that wrap up the previously confounding vfw.dll library into a nice object-oriented package. The functionality of the library is divided pretty evenly between capturing media and managing it afterwards.

Media capture

The most useful class in the namespace is probably Capture. All of the Windows 8 UI for selecting a device can be found here, such as the CameraCaptureUI. This class is the super-simplified version of the media API, with one-liner photo capture capabilities:

var cameraCapture = new Windows.Media.Capture.CameraCaptureUI();

cameraCapture.captureFileAsync(

    Windows.Media.Capture.CameraCaptureUIMode.photo).then(

       process(item), error(err));

That’s all there is to it, but you don’t get much in the way of options.

There are more detailed, finer-grained features of the API too. Capturing media generally depends on selecting a device, and the Windows.Media.Devices namespace should help you out with that. For instance, the MediaDevice class has a getVideoChapterSelector class that gets the device ID for the selected video recording device on the host machine. This can be used with the MediaCapture class for much more detailed control of the webcam.

The section, “Accessing the webcam from your app” later in this chapter offers good initial case studies of using the capture APIs.

Encoding and management

Getting the media from the device is one thing, but making it useful for your app is something else altogether. There is a fine selection of tools in the Windows.Media namespace, which assist with the encoding, transcoding, and conversion of video and audio files in a number of different ways.

Encoding in WinRT is based on the Microsoft Media Foundation introduced in Windows 7. It does three main jobs for you in the encoding department:

Taking a video or audio stream and turning it into a formatted file — for instance, encoding a video as an MP4.

Encoding two streams into a single file — for example, taking an audio and video stream and making a single stream out of it.

Taking an encoded, multiplexed stream and saving it as a file.

The principle class for all of this media magic is the Windows.Media.Transcoding.MediaTranscoder. The MediaTranscoder has a prepare FileTranscodeAsynch function that accepts a profile and asynchronously creates an output file. The profile contains the information about the encoding that is to take place.

For instance, this is how the MediaTranscoder might create an MP4 video:

var transcoder = new Windows.Media.Transcoding.MediaTranscoder();

var profile = Windows.Media.MediaProperties.MediaEncodingProfile.createMp4(

    Windows.Media.MediaProperties.VideoEncodingQuality.hd1080p);

return transcoder.prepareFileTranscodeAsync(sourceFile, destinationFile, profile);

Signing the PlayTo contract

Windows.Media is where you will find the PlayTo contract classes, too. PlayTo is the contract that you have to fulfill to get your media somewhere other than your app, or to hold up your app as an example of something that can play certain types of media.

You can read about using the PlayTo contract in Chapter 9.

Accessing the webcam from your app

These are the main things that users want to do with their webcam: take a picture, record audio, or capture a video. They each have their own ins and outs, but are largely very similar.

Videoconferencing is a popular use of webcams, too. Passing network streams of video is outside the scope of WinJS, however (at least as of this writing). You need to work in C++ to make that happen.

Snap a photo

Taking a photo is pretty simple with the CameraCaptureUI class. You can use this for simple in-app image captures for things like profile pictures and whatnot.

1. Add the webcam capability to the package.appxmanifest file by clicking on the Capabilities tab and selecting the Webcam check box.

The user sees as message asking if it’s okay to use the webcam, as shown in Figure 13-1.

2. Add a button to the HTML that takes the picture:

<body>

   <input type=”button” id=”takeThatPic” value=”Click! Take a pic!” /><br />

</body>

3. Write a function that uses the Camera CaptureUI. It eventually looks like this:

    function takeaPic() {

        var captureUI = new Windows.Media.Capture.CameraCaptureUI();

        captureUI.captureFileAsync(Windows.Media.Capture.CameraCaptureUIMode.photo).then(function (capturedItem) {

            if (capturedItem) {

                //Save the file

            }

            else {

                //the user must have changed his mind

            }

        });

    }

4. Tie the new function to the button using addEventHandler.

var picButton = document.getElementById(‘takeThatPic’);

picButton.addEventListener(‘click’, takeaPic);

That’s about all there is to it. The picture brings up the Capture user interface, with the preview and everything. That leaves you with not much to do except to save the file.

Record a video

I’m sure you aren’t surprised to find out that the CameraCaptureUI can record video instead of audio if you want it to. All you have to do is change the mode in the takeaPic function:

    function takeaPic() {

        var captureUI = new Windows.Media.Capture.CameraCaptureUI();

        captureUI.captureFileAsync(Windows.Media.Capture.CameraCaptureUIMode.video).then(function (capturedItem) {

            if (capturedItem) {

                //Save the file

            }

            else {

                //the user must have changed his mind

            }

        });

    }

You can also capture video using the MediaCapture class. This has a few benefits, notably the profile you can set up. It gives you a lot more options in the way your video is recorded.

That profile is made up of MediaProperties, which is a class that is designed around defining specific properties of the video or audio being stored. Some of the property values that you can set include

AudioEncodingProperties

ImageEncodingProperties

videoEncodingProperties

ContainerEncodingProperties

MediaRatio

MediaPropertySet

MediaEncodingProfile

These settings allow you to get much greater control over the video, but you pay a price in greater code complexity. You’ll need to wire up your own asynchronous methods and provide your own UI for the video capture.

The general process is

1. Enable the webcam capability.

2. Create and initialize the MediaCapture object.

3. Create and initialize the encoding profile.

4. Make a recording UI.

5. Handle the start and stop recording methods.

To see all of this in action, check out the recording sample in the SDK samples. It has a good, if slightly overly complex, breakdown of how it all works.

Capturing audio

Audio recording has about the same structure as video recording using MediaCapture. The MediaProperties API has the audio encoding bits that you need to save audio in a variety of formats. The built-in formats include

AAC audio (M4A)

MP3 audio

Windows Media Audio (WMA)

There will be more formats, and, of course, there is an API to build your own MediaProperties from existing codecs.

Collecting Data from Sensors

Laptops don’t have a lot of sensors. They are pretty much about the keyboard and the screen. Boring, but true. Some of them have a camera, which adds a little bit to the experience.

Contemporary devices, like the Surface and tablet computers, have large numbers of sensors baked in. WinRT gives you access to all of them with a minimum of fuss. The sensor namespaces in WinRT include

Geolocation: GPS and the like

Input: Touch screen and keyboard

Sensors: All the good stuff, like a compass, accelerometer, gyrometer, inclinometer, light sensor, and orientation sensor

Portable: MP3 players and phones you might hook to the device

Printers: You know, for dead trees

SMS: Talks to the cell network

No amount of technical information will help you use the sensor array well in your app. All I can ask you to do is to use good apps. The good apps use the sensors well. The mapping application changes from white background to black background when it gets dark. Stuff like that. You can’t teach that stuff.

Getting the user’s location

There was a day, not long ago (wait, it was yesterday!) when you needed to search through three or four different location APIs to see what the most accurate location of the user is. In WinRT, that at least is taken from you. All of the items that might have location are checked by the API.

Determining if the device is moved

Once upon a time, there was a wonderful sensor called the accelerometer. It could tell if the device was shifted from one position to another. Sadly, Windows programmers could not access such a device because it was so incredibly tough to program for.

Well, no longer is that true.

The Accelerometer class has made things a lot easier. Now calling for the default sensor in this class makes the best of an often bad set of options.

accelerometer = Windows.Devices.Sensors.Accelerometer.getDefault();

WinRT does its best to find the accelerometer for you. This could be under a number of guises — even the hard-drive lock protection. As long as it has a driver, it will be found.

You might want to code for a shake sometime. It’s actually a pretty cool feature. I like my Twitter client, which updates my feed when I shake my phone. It’s pretty straightforward to use the accelerometer in your project now.

You don’t need to declare the accelerometer as a capability. It’s a freebie.

1. Write a function that will be called when the accelerometer registers a shake. It can do whatever you want.

2. Get an instance of the default accelerometer.

3. Set the onreadingchanged event to the function you added.

The code for my example looks like this (working from a default PageControl):

(function () {

    “use strict”;

    var checkingShake, accelerometer;

    function itshook()

    {

        var field = document.getElementById(“isitshaken”);

        field.innerHTML=”It shook!”;

    }

    WinJS.UI.Pages.define(“/pages/shaken.html”, {

        // This function is called whenever a user navigates to this page. It

        // populates the page elements with the app’s data.

        ready: function (element, options) {

            accelerometer = Windows.Devices.Sensors.Accelerometer.getDefault();

            accelerometer.onreadingchanged = itshook;

        },

        updateLayout: function (element, viewState, lastViewState) {

            /// <param name=”element” domElement=”true” />

            /// <param name=”viewState” value=”Windows.UI.ViewManagement.ApplicationViewState” />

            /// <param name=”lastViewState” value=”Windows.UI.ViewManagement.ApplicationViewState” />

            // TODO: Respond to changes in viewState.

        },

        unload: function () {

            // TODO: Respond to navigations away from this page.

        }

    });

})();

If your device does not have an accelerometer, this code doesn’t do anything. No amount of shaking your monitor helps (trust me, I’ve tried). What’s more, I don’t think the simulator does it either. If you have a device without an accelerometer, it won’t work. I tried it with my wife’s ThinkPad, which has a hard-drive shake monitor, and it worked there, for what that’s worth.

But that onreadingchanged event isn’t the only toy. GetCurrentReading gives you a three-dimensional idea of where the device is in space. Although it’s all relative, it does tell you if the user is moving the device left or right, up or down.

The result from getCurrentReading gives you those X, Y, and Z parameters in the return values. To use it, try something like this:

    function getThePosition() {

        var position = accelerometer.getCurrentReading();

        if (position) {

            var x = reading.accelerationX.toFixed(2);

            var y = reading.accelerationY.toFixed(2);

            var z = reading.accelerationZ.toFixed(2);

    }

Using this isn’t straightforward because you need to keep checking the position when the user is in a position to change it. It requires a little algorithmically trickery. Just remember that anything that speaks directly to the hardware of the device like this is streaming cast under WinRT, so you might be in a good position to check the values more often than you think.

Being aware of lighting

You might not be thinking about ambient lighting when you are writing you app, but you should be. You can use the lighting in the room where the user is to determine what styles to show in your app.

The light sensor is less obscure than the others I’ve covered in this chapter. It’s either built into your machine or it isn’t, so as long as you do a little defensive coding, it’s a good tool to use.

Using the light sensor is a lot like the other sensors. You’ll see this pattern repeated throughout the sensor array:

1. Declare function variables.

2. Write a function for the on<sensor>changed event handler.

3. Handle the event.

4. In the handler, get the current reading and do something with it.

This process a pretty common pattern for Windows 8, I’m starting to think. If you are familiar with mobile development in other platforms, you’ve probably seen this before. Here’s how it looks in JavaScript:

(function () {

    “use strict”;

    var light;

    function onLightChanged(eventArgs) {

        light = lightSensor.getCurrentReading();

    }

    WinJS.UI.Pages.define(“/pages/light.html”, {

        // This function is called whenever a user navigates to this page. It

        // populates the page elements with the app’s data.

        ready: function (element, options) {

            lightSensor.onreadingchanged = onLightChanged;

        },

        updateLayout: function (element, viewState, lastViewState) {

            // TODO: Respond to changes in viewState.

        },

        unload: function () {

            // TODO: Respond to navigations away from this page.

        }

    });

})();

This all goes back to giving users a good experience — as though the device is an extension of them. Knowing what sensors your users are likely to have, and taking advantage of that, is an important part of application development in the Windows 8 space.

Handling the fact that the sensor is not there is important too. Don’t forget to check for null values. Don’t depend on user input. Remember, there is a chance that the machine is misconfigured, or that the user has a switch turned off. Assume the worst.

Touching the Surface

Windows 8 is a “touch first” operating system. Machines that come out with Windows 8 preinstalled largely have touch screens enabled, and the tablets of course have a touch screen.

As such, you would expect WinRT to have largely touch-ready APIs and tools, and you would be right. All of the user controls in WinJS are touch-ready, and there is a very solid touch API. All of this is covered in Chapter 6 in depth.

Comprehensive mouse and pointer features

Not every device has a touch screen. Desktops and some laptops just have a mouse and a keyboard, and that’s fine. Everything that is touch-enabled is mouse-enabled too.

There are some design considerations to use with the mouse and the keyboard with Windows Store apps, however. The scroll wheel and whatnot may mimic certain functions of the touch screen, for instance.

I cover the mouse and keyboard and using them with WinJS UI controls in Chapter 5.

Writing with the pen

The last of the input devices you are likely to encounter (at least until the Kinect gets installed on everything — that’s a different API) is the pen. An awesome interface for the pen is built into the WinRT API, but it doesn’t get used often enough. In this section, I take you on a brief tour of the pen — at least enough to get some ink on the screen.

Making a place to draw

To start off, you need a place to draw. In the WinJS world, that means a Canvas element. The Canvas element is an element devised by the W3C (the group that creates the web standards) to allow for faster graphic creation on the web. Scalable vector graphics, or SVG, have been the standard for dynamic graphics on the Internet for many years, but there was a need to move to an updatable bitmap model for more advanced color and motion animation.

As it turns out, it’s perfect for Windows 8, too.

Adding a canvas element is easy:

<body>

    <canvas id=”pieceOfPaper”></canvas>

</body>

Using it is less so. Fortunately, a few WinRT bits make writing on the Canvas a lot easier.

Listening for pen input

To get the pen input, you need to use parts of the Windows.UI.Input.PointerPoint class. (For the record, I did not name that class.) Either way, that’s where you’re going.

1. The first thing that you need to do is to get a reference to the InkManager. Just put that in the declarations section of your JavaScript page.

var inkManager = new Windows.UI.Input.Inking.InkManager();

2. Get an instance of the canvas and set up some event handlers.

You’ll write them in the next few steps. You can put all of this in the onactivated event handler.

    app.onactivated = function (args) {

        if (args.detail.kind === activation.ActivationKind.launch) {

            if (args.detail.previousExecutionState !== activation.ApplicationExecutionState.terminated) {

                inkCanvas = document.getElementById(“Canvas”);

                inkContext = inkCanvas.getContext(“2d”);

                inkCanvas.addEventListener(“MSPointer Down”, onPointerDown, false);

                inkCanvas.addEventListener(“MSPointer Move”, onPointerMove, false);

                inkCanvas.addEventListener(“MSPointerUp”, onPointerUp, false);

            } else {

                // TODO: This application has been reactivated from suspension.

                // Restore application state here.

            }

            args.setPromise(WinJS.UI.processAll());

        }

    };

3. You need three functions for those three events in Step 2.

One handles the down event, one handles move, and one handles up.

    function onPointerDown(evt) {

        pointerDeviceType = getPointerDeviceType(evt.pointerId);

        if ((pointerDeviceType === “Pen”) || ((pointerDeviceType === “Mouse”) && (evt.button === 1))) {

            if (pointerId === -1) {

                var current = evt.currentPoint;

                inkContext.beginPath();

                inkContext.lineWidth = strokeWidth;

                inkContext.strokeStyle = strokeColor;

                inkContext.moveTo(current.position.x, current.position.y);

                inkManager.processPointerDown(current);

                pointerId = evt.pointerId;

            }

        }

    }

    function onPointerMove(evt) {

        pointerDeviceType = getPointerDeviceType(evt.pointerId);

        if ((pointerDeviceType === “Pen”) || ((pointerDeviceType === “Mouse”) && (evt.button === 1))) {

            if (evt.pointerId === pointerId) {

                var current = evt.currentPoint;

                inkContext.lineTo(current.rawPosition.x, current.rawPosition.y);

                inkContext.stroke();

                inkManager.processPointerUpdate(current);

            }

        }

    }

    function onPointerUp(evt) {

        pointerDeviceType = getPointerDeviceType(evt.pointerId);

        if ((pointerDeviceType === “Pen”) || ((pointerDeviceType === “Mouse”) && (evt.button === 0))) {

            if (evt.pointerId === pointerId) {

                inkManager.processPointerUp(evt.currentPoint);

                inkContext.closePath();

                renderAllStrokes();

                pointerId = -1;

            }

        }

    }

4. Last, you need that getPointerDeviceType function.

You could duplicate the code in every event, but why would you?

    function getPointerDeviceType(pId) {

        var pointerDeviceType;

        var pointerPoint = Windows.UI.Input.PointerPoint.getCurrentPoint(pId);

        switch (pointerPoint.pointerDevice.pointerDeviceType) {

            case Windows.Devices.Input.PointerDeviceType.touch:

                pointerDeviceType = “Touch”;

                break;

            case Windows.Devices.Input.PointerDeviceType.pen:

                pointerDeviceType = “Pen”;

                break;

            case Windows.Devices.Input.PointerDeviceType.mouse:

                pointerDeviceType = “Mouse”;

                break;

            default:

                pointerDeviceType = “Undefined”;

        }

        deviceMessage.innerText = pointerDeviceType;

        return pointerDeviceType;

    }

Collecting the ink

Now that you have the event handlers in place, you can just run the app and give it a draw. Remember, if you don’t have a pen on your device, you can use the Simulator to make like a pen.

This is a barebones example of pen input. Two awesome samples in the Windows SDK Samples are Input: Ink Sample and Input: Simplified Ink Sample. What I’ve shown you is simpler than both of those, so if you need something more, you might want to check out the Microsoft code as well.