Let’s Build Something!

Now that you’ve seen multiple types of client-side storage techniques as well as some libraries to help make using them easier, let’s build a real, if simple, application that makes use of some of these techniques. Our application will be a tool for a company intranet (“Camden Incorporated”—coming to the NYSE soon) that lets you search for your coworkers. This could be built using a traditional application server model, but we’ve decided to build something fancy using modern web standards. To make the search near instantaneous, we’ll use client-side storage to keep a copy of the employee database on the user’s browser. This, of course, opens up all kinds of interesting issues.

First off, how do we handle synchronization? Companies aren’t static. People join or leave companies all the time. How often that happens, of course, depends on the company itself, but obviously you have to consider some form of strategy for keeping the user’s copy of data in sync with the real list on the server. Luckily, in our scenario we don’t have to worry about user edits. The server side is always “truth,” which means we can ignore changes on the client side when syncs happen. For our demo we’re not going to worry about syncing at all, but in a real-world demo your application server could provide an API where the client says—and by “says” I mean via code, of course—“My copy of the data was last updated on October 10, 2015 at 8:55 AM.” The server could then respond with a set of changes that have occurred since that date. Those changes could cover deletions (people who left the company), changes (people getting married and changing their name, or getting new titles), and additions (new hires). The client-side code would apply those changes and then make a note of the current time so that the next time it speaks to the server it can correctly receive the changes.

The next issue is a thorny one: privacy. The company database probably has a good deal of data about you that you don’t want to share—like your salary. Remember that we are essentially sending private information to each employee, and while you may trust your employees, you still can’t send information that could put their privacy at risk. A safe metric might be, “If it is on their business card, share it,” but certainly you want to be overly cautious here. And to be clear, you cannot “filter” out the insecure data on the client side. If your app server is returning private data, anyone can clearly see it by opening up their browser developer tools. Anything the browser gets is open to inspection by the user. As a matter of habit I tend to browse the Web with my browser tools open, and I’ll naturally look at Ajax calls and the data just for curiosity’s sake. I’m a “good guy,” but you have to assume that the “not-so-good guys and gals” are looking as well.

The last issue is performance. Given a “small” company of 10,000 people, how do you handle transferring that data to the browser in a performant matter? We’ve said our hypothetical situation here is a company intranet, so we’re already kind of assuming desktop/LAN, but you’ll want to be cognizant of the size of your data packets going to the client. We’ll discuss a way to handle this later in the chapter.

OK, let’s talk data!

Our Sample Data

To keep things as easy as possible, our “server” will be a simple JSON file of data. As we said earlier, we are not going to work with synchronization and creating updates, so a flat JSON file will serve our needs just fine. To make things even easier, we’re going to use a cool, free web service to generate our data: the Random User Generator, shown in Figure 7-1.

Figure 7-1. The Random User Generator

This site provides a free API that returns user information. The user information includes quite a bit of detail and can be useful for demos like the one we’re building here. Example 7-1 is a sample of the output taken from their docs.

{
  results: [{
    user: {
      gender: "female",
      name: {
        title: "ms",
        first: "manuela",
        last: "velasco"
      },
      location: {
        street: "1969 calle de alberto aguilera",
        city: "la coruña",
        state: "asturias",
        zip: "56298"
      },
      email: "[email protected]",
      username: "heavybutterfly920",
      password: "enterprise",
      salt: ">egEn6YsO",
      md5: "2dd1894ea9d19bf5479992da95713a3a",
      sha1: "ba230bc400723f470b68e9609ab7d0e6cf123b59",
      sha256: "f4f52bf8c5ad7fc759d1d415e508aa0b7946d4ba",
      registered: "1303647245",
      dob: "415458547",
      phone: "994-131-106",
      cell: "626-695-164",
      DNI: "52434048-I",
      picture: {
        large: "http://api.randomuser.me/portraits/women/39.jpg",
        medium: "http://api.randomuser.me/portraits/med/women/39.jpg",
        thumbnail: "http://api.randomuser.me/portraits/thumb/women/39.jpg",
      },
      version: "0.6"
      nationality: "ES"
    },
    seed: "graywolf"
  }]
}

While the API is incredibly easy to use, we want a static set of data for our demo. If you sign up at RandomAPI, you get permission to use the random user API for up to 10,000 results. The RandomAPI site is—as you can imagine—a collection of APIs that provide random data. All in all, both sites are really darn useful and you can use them within your own applications as well. It is a great way to work with “sensible” random data while building your application.

For this demo, I signed up and requested 10,000 users. In the zip file of sample code from this book, you can find it in c7/data/users.json. Earlier in this chapter we discussed how you would want to be careful about what data you expose in your application. Since we’re just taking the random user data as is, we definitely have information here that we would absolutely not want to share. Not only that, but our demo is only going to use about half of the user values present in the data, which means quite a bit of wasted data will be sent from the server to the frontend. These are all things you would want to be very cognizant of in a proper, production-ready application. But imagine for a moment that we’ve done that. We’ve streamlined our API down to the bare essentials required to meet the application’s needs. What else can we do to make the data load quicker?

One simple method is GZip compression. This is a setting your web server can use to enable zip compression of assets before they are sent to the browser. The web server is intelligent enough to use this feature only when the browser sends a header saying it supports it, and since almost all modern browsers support it, this is an “easy win” to help speed up your transfers. Apache, especially, makes it fairly trivial to enable. How much does it help?

Our users.json file is 13.5 MB. That isn’t small. Poorly optimized graphics probably won’t go over a megabyte each, so you’re really looking at a big hit here to download that file. Figure 7-2 shows Chrome reporting on the size of the JSON file when it’s requested via the browser. This is before any compression is added.

Figure 7-2. Chrome’s report of the file request

And Figure 7-3 shows the size after compression is enabled in Apache.

Figure 7-3. Yep, that’s smaller

The difference is pretty staggering. Keep in mind that the browser still has to decompress that file on the client side, so you’ll want to use this approach with caution. I still wouldn’t recommend sending more than 10 MB of data over the wire. At least in our case this is an initial, “worst case” load, and later calls—again using an imaginary application server—would send only the changes.

Now that you’ve seen the data in play, let’s look at the finished application.

The Application

When you first hit the application, it will fetch the initial data set (that large JSON file) and begin inserting it into a local data store. Since this can take a little while, a modal window is used to let the user know what’s going on. For the application it is kept rather simple—just one message (Figure 7-4). You could enhance this messaging to report on whether or not the application is downloading the initial data or has moved on to inserting it for storage locally.

Figure 7-4. A message is displayed while the application is setting up

After everything is loaded, a basic form, shown in Figure 7-5, is presented to the user. In this application, you can search only by first and last name.

Figure 7-5. The search form

You can then begin searching. You can search on just the first name, last name, or both. Figure 7-6 shows some sample search results.

Figure 7-6. Search results

The application will also correctly let you know when nothing is found. All in all, this is a rather simple interface. You could add more filters (like business department or managers) to further enhance the searching later. In case you’re curious, those pictures all come from the Random User API itself.

Now that we’ve discussed our data and demonstrated how the application looks, let’s begin looking at the code behind it.

The Code

The application will make use of Local Storage and IndexedDB. Local Storage will just be used as a way to remember if data has been loaded. IndexedDB will store the data itself. For Local Storage, even though the usage will be rather trivial, we’ll use Lockr. For IndexedDB, we’ll use Dexie to simplify both inserting data as well as searching.

Let’s begin by discussing how we’ll determine if data has been cached on the client side. Example 7-2 demonstrates the function written to determine if local data exists.

function haveData() {
    var def = $.Deferred();

    var lastFetch = Lockr.get("lastDataSync");

    if(lastFetch) def.resolve(true);
    else def.resolve(false);

    return def.promise();
}

There are a couple of interesting things going on here. On line one, a deferred object is created so that a promise can be returned. This allows us to use the function in an asynchronous matter. We’re not actually using an asynchronous process, though. As we learned, Local Storage access is synchronous, but in the future we may update the process so that it becomes asynchronous. The code calling this function won’t have to change.

For now, our code simply uses Lockr to check for a lastDataSync property. If this exists, then we have data. It’s going to be a date value we set later, with the idea being that if you hook this code up to a “real” app server in the future, the date would be valuable information to use in determining what new data you need. Let’s look at how this code is called (Example 7-3).

haveData().then(function(hasData) {

    if(!hasData) {
        console.log("I need to setup the db.");
        setupData().then(appReady);
    } else {
        appReady();
    }

});

The code calling haveData() uses the then method of the returned promise to set up what is going to run when the asynchronous process is done. Yes, it really isn’t asynchronous, but as we’ve said, the caller doesn’t need to worry about that. If there is no data, then a call to set up the data will be fired; otherwise, the code will run a function signifying that the search application is ready to go. Let’s look at the data setup function.

First, Dexie requires us to create an IndexedDB database and define the object store that will store data (Example 7-4). This is done earlier in the code within the $(document).ready block.

myDb = new Dexie("employee_database");
myDb.version(1).stores({
    employees:"++id,&email,name.first,name.last"
});
myDb.open();

As demonstrated in the previous chapter, Dexie goes a long way to simplifying IndexedDB usage. You can see the database being created as well as the employees object store. The employees store has an autoincrementing number id, a unique index on email, and indexes on name.first and name.last. These indexes were created based on how we plan on searching for employees. Now let’s move to the function run to  set up data (Example 7-5).

function setupData() {
    var def = $.Deferred();

    //setup modal options
    $("#setUpModal").modal({
        keyboard: false
    });

    //now show it
    $("#setupModal").modal("show");

    //now, fetch the remote data
    $.get("data/users.json", function(data) {
        console.log("Loaded JSON, have "+data.results.length+" records.");
        console.dir(data.results[0].user);

        myDb.transaction("rw", myDb.employees, function() {

            data.results.forEach(function(rawEmp) {

                /*
                We aren't copying the data as is, we modify it a bit.
                Specifically the raw data has some dupes on email/username
                so I make an email based on the compation of both
                */
                var emp = {
                    cell:rawEmp.user.cell,
                    dob:rawEmp.user.dob,
                    email:rawEmp.user.email.split("@")[0]+"."
                    + rawEmp.user.username + "@gmail.com",
                    gender:rawEmp.user.gender,
                    location:rawEmp.user.location,
                    name:rawEmp.user.name,
                    phone:rawEmp.user.phone,
                    picture:rawEmp.user.picture
                };

                myDb.employees.add(emp);
            });

        }).then(function() {

            //hide the modal
            $("#setupModal").modal("hide");

            //store that we synced
            Lockr.set("lastDataSync", new Date());

            def.resolve();

        }).catch(function(err) {
            console.log("error in transaction", err);
        });

    }, "json");

    return def.promise();

}

This one, as you can imagine, is the big one. As with the haveData function, jQuery Deferreds are used to handle the asynchronous nature of the setup. Ignoring the UI items (Bootstrap makes this so easy), the real meat begins with an Ajax call to the JSON file. Once the file is retrieved, a transaction is opened via Dexie. For each user in the JSON data, we need to create a new object that will be stored. In an ideal world, our data would match what we want to store exactly, but since we’re working with data from the Random User API, we need to manipulate it a bit. If you are setting up an app server to feed your code data like this, you will want to try to make it match as best you can. Note that the email address is modified a bit, as the Random User data did not have unique email addresses. This is most likely just a bug on their side, and it was quicker to work around it in the JavaScript. This object is added and—that’s it. When the transaction finishes, the UI is updated again and the earlier deferred is resolved. Note that the final step is to update Local Storage, again via Lockr, with the current time.

Now let’s turn to search. After the data is loaded, or it was determined that the data already existed, appReady is executed as shown in Example 7-6.

function appReady() {
    console.log('appReady fired, lets do this');
    //show the search form
    $("#searchFormDiv").show();
    $("#searchForm").on("submit", doSearch);
}

There isn’t much here. Basically the form is displayed and an event handler for running that search is registered. Let’s look at that in Example 7-7.

function doSearch(e) {
    e.preventDefault();
    var fName = $.trim($firstNameField.val());
    var lName = $.trim($lastNameField.val());

    $results.empty();
    console.log('search for -'+fName+'- -'+lName);

    var fnEmps = [];
    var lnEmps = [];
    myDb.transaction('r', myDb.employees, function() {

        if(fName !== '') {
            myDb.employees.where("name.first").startsWithIgnoreCase(fName)
            .each(function(emp) {
                fnEmps.push(emp);
            });
        }

        if(lName !== '') {
            myDb.employees.where("name.last").startsWithIgnoreCase(lName)
            .each(function(emp) {
                lnEmps.push(emp);
            });
        }

    }).then(function() {
        console.log('done');
        var results = [];

        //just a first name
        if(fName !== '' && lName === '') {
            console.log('first');
            fnEmps.forEach(function(emp) { results.push(emp); });
        //just a last name
        } else if(lName !== '' && fName === '') {
            lnEmps.forEach(function(emp) { results.push(emp); });
        //both
        } else {

            //only return items where ob exists in both
            //to make it simpler, we'll make an index of
            //email values in lnEmps so we can check them
            //quicker while going over fnEmps
            var lnEmails = [];
            lnEmps.forEach(function(emp) { lnEmails.push(emp.email); });

            results = fnEmps.filter(function(emp) {
                return lnEmails.indexOf(emp.email) >= 0;
            });
        }

        //Begin rendering the results.
        if(results.length) {
            results.forEach(function(r) {
                console.log(r.name.first+' '+r.name.last);
                var result = resultTemplate(r);
                $results.append(result);
            });
        } else {
            $results.html("Sorry, nothing matched your search.");
        }

    }).catch(function(err) {
        console.log('error', err);
    });

}

This is another large one, so let’s take it bit by bit. First, the current fields in the search form are retrieved and trimmed. Once we have those fields, the search can begin. Unfortunately, we can’t search for both values in one call, but we can do them both in one transaction. So a transaction is opened and then a search, again using the nice functions Dexie provides, against the first and last name indexes. For each search, the results are placed in an array.

When the transaction is done, that means both searches (or one if only one search field was used) are finished. We then have to merge the results. If you didn’t use both fields, this is a simple matter: the result array for the field you searched for is copied to a results array.

If you used both fields, it is a bit more complex. We want to return results that exist in both arrays. The lnEmps array is looped over to create a simpler array of just email addresses. This then lets us loop over the fnEmps array and accept only those values where a corresponding email address in the last name result exists as well.

Finally, the results are ready. But how to display them? To make it simpler to dynamically write out content in the template, we’ll use Handlebars as a client-side templating language. Handlebars lets us define a template with variable tokens. We can load this template, automatically replace the tokens, and then render out HTML. The template for handling results is defined in index.html (see Example 7-8).

<script id="result-template" type="text/x-handlebars-template">
<div class="panel panel-primary">
    <div class="panel-heading">
        <h3 class="panel-title">{{name.first}}{{name.last}}</h3>
    </div>
    <div class="panel-body">
        <div class="row">
            <div class="col-md-2">
                <img src="{{picture.medium}}" class="img-rounded">
            </div>
            <div class="col-md-10">
                <table style="width:100%">
                    <tr>
                        <td><b>Email:</b></td>
                        <td><a href="mailto:{{email}}">{{email}}</a></td>
                    </tr>
                    <tr>
                        <td><b>Phone:</b></td>
                        <td>{{phone}}</td>
                    </tr>
                    <tr>
                        <td><b>Cell:</b></td>
                        <td>{{cell}}</td>
                    </tr>
                    <tr valign="top">
                        <td><b>Location:</b></td>
                        <td>{{location.street}}<br/>
                        {{location.city}}, {{location.state}} {{location.zip}}</td>
                    </tr>
                </table>
            </div>
        </div>
    </div>
</div>
</script>

In the preceding listing, you can see each token as a value wrapped in double curly braces, {{ and }}. Handlebars can process these tokens and replace them with the actual result data from our search. What’s nice about a client-side templating language is that it makes it much easier to generate dynamic output from JavaScript.

Wrap-up

You can find the complete code for the demo in the zip file you downloaded, and I strongly encourage you to play with it yourself to see what changes you could make. More search filters could be added, and if you’re really motivated you could set up an application server to start working on a “send only what’s changed” API. Good luck!