Searching with Ajax

Our users are ecstatic about the search feature we released, and Squeaker has taken over most of the world, leaving millions of messages in our system. Some of the users have complained that despite the new search functionality, it takes too long to find what they are looking for. During usability testing sessions, we also discovered that many of the users don’t even know what they are looking for!

We have come up with an idea that we think might improve this. We’re going to try to make search results more incremental by displaying them as the user types—even before they hit the Search button. This is described by a new scenario:

There is a subtle but essential difference in our When step here. We’re only typing text, and unlike the When step in our previous scenario, we are not submitting the form by clicking the button or hitting the Enter/Return key. We have an idea about how to implement this feature—we’ll issue an Ajax request whenever the user types a character, and when the response comes back, we’ll display the results on the same page without doing a full page load.

Using Selenium

When we implemented our basic search feature, we used Capybara to access our web interface using Capybara’s Rack driver, which is what Capybara uses by default. Capybara’s Rack driver is a browser simulator implemented in Ruby that connects directly to the web server interface of your Rails application, making it nice and fast. While it knows how to parse the HTML on our page and create a simplified DOM^[64] that we can interact with and query, it completely ignores any JavaScript on the page because it doesn’t have a JavaScript engine built in. So, how can we use Capybara to test the JavaScript-based autosearch we’re about to write?

We have to tell Capybara to use a driver that supports JavaScript instead of the more limited Rack driver. Capybara, being one of Cucumber’s best friends, defines a couple of tagged hooks, as described in ​Tagged Hooks​, to make this easy. One of those tagged hooks will run for each scenario tagged with @javascript and set up Capybara to use a JavaScript-aware driver instead of the default one for any scenario carrying the same tag. Capybara’s default JavaScript driver uses Selenium^[66] to interact with our application. First we need to edit our application’s Gemfile to add selenium-webdriver.

Let’s add that @javascript tag to our scenario to see how it works:

Running this feature will make a browser appear, and all of the interaction now happens via the real browser. If your web application is based on the Rack API (most Ruby-based web frameworks are), Capybara also takes care of starting your application’s web server in a separate thread before opening the browser.

Unsurprisingly, Cucumber exits with a missing step definition. We need to tell it how to enter text in a field without submitting the form. That’s easy! We’ll implement the undefined step definition almost like the one we used in our previous scenario, except this time we won’t click the Search button—we are only going to fill in the field:

Matt says:

Pausing Capybara

It can often be quite frustrating watching Capybara’s browser flash past something that you want to take a closer look at. An easy trick to get Cucumber, and therefore Capybara, to pause during a step is to call Cucumber’s ask method in the step definition. You can use it like this:

​	When ​/^I enter "([^"])*" in the search field$/​ ​do​ |query|
​	visit(​'/search'​)
​	fill_in(​'query'​, ​:with​ => query)
​	ask(​'does that look right?'​)
​	​end​

An alternative is to use a debugger statement to pause the code, which will mean installing a debugger. For Ruby 1.9, you need ruby-debug19.^[67] This gives you the added flexibility of being able to actually try commands on the fly. I especially like to use this when I need to compose a tricky selector in Capybara.

If you run the feature again and look carefully, you will see that the search field is populated with the search text from our Gherkin scenario before the browser is closed when the scenario is done. Neat! This leaves us with only one more pending step—the one where we need to verify that intermediate search results are coming back. It’s time to take a little break and think about how we are going to implement this with Ajax and dynamic HTML.

Designing Our Live Search

When the user types in the search field and waits a short time, the browser will issue an Ajax GET request identical to the one we used for our non-JavaScript version, with one small modification: the request will have an additional Accepts HTTP header, indicating to the server that we want the response as JSON (instead of HTML). When the response comes back to the browser, we’ll iterate over the search results in the JSON we get back and create those <li> elements dynamically with JavaScript.

Implementing this in JavaScript, even with the help of jQuery, will require somewhere between twenty and sixty lines of code, depending on your style. This is not an unsurmountable amount of code, but running Cucumber every time you make a small change to this code is something we recommend against because the feedback loop will be too slow. So, how do we proceed now? There are two main directions you can take from here: TDD or not.

JavaScript TDD

Developing JavaScript with TDD used to be hard to do, because of a lack of good tools, but this is a thing of the past. Unfortunately, Cucumber is not one of those tools—it’s too high level. Low-level TDD with JavaScript is beyond the scope of this book, but if you want to give it a try, we can recommend QUnit^[68] or Jasmine.^[69] Test-Driven JavaScript Development [Joh10] is an excellent book that treats the topic really well.

Without JavaScript TDD

This is how we all started programming. Just write the code and test it manually! Since JavaScript TDD is a big topic that we are not going to cover in this book, we are going to cheat a little and “just write the code.” We are going to add a couple of messages to the development database using the rails console, start up the web server, and code until we have something that works, testing our live search manually in the browser. When we think we have something that works, we can run Cucumber to get a final verification.

This approach is not ideal—on real projects we would use TDD for our JavaScript as well. Still, our Cucumber scenario both will serve as validation and will protect us against regressions should someone change the JavaScript in the future. So, without further ado, we’ll just give you some JavaScript code that we developed exactly this way. Create a new file in app/assets/javascripts/search.js with the following content:

This code defines a jQuery search plug-in and activates it on the input element with a DOM ID of search. Whenever a user (or Selenium!) enters a character, the browser fires the keyup event, and this starts a timer. If a delay of 150ms elapses without any more characters being typed, it issues an Ajax request with the search. We use the delay so that we don’t hammer our server with search requests if the user is a really fast typist. This request also indicates that it would like to have the response back as JSON. When the response successfully comes back with the search results in a JSON array, we iterate over them and render our search results.

Making the Web App Return JSON

As we mentioned earlier, we want the response from the search request to contain JSON when the client explicitly says it wants JSON. This is just a little tweak to the SearchController:

Here we have added a respond_to call at the class level to indicate that we can serve both HTML and JSON. We have also added a respond_with call at the bottom of our show method so that Rails knows what it should turn into JSON—the search results. Rails makes this very easy for us.

Finally, we’ll add an autocomplete="off" attribute to our input field to prevent the browser itself from suggesting values as we type:

That’s a lot of code all of a sudden, and it’s high time we ran Cucumber again!

Our first feature is still passing, but our new one testing the Ajax isn’t yet. The table diffing indicates that the two expected lines did not show up on the page. If you were running this in a console with colors enabled, you would see that the rows I am making dinner and I am going to work are yellow, indicating that they did not show up on the page.

What’s missing? It can be a little hard to tell at this point. Opening up the search page in a browser and doing a little bit of manual testing will help us understand what is going on here. If the search results do indeed show up in the browser, we might have to tweak our step definition a little. Before we bring up the search page, we need to make sure we have something to search for. Cucumber (using DatabaseCleaner) would have cleaned out any data, so we can’t use that. And besides, Cucumber uses the test database, whereas the running server we are about to start will use the development one.

Matt says:

The Progressive Enhancement Pattern

When I first started using Cucumber, Capybara’s predecessor Webrat was the new kid on the testing block. Although using Selenium for full-stack JavaScript testing was possible with Webrat, it wasn’t easy to set up, and we quickly made a decision: we would build the first iteration of every feature to work without JavaScript. This was mainly driven by pragmatism; we could easily use Webrat to test the app through Rails integration testing stack (similar to how Capybara’s rack mode works), and these tests ran quickly rather than waiting for a browser to start up.

What happened surprised us. First, we would receive fancy designs from our designers that could be implemented only with fairly complex JavaScript. Because of our rule, we had to figure out how to deliver the same behavior using basic HTTP posts and gets of forms and URLs—no JavaScript. This required us to simplify the designs for our first iteration. We got a little pushback at first, but we were a tight team, and the designers quickly caught on, especially when we started shipping these first-iteration features quickly and reliably. And funnily enough, it turned out that often these simple HTTP-only versions of the features were actually fine, and the designers decided we should move onto other stuff, instead of building all that complex JavaScript that had been implied by their earlier designs. So, this rule had helped us to do the simple thing. We shipped features and moved on.

When we did have to add JavaScript, we added it on top of the existing working HTTP-only implementation, and we got another surprise: it was easy! Nearly every time, because we’d built the server-side infrastructure to a clean, standard pattern, the JavaScript we needed was clean and simple to write. This technique is known as progressive enhancement.^[70]

Since we haven’t yet created a user interface for posting messages, let’s stick some messages in the database manually:

Let’s start the application:

Now we can open a browser, go to http://localhost:3000/search, and start to poke around. Try to enter I am in the search field without hitting Enter. You should see two results show up automatically. The app seems to be behaving as expected. So, why was our scenario failing? The reason for this is timing.

Dealing with the Asynchronous Nature of Ajax

In our couple of steps, we are typing the search term and then verifying that we have results in our list. The problem is—we are looking for the results too soon, before the Ajax query has a chance to complete! As we discussed in Chapter 9, ​Dealing with Message Queues and Asynchronous Components​, we need to introduce a synchronization point so that we can be sure the search has completed before we proceed in checking what it has returned.

Luckily, Capybara knows how to deal with this situation in a simple way. If we ask Capybara to expect the page to contain a specific DOM element that doesn’t yet exist, Capybara will wait a little (50ms) and try again until the element appears. If it doesn’t appear after a while (two seconds by default, though this is configurable), it will raise an exception, causing the step definition to fail. Let’s add that expectation:

This will make the scenario pass, and what’s more—we have covered the essential parts of the Capybara API. Capybara has more to offer, but it’s mostly variations of what we have already seen.