Iteration I2: Connecting to a Slow Payment Processor with Active Job

The code inside the controllers is relatively fast and returns a response to the user quickly. This means we can reliably give users feedback by checking and validating their orders and the users won’t have to wait too long for a response.

The more we add to the controller, the slower it will become. Slow controllers create several problems. First, the user must wait a long time for a response, even though the processing that’s going on might not be relevant to the user experience. In the previous section, we set up sending email. The user certainly needs to get that email but doesn’t need to wait for Rails to format and send it just to show a confirmation in the browser.

The second problem caused by slow code is timeouts. A timeout is when Rails, a web server, or a browser decides that a request has taken too long and terminates it. This is jarring to the user and to the code, because it means the code is interrupted at a potentially odd time. What if we’ve recorded the order but haven’t sent the email? The customer won’t get a notification.

In the common case of sending email, Rails handles sending it in the background. We used deliver_later to trigger sending an email, and Rails executes that code in the background. This means that users don’t have to wait for email to be sent before we render a response. This is a great hidden benefit to Rails’ integrated approach to building a web app.

Rails achieves this using Active Job, which is a generic framework for running code in the background. We’ll use this framework to connect to the slow payment processor.

To make this change, you’ll implement the integration with the payment processor as a method inside Order, then have the controller use Active Job to execute that method in a background job. Because the end result will be somewhat complex, you’ll write a system test to ensure everything is working together.

Moving Logic Into the Model

It’s way outside the scope of this book to integrate with an actual payment processor, so we’ve cooked up a fake one named Pago, along with an implementation, which we’ll see in a bit. First, this is the API it provides and a sketch of how you can use it:

The fake implementation does some basic validations of the parameters, prints out the payment details it received, pauses for a few seconds, and returns a structure that responds to succeeded?.

If you aren’t familiar with OpenStruct, it’s part of Ruby’s standard library and provides a quick-and-dirty way to make an object that responds to the methods given to its constructor.^[73] In this case, we can call succeeded? on the return value from make_payment. OpenStruct is handy for creating realistic objects from prototype or faked-out code like Pago.

With the payment API in hand, you need logic to adapt the payment details that you added in Chapter 13, ​Task H: Entering Additional Payment Details​, to Pago’s API. You’ll also move the call to OrderMailer into this method, because you don’t want to send the email if there was a problem collecting payment.

In a Rails app, when a bit of logic becomes more complex than a line or two of code, you want to move that out of the controller and into a model. You’ll create a new method in Order called charge! that will handle all this logic.

The method will be somewhat long and has to do three things. First, it must adapt pay_type_params (which you created in ​Dynamically Replacing Components Based on User Actions​, but didn’t use) to the parameters that Pago requires. Second, it should make the call to Pago to collect payment. Finally, it must check to see if the payment succeeded and, if so, send the confirmation email. Here’s what the method looks like:

If you weren’t concerned with how slow Pago’s API is, you’d change the code in the create method of OrdersController to call charge!:

Since you already know the call to Pago will be slow, you want it to happen in a background job, so that users can see the confirmation message in their browser immediately without having to wait for the charge to actually happen. To do this, you must create an Active Job class, implement that class to call charge!, and then add code to the controller to execute this job. The flow looks like the following figure.

Creating an Active Job Class

Rails provides a generator to create a shell of a job class for us. Create the job using it like so:

The argument charge_order tells Rails that the job’s class name should be ChargeOrderJob.

You’ve implemented the logic in the charge! method of Order, so what goes in the newly created ChargeOrderJob? The purpose of job classes like ChargeOrderJob is to act as a glue between the controller—--which wants to run some logic later—--and the actual logic in the models.

Here’s the code that implements this:

Next, you need to fire this job in the background from the controller.

Queuing a Background Job

Because background jobs run in parallel to the code in the controller, the code you write to initiate the background job isn’t the same as calling a method. When you call a method, you expect that method’s code to be executed while you wait. Background jobs are different. They often go to a queue, where they wait to be executed outside the controller. Thus, when we talk about executing code in a background job, we often use the phrase “queue the job.”

To queue a job using Active Job, use the method perform_later on the job class and pass it the arguments you want to be given to the perform method you implemented above. Here’s where to do that in the controller (note that this replaces the call to OrderMailer, since that’s now part of the charge! method):

With this in place, you can now add an item to the cart, check out, and see everything working just as we did before, with the addition of seeing the calls to Pago. If you look at the Rails log when you check out, you should see some logging, like so (formatted to fit the page):

This shows the guts of how Active Job works and is useful for debugging if things aren’t working right.

Speaking of debugging and possible failures, this interaction really should have a test.

System Testing the Checkout Flow

In ​Iteration H2: Testing Our JavaScript Functionality​, you wrote a system test that uses a real browser to simulate user interaction. To test the entire flow of checking out, communicating with the payment processor, and sending an email, you’ll expand that test.

To test the full, end-to-end workflow, including execution of Active Jobs, you want to do the following:

1. Add a book to the cart.
2. Fill in the checkout form completely (including selecting a pay type).
3. Submit the order.
4. Process all background jobs.
5. Check that the order was created properly.
6. Check that email was sent.

You should already be familiar with how to write most parts of this test. Processing background jobs and checking mail, however, are new. Rails provides helpers for us, so the test will be short and readable when you’re done. One of those helpers is available by mixing in the ActiveJob::TestHelper module:

This provides the method perform_enqueued_jobs, which you’ll see in a moment.

The current test just makes assertions about how the pay type selector changes the DOM. Since you now need to submit the form and assert that an order was created, you need to clear out any orders in the test database that might be hanging around from previous test runs.

Next, you’ll need to fill in the pay type details. Since the test currently selects the Check pay type, you can use fill_in to provide a routing number and an account number:

Next, you need to submit the form. Capybara provides the method click_button that will do that; however, it’s important to consider what will happen with the background jobs. In a system test, Rails won’t process the background jobs automatically. This allows you to have the chance to inspect them and make assertions about them.

Since this test is about the user’s experience end-to-end, you don’t need to look at the jobs that have been queued—instead we need to make sure they are executed. It’s sufficient to assert the results of those jobs having been executed. To that end, the method perform_enqueued_jobs will perform any jobs that get enqueued inside the block of code given to it:

When the Place Order button is pressed, the controller executes its code, including queuing a ChargeOrderJob. Because that was initiated inside the block given to perform_enqueued_jobs, Rails will process any and all jobs that get queued.

Joe asks:

How Are Background Jobs Run in Development or Production?

When running the application locally, the background jobs are executed and emails are sent by Rails. By default, Rails uses an in-memory queue to manage the jobs. This is fine for development, but it could be a problem in production. If your app were to crash before all background jobs were processed or before emails were sent, those jobs would be lost and unrecoverable.

In production, you’d need to use a different back end, as detailed in the Active Job Rails Guide.^[74] Sidekiq is a popular open-source back end that works great.^[75] Setting it up is a bit tricky, since you must have access to a Redis database to store the waiting jobs.^[76] If you are using Postgres for your Active Records, Queue Classic is another option for a back end that doesn’t require Redis—it uses your existing Postgres database.^[77]

Next, check that an order was created in the way you expect by locating the created order and asserting that the values provided in the checkout form were properly saved.

Lastly, you need to check that the mail was sent. In the test environment, Rails doesn’t actually deliver mail but instead saves it in an array available via ActionMailer::Base.deliveries. The objects in there respond to various methods that allow you to examine the email:

Note that if you had not used perform_enqueued_jobs around the call to click_button "Place Order", the test would fail. This is because ChargeOrderJob would not have executed, and therefore it would not have created and sent the email.

If you run this test via bin/rails test test/system/orders_test.rb, it should pass. You’ve now tested a complex workflow using the browser, background jobs, and email.

What We Just Did

Without much code and with just a few templates, we’ve managed to pull off the following:

• We configured our development, test, and production environments for our Rails application to enable the sending of outbound emails.

• We created and tailored a mailer that can send confirmation emails in both plain-text and HTML formats to people who order our products.

• We used Active Job to execute slow-running code in the background, so the user doesn’t have to wait.

• We enhanced a system test to cover the entire end-to-end workflow, including verifying that the background job executed and the email was sent.

Playtime

Here’s some stuff to try on your own:

• Add a ship_date column to the orders table, and send a notification when this value is updated by the OrdersController.

• Update the application to send an email to the system administrator— namely, yourself—when an application failure occurs, such as the one we handled in ​Iteration E2: Handling Errors​.

• Modify Pago to sometimes return a failure (OpenStruct.new(succeeded?: false)), and handle that by sending a different email with the details of the failure.

• Add system tests for all of the above.