Sending Client Data to the Server

Broadly speaking, your two options for sending client data to the server are the querystring and the request body. Normally, if you’re using the querystring, you’re making a GET request, and if you’re using the request body, you’re using a POST request. (The HTTP protocol doesn’t prevent you from doing it the other way around, but there’s no point to it: best to stick to standard practice here.)

It is a common misperception that POST is secure and GET is not: in reality, both are secure if you use HTTPS, and neither is secure if you don’t. If you’re not using HTTPS, an intruder can look at the body data for a POST just as easily as the querystring of a GET request. However, if you’re using GET requests, your users will see all of their input (including hidden fields) in the querystring, which is ugly and messy. Also, browsers often place limits on querystring length (there is no such restriction for body length). For these reasons, I generally recommend using POST for form submission.

Encoding

When the form is submitted (either by the browser or via Ajax), it must be encoded somehow. If you don’t explicitly specify an encoding, it defaults to application/x-www-form-urlencoded (this is just a lengthy media type for “URL encoded”). This is a basic, easy-to-use encoding that’s supported by Express out of the box.

If you need to upload files, things get more complicated. There’s no easy way to send files using URL encoding, so you’re forced to use the multipart/form-data encoding type, which is not handled directly by Express.

Different Approaches to Form Handling

If you’re not using Ajax, your only option is to submit the form through the browser, which will reload the page. However, how the page is reloaded is up to you. There are two things to consider when processing forms: what path handles the form (the action) and what response is sent to the browser.

If your form uses method="POST" (which is recommended), it is quite common to use the same path for displaying the form and processing the form: these can be distinguished because the former is a GET request, and the latter is a POST request. If you take this approach, you can omit the action attribute on the form.

The other option is to use a separate path to process the form. For example, if your contact page uses the path /contact, you might use the path /process-contact to process the form (by specifying action="/process-contact"). If you use this approach, you have the option of submitting the form via GET (which I do not recommend; it needlessly exposes your form fields on the URL). Using a separate endpoint for form submission might be preferred if you have multiple URLs that use the same submission mechanism (for example, you might have an email sign-up box on multiple pages on the site).

Whatever path you use to process the form, you have to decide what response to send back to the browser. Here are your options:

Direct HTML response

After processing the form, you can send HTML directly back to the browser (a view, for example). This approach will produce a warning if the user attempts to reload the page and can interfere with bookmarking and the Back button, and for these reasons, it is not recommended.

302 redirect

While this is a common approach, it is a misuse of the original meaning of the 302 (Found) response code. HTTP 1.1 added the 303 (See Other) response code, which is preferable. Unless you have reason to target browsers made before 1996, you should use 303 instead.

303 redirect

The 303 (See Other) response code was added in HTTP 1.1 to address the misuse of the 302 redirect. The HTTP specification specifically indicates that the browser should use a GET request when following a 303 redirect, regardless of the original method. This is the recommended method for responding to a form submission request.

Since the recommendation is that you respond to a form submission with a 303 redirect, the next question is, “Where does the redirection point to?” The answer to that is up to you. Here are the most common approaches:

Redirect to dedicated success/failure pages

This method requires that you dedicate URLs for appropriate success or failure messages. For example, if the user signs up for promotional emails but there was a database error, you might want to redirect to /error/database. If a user’s email address were invalid, you could redirect to /error/invalid-email, and if everything was successful, you could redirect to /promo-email/thank-you. One of the advantages of this method is that it’s analytics friendly: the number of visits to your /promo-email/thank-you page should roughly correlate to the number of people signing up for your promotional email. It is also straightforward to implement. It has some downsides, however. It does mean you have to allocate URLs to every possibility, which means pages to design, write copy for, and maintain. Another disadvantage is that the user experience can be suboptimal: users like to be thanked, but then they have to navigate back to where they were or where they want to go next. This is the approach we’ll be using for now: we’ll switch to using flash messages (not to be confused with Adobe Flash) in Chapter 9.

Redirect to the original location with a flash message

For small forms that are scattered throughout your site (like an email sign-up, for example), the best user experience is not to interrupt the user’s navigation flow. That is, provide a way to submit an email address without leaving the page. One way to do this, of course, is Ajax, but if you don’t want to use Ajax (or you want your fallback mechanism to provide a good user experience), you can redirect back to the page the user was originally on. The easiest way to do this is to use a hidden field in the form that’s populated with the current URL. Since you want there to be some feedback that the user’s submission was received, you can use flash messages.

Redirect to a new location with a flash message

Large forms generally have their own page, and it doesn’t make sense to stay on that page once you’ve submitted the form. In this situation, you have to make an intelligent guess about where the user might want to go next and redirect accordingly. For example, if you’re building an admin interface, and you have a form to create a new vacation package, you might reasonably expect your user to want to go to the admin page that lists all vacation packages after submitting the form. However, you should still employ a flash message to give the user feedback about the result of the submission.

If you are using Ajax, I recommend a dedicated URL. It’s tempting to start Ajax handlers with a prefix (for example, /ajax/enter), but I discourage this approach: it’s attaching implementation details to a URL. Also, as we’ll see shortly, your Ajax handler should handle regular browser submissions as a fail-safe.

Form Handling with Express

If you’re using GET for your form handling, your fields will be available on the req.query object. For example, if you have an HTML input field with a name attribute of email, its value will be passed to the handler as req.query.email. There’s really not much more that needs to be said about this approach; it’s just that simple.

If you’re using POST (which I recommend), you’ll have to link in middleware to parse the URL-encoded body. First, install the body-parser middleware (npm install body-parser); then, link it in (ch08/meadowlark.js in the companion repo):

const bodyParser = require('body-parser')
app.use(bodyParser.urlencoded({ extended: true }))

Once you’ve linked in body-parser, you’ll find that req.body now becomes available for you, and that’s where all of your form fields will be made available. Note that req.body doesn’t prevent you from using the querystring. Let’s go ahead and add a form to Meadowlark Travel that lets the user sign up for a mailing list. For demonstration’s sake, we’ll use the querystring, a hidden field, and visible fields in /views/newsletter-signup.handlebars:

<h2>Sign up for our newsletter to receive news and specials!</h2>
<form class="form-horizontal" role="form"
    action="/newsletter-signup/process?form=newsletter" method="POST">
  <input type="hidden" name="_csrf" value="{{csrf}}">
  <div class="form-group">
    <label for="fieldName" class="col-sm-2 control-label">Name</label>
    <div class="col-sm-4">
      <input type="text" class="form-control"
      id="fieldName" name="name">
    </div>
  </div>
  <div class="form-group">
    <label for="fieldEmail" class="col-sm-2 control-label">Email</label>
    <div class="col-sm-4">
      <input type="email" class="form-control" required
          id="fieldEmail" name="email">
    </div>
  </div>
  <div class="form-group">
    <div class="col-sm-offset-2 col-sm-4">
      <button type="submit" class="btn btn-primary">Register</button>
    </div>
  </div>
</form>

Note we are using Bootstrap styles, as we will be throughout the rest of the book. If you are unfamiliar with Bootstrap, you may want to refer to the Bootstrap documentation.

We’ve already linked in our body parser, so now we need to add handlers for our newsletter sign-up page, processing function, and thank-you page (ch08/lib/handlers.js in the companion repo):

exports.newsletterSignup = (req, res) => {
  // we will learn about CSRF later...for now, we just
  // provide a dummy value
  res.render('newsletter-signup', { csrf: 'CSRF token goes here' })
}
exports.newsletterSignupProcess = (req, res) => {
  console.log('Form (from querystring): ' + req.query.form)
  console.log('CSRF token (from hidden form field): ' + req.body._csrf)
  console.log('Name (from visible form field): ' + req.body.name)
  console.log('Email (from visible form field): ' + req.body.email)
  res.redirect(303, '/newsletter-signup/thank-you')
}
exports.newsletterSignupThankYou = (req, res) =>
  res.render('newsletter-signup-thank-you')

(If you haven’t already, create a views/newsletter-signup-thank-you.handlebars file.)

Lastly, we’ll link our handlers into our application (ch08/meadowlark.js in the companion repo):

app.get('/newsletter-signup', handlers.newsletterSignup)
app.post('/newsletter-signup/process', handlers.newsletterSignupProcess)
app.get('/newsletter-signup/thank-you', handlers.newsletterSignupThankYou)

That’s all there is to it. Note that in our handler, we’re redirecting to a “thank you” view. We could render a view here, but if we did, the URL field in the visitor’s browser would remain /process, which could be confusing. Issuing a redirect solves that problem.

With most frontend frameworks, it is more common to see form data sent in JSON form with the fetch API, which we’ll be looking at next. However, it’s still good to understand how browsers handle form submission by default, as you will still find forms implemented this way.

Let’s turn our attention to form submission with fetch.

Using Fetch to Send Form Data

Using the fetch API to send JSON-encoded form data is a much more modern approach that gives you more control over the client/server communication and allows you to have fewer page refreshes.

Since we are not making round-trip requests to the server, we no longer have to worry about redirects and multiple user URLs (we’ll still have a separate URL for the form processing itself), and for that reason, we’ll just consolidate our entire “newsletter signup experience” under a single URL called /newsletter.

Let’s start with the frontend code. The contents of the HTML form itself don’t need to be changed (the fields and layout are all the same), but we don’t need to specify an action or method, and we’ll wrap our form in a container <div> element that will make it easier to display our “thank you” message:

<div id="newsletterSignupFormContainer">
  <form class="form-horizontal role="form" id="newsletterSignupForm">
    <!-- the rest of the form contents are the same... -->
  </form>
</div>

Then we’ll have a script that intercepts the form submit event and cancels it (using Event#preventDefault) so we can handle the form processing ourselves (ch08/views/newsletter.handlebars in the companion repo):

<script>
  document.getElementById('newsletterSignupForm')
    .addEventListener('submit', evt => {
      evt.preventDefault()
      const form = evt.target
      const body = JSON.stringify({
        _csrf: form.elements._csrf.value,
        name: form.elements.name.value,
        email: form.elements.email.value,
      })
      const headers = { 'Content-Type': 'application/json' }
      const container =
        document.getElementById('newsletterSignupFormContainer')
      fetch('/api/newsletter-signup', { method: 'post', body, headers })
        .then(resp => {
          if(resp.status < 200 || resp.status >= 300)
            throw new Error(`Request failed with status ${resp.status}`)
          return resp.json()
        })
        .then(json => {
          container.innerHTML = '<b>Thank you for signing up!</b>'
        })
        .catch(err => {
          container.innerHTML = `<b>We're sorry, we had a problem ` +
            `signing you up. Please <a href="/newsletter">try again</a>`
        })
  })
</script>

Now in our server file (meadowlark.js), make sure we’re linking in middleware that can parse JSON bodies, before we specify our two endpoints:

app.use(bodyParser.json())

//...

app.get('/newsletter', handlers.newsletter)
app.post('/api/newsletter-signup', handlers.api.newsletterSignup)

Note that we’re putting our form-processing endpoint at a URL starting with api; this is a common technique to distinguish between user (browser) endpoints and API endpoints meant to be accessed with fetch.

Now we’ll add those endpoints to our lib/handlers.js file:

exports.newsletter = (req, res) => {
  // we will learn about CSRF later...for now, we just
  // provide a dummy value
  res.render('newsletter', { csrf: 'CSRF token goes here' })
}
exports.api = {
  newsletterSignup: (req, res) => {
    console.log('CSRF token (from hidden form field): ' + req.body._csrf)
    console.log('Name (from visible form field): ' + req.body.name)
    console.log('Email (from visible form field): ' + req.body.email)
    res.send({ result: 'success' })
  },
}

We can do whatever processing we need in the form processing handler; usually we would be saving the data to the database. If there are problems, we send back a JSON object with an err property (instead of result: success).

File Uploads

We’ve already mentioned that file uploads bring a raft of complications. Fortunately, there are some great projects that help make file handling a snap.

There are four popular and robust options for multipart form processing: busboy, multiparty, formidable, and multer. I have used all four, and they’re all good, but I feel multiparty is the best maintained, and so we’ll use it here.

Let’s create a file upload form for a Meadowlark Travel vacation photo contest (views/contest/vacation-photo.handlebars):

<h2>Vacation Photo Contest</h2>

<form class="form-horizontal" role="form"
    enctype="multipart/form-data" method="POST"
    action="/contest/vacation-photo/{{year}}/{{month}}">
  <input type="hidden" name="_csrf" value="{{csrf}}">
  <div class="form-group">
    <label for="fieldName" class="col-sm-2 control-label">Name</label>
    <div class="col-sm-4">
      <input type="text" class="form-control"
      id="fieldName" name="name">
    </div>
  </div>
  <div class="form-group">
    <label for="fieldEmail" class="col-sm-2 control-label">Email</label>
    <div class="col-sm-4">
      <input type="email" class="form-control" required
          id="fieldEmail" name="email">
    </div>
  </div>
  <div class="form-group">
    <label for="fieldPhoto" class="col-sm-2 control-label">Vacation photo</label>
    <div class="col-sm-4">
      <input type="file" class="form-control" required  accept="image/*"
          id="fieldPhoto" name="photo">
    </div>
  </div>
  <div class="form-group">
    <div class="col-sm-offset-2 col-sm-4">
      <button type="submit" class="btn btn-primary">Register</button>
    </div>
  </div>
</form>

Note that we must specify enctype="multipart/form-data" to enable file uploads. We’re also restricting the type of files that can be uploaded by using the accept attribute (which is optional).

Now we need to create route handlers, but we have something of a dilemma. We want to maintain our ability to easily test our route handlers, which will be complicated by multipart form processing (in the same way we use middleware to process other types of body encoding before we even get to our handlers). Since we don’t want to test multipart form decoding ourselves (we can assume this is done thoroughly by multiparty), we’ll keep our handlers “pure” by passing them the already-processed information. Since we don’t know what that looks like yet, we’ll start with the Express plumbing in meadowlark.js:

const multiparty = require('multiparty')

app.post('/contest/vacation-photo/:year/:month', (req, res) => {
  const form = new multiparty.Form()
  form.parse(req, (err, fields, files) => {
    if(err) return res.status(500).send({ error: err.message })
    handlers.vacationPhotoContestProcess(req, res, fields, files)
  })
})

We’re using multiparty’s parse method to parse the request data into the data fields and the files. This method will store the files in a temporary directory on the server, and that information will be returned in the files array passed back.

So now we have extra information to pass to our (testable) route handler: the fields (which won’t be in req.body as in previous examples since we’re using a different body parser) and information about the file(s) that were collected. Now that we know what that looks like, we can write our route handler:

exports.vacationPhotoContestProcess = (req, res, fields, files) => {
  console.log('field data: ', fields)
  console.log('files: ', files)
  res.redirect(303, '/contest/vacation-photo-thank-you')
}

(Year and month are being specified as route parameters, which you’ll learn about in Chapter 14.) Go ahead and run this and examine the console log. You’ll see that your form fields come across as you would expect: as an object with properties corresponding to your field names. The files object contains more data, but it’s relatively straightforward. For each file uploaded, you’ll see there are properties for size, the path it was uploaded to (usually a random name in a temporary directory), and the original name of the file that the user uploaded (just the filename, not the whole path, for security and privacy reasons).

What you do with this file is now up to you: you can store it in a database, copy it to a more permanent location, or upload it to a cloud-based file storage system. Remember that if you’re relying on local storage for saving files, your application won’t scale well, making this a poor choice for cloud-based hosting. We will be revisiting this example in Chapter 13.

<script>
  document.getElementById('vacationPhotoContestForm')
    .addEventListener('submit', evt => {
      evt.preventDefault()
      const body = new FormData(evt.target)
      const container =
        document.getElementById('vacationPhotoContestFormContainer')
      const url = '/api/vacation-photo-contest/{{year}}/{{month}}'
      fetch(url, { method: 'post', body })
        .then(resp => {
          if(resp.status < 200 || resp.status >= 300)
            throw new Error(`Request failed with status ${resp.status}`)
          return resp.json()
        })
        .then(json => {
          container.innerHTML = '<b>Thank you for submitting your photo!</b>'
        })
        .catch(err => {
          container.innerHTML = `<b>We're sorry, we had a problem processing ` +
            `your submission.  Please <a href="/newsletter">try again</a>`
        })
    })
</script>

exports.api.vacationPhotoContest = (req, res, fields, files) => {
  console.log('field data: ', fields)
  console.log('files: ', files)
  res.send({ result: 'success' })
}

Improving File Upload UI

The browser’s built-in <input> control for file uploads is, shall we say, a bit lacking from a UI perspective. You’ve probably seen drag-and-drop interfaces and file upload buttons that are styled more attractively.

The good news is that the techniques you’ve learned here will apply to almost all of the popular “fancy” file upload components. At the end of the day, most of them are putting a pretty face on the same form upload mechanism.

Some of the most popular file upload frontends are as follows:

• jQuery File Upload

• Uppy (this one has the benefit of offering support for many popular upload targets)

• file-upload-with-preview (this one gives you full control; you have access to an array of file objects that you can use to construct a FormData object to use with fetch)

Conclusion

In this chapter, you learned the various techniques to use for processing forms. We explored the traditional way forms are handled by browsers (letting the browser issue a POST request to the server with the form contents and rendering the response from the server, usually a redirect) as well as the increasingly ubiquitous approach of preventing the browser from submitting the form and handling it ourselves with fetch.

We learned about the common ways forms are encoded:

application/x-www-form-urlencoded

Default and easy-to-use encoding typically associated with traditional form processing

application/json

Common for (nonfile) data sent with fetch

multipart/form-data

The encoding to use when you need to transfer files

Now that we’ve covered how to get user data into our server, let’s turn our attention to cookies and sessions, which also help synchronize the server and the client.