File and Directory Structure

Structuring applications has spawned many a religious debate, and there’s no one right way to do it. However, there are some common conventions that are helpful to know about.

It’s typical to try to restrict the number of files in your project root. Typically, you’ll find configuration files (like package.json), a README.md file, and a bunch of directories. Most source code goes under a directory often called src. For the sake of brevity, we won’t be using that convention in this book (nor does the Express scaffolding application do this, surprisingly). For real-world projects, you’ll probably eventually find that your project root gets cluttered if you’re putting source code there, and you’ll want to collect those files under a directory like src.

I’ve also mentioned that I prefer to name my main application file (sometimes called the entry point) after the project itself (meadowlark.js) as opposed to something generic like index.js, app.js, or server.js.

It’s largely up to you how to structure your application, and I recommend providing a road map to your structure in the README.md file (or a readme linked from it).

At minimum, I recommend you always have the following two files in your project root: package.json and README.md. The rest is up to your imagination.

Best Practices

The phrase best practices is one you hear thrown around a lot these days, and it means that you should “do things right” and not cut corners (we’ll talk about what this means specifically in a moment). No doubt you’ve heard the engineering adage that your options are “fast,” “cheap,” and “good,” and you can pick any two. The thing that’s always bothered me about this model is that it doesn’t take into account the accrual value of doing things correctly. The first time you do something correctly, it may take five times as long to do it as it would have to do it quick and dirty. The second time, though, it’s going to take only three times as long. By the time you’ve done it correctly a dozen times, you’ll be doing it almost as fast as the quick and dirty way.

I had a fencing coach who would always remind us that practice doesn’t make perfect; practice makes permanent. That is, if you do something over and over again, eventually it will become automatic, rote. That is true, but it says nothing about the quality of the thing you are practicing. If you practice bad habits, then bad habits become rote. Instead, you should follow the rule that perfect practice makes perfect. In that spirit, I encourage you to follow the rest of the examples in this book as if you were making a real-live website, as if your reputation and remuneration were depending on the quality of the outcome. Use this book to not only learn new skills but to practice building good habits.

The practices we will be focusing on are version control and QA. In this chapter, we’ll be discussing version control, and we’ll discuss QA in the next chapter.

Version Control

I hope I don’t have to convince you of the value of version control (if I did, that might take a whole book itself). Broadly speaking, version control offers these benefits:

Documentation

Being able to go back through the history of a project to see the decisions that were made and the order in which components were developed can be valuable documentation. Having a technical history of your project can be quite useful.

Attribution

If you work on a team, attribution can be hugely important. Whenever you find something in code that is opaque or questionable, knowing who made that change can save you many hours. It could be that the comments associated with the change are sufficient to answer your questions, and if not, you’ll know who to talk to.

Experimentation

A good version control system enables experimentation. You can go off on a tangent, trying something new, without fear of affecting the stability of your project. If the experiment is successful, you can fold it back into the project, and if it is not successful, you can abandon it.

Years ago, I made the switch to distributed version control systems (DVCSs). I narrowed my choices down to Git and Mercurial and went with Git, because of its ubiquity and flexibility. Both are excellent and free version control systems, and I recommend you use one of them. In this book, we will be using Git, but you are welcome to substitute Mercurial (or another version control system altogether).

If you are unfamiliar with Git, I recommend Jon Loeliger’s excellent Version Control with Git (O’Reilly). Also, GitHub has a good listing of Git learning resources.

How to Use Git with This Book

First, make sure you have Git. Type git --version. If it doesn’t respond with a version number, you’ll need to install Git. See the Git documentation for installation instructions.

There are two ways to follow along with the examples in this book. One is to type out the examples yourself and follow along with the Git commands. The other is to clone the companion repository I am using for all of the examples and check out the associated files for each example. Some people learn better by typing out examples, while some prefer to just see and run the changes without having to type it all in.

git init

node_modules
*~
.DS_Store

git add -A

git commit -m "Initial commit."

git clone https://github.com/EthanRBrown/web-development-with-node-and-express-2e

npm Packages

The npm packages that your project relies on reside in a directory called node_modules. (It’s unfortunate that this is called node_modules and not npm_packages, as Node modules are a related but different concept.) Feel free to explore that directory to satisfy your curiosity or to debug your program, but you should never modify any code in this directory. In addition to that being bad practice, all of your changes could easily be undone by npm.

If you need to make a modification to a package your project depends on, the correct course of action would be to create your own fork of the package. If you do go this route and feel that your improvements would be useful to others, congratulations: you’re now involved in an open source project! You can submit your changes, and if they meet the project standards, they’ll be included in the official package. Contributing to existing packages and creating customized builds is beyond the scope of this book, but there is a vibrant community of developers out there to help you if you want to contribute to existing packages.

Two of the main purposes of the package.json file are to describe your project and to list its dependencies. Go ahead and look at your package.json file now. You should see something like this (the exact version numbers will probably be different, as these packages get updated often):

{
  "dependencies": {
    "express": "^4.16.4",
    "express-handlebars": "^3.0.0"
  }
}

Right now, our package.json file contains only information about dependencies. The caret (^) in front of the package versions indicates that any version that starts with the specified version number—up to the next major version number—will work. For example, this package.json indicates that any version of Express that starts with 4.0.0 will work, so 4.0.1 and 4.9.9 would both work, but 3.4.7 would not, nor would 5.0.0. This is the default version specificity when you use npm install, and is generally a pretty safe bet. The consequence of this approach is that if you want to move up to a newer version, you will have to edit the file to specify the new version. Generally, that’s a good thing because it prevents changes in dependencies from breaking your project without your knowing about it. Version numbers in npm are parsed by a component called semver (for “semantic versioning”). If you want more information about versioning in npm, consult the Semantic Versioning Specification and this article by Tamas Piros.

Since the package.json file lists all the dependencies, the node_modules directory is really a derived artifact. That is, if you were to delete it, all you would have to do to get the project working again would be to run npm install, which will re-create the directory and put all the necessary dependencies in it. It is for this reason that I recommend putting node_modules in your .gitignore file and not including it in source control. However, some people feel that your repository should contain everything necessary to run the project and prefer to keep node_modules in source control. I find that this is “noise” in the repository, and I prefer to omit it.

Project Metadata

The other purpose of the package.json file is to store project metadata, such as the name of the project, authors, license information, and so on. If you use npm init to initially create your package.json file, it will populate the file with the necessary fields for you, and you can update them at any time. If you intend to make your project available on npm or GitHub, this metadata becomes critical. If you would like more information about the fields in package.json, see the package.json documentation. The other important piece of metadata is the README.md file. This file can be a handy place to describe the overall architecture of the website, as well as any critical information that someone new to the project might need. It is in a text-based wiki format called Markdown. Refer to the Markdown documentation for more information.

Node Modules

As mentioned earlier, Node modules and npm packages are related but different concepts. Node modules, as the name implies, offer a mechanism for modularization and encapsulation. npm packages provide a standardized scheme for storing, versioning, and referencing projects (which are not restricted to modules). For example, we import Express itself as a module in our main application file:

const express = require('express')

require is a Node function for importing a module. By default, Node looks for modules in the directory node_modules (it should be no surprise, then, that there’s an express directory inside of node_modules). However, Node also provides a mechanism for creating your own modules (you should never create your own modules in the node_modules directory). In addition to modules installed into node_modules via a package manager, there are more than 30 “core modules” provided by Node, such as fs, http, os, and path. To see the whole list, see this illuminating Stack Overflow question and refer to the official Node documentation.

Let’s see how we can modularize the fortune cookie functionality we implemented in the previous chapter.

First let’s create a directory to store our modules. You can call it whatever you want, but lib (short for “library”) is a common choice. In that folder, create a file called fortune.js (ch04/lib/fortune.js in the companion repo):

const fortuneCookies = [
  "Conquer your fears or they will conquer you.",
  "Rivers need springs.",
  "Do not fear what you don't know.",
  "You will have a pleasant surprise.",
  "Whenever possible, keep it simple.",
]

exports.getFortune = () => {
  const idx = Math.floor(Math.random()*fortuneCookies.length)
  return fortuneCookies[idx]
}

The important thing to note here is the use of the global variable exports. If you want something to be visible outside of the module, you have to add it to exports. In this example, the function getFortune will be available from outside this module, but our array fortuneCookies will be completely hidden. This is a good thing: encapsulation allows for less error-prone and fragile code.

Now in meadowlark.js, we can remove the fortuneCookies array (though there would be no harm in leaving it; it can’t conflict in any way with the array of the same name defined in lib/fortune.js). It is traditional (but not required) to specify imports at the top of the file, so at the top of the meadowlark.js file, add the following line (ch04/meadowlark.js in the companion repo):

const fortune = require('./lib/fortune')

Note that we prefix our module name with ./. This signals to Node that it should not look for the module in the node_modules directory; if we omitted that prefix, this would fail.

Now in our route for the About page, we can utilize the getFortune method from our module:

app.get('/about', (req, res) => {
  res.render('about', { fortune: fortune.getFortune() } )
})

If you’re following along, let’s commit those changes:

git add -A git commit -m "Moved 'fortune cookie' into module."

You will find modules to be a powerful and easy way to encapsulate functionality, which will improve the overall design and maintainability of your project, as well as make testing easier. Refer to the official Node module documentation for more information.

Conclusion

Now that we’re armed with some more information about Git, npm, and modules, we’re ready to discuss how we can produce a better product by employing good quality assurance (QA) practices in our coding.

I encourage you to keep in mind the following lessons from this chapter:

• Version control makes the software development process safer and more predictable, and I encourage you to use it even for small projects; it builds good habits!

• Modularization is an important technique for managing the complexity of software. In addition to providing a rich ecosystem of modules others have developed through npm, you can package your own code in modules to better organize your project.

• Node modules (also called CJS) use a different syntax than ECMAScript modules (ESM), and you may have to switch between the two syntaxes when you go between frontend and backend code. It’s a good idea to be familiar with both.