Do KISS and DRY

For every complex problem there is an answer that is clear, simple, and wrong.

Henry Louis Mencken,
American journalist

KISS is an acronym that stands for “Keep It Stupid Simple” or “Keep It Simple, Stupid,” or even the more polite “Keep It Simple and Straightforward.” KISS describes a design principle according to which simplicity should be a key goal, and unnecessary complexity should be avoided. KISS is a mantra that is sometimes repeated over and over and can be easily abused.

Although keeping any design neat, clean, and uncomplicated is usually a good thing, you will sooner or later face a case in which a software project or IT system carries an inherent complexity that cannot be eliminated and inevitably will be carried over from the initial requirements to the final result.

Also, any API grows in size and complexity as the number of use cases grows, and maintaining a clear interface that users can feel comfortable with over time is certainly a challenge. However, this does not mean that size and complexity are fundamentally related and it will not be possible to scale the methods and actions of a large or complex API.

Therefore, the KISS concept could be described as stating that a good API is easy to read, to use, and to extend, and is complete and consistent.

Another acronym often encountered in software development is “Don’t Repeat Yourself” (DRY). The DRY principle was formulated by Andy Hunt and Dave Thomas in their book The Pragmatic Programmer, and states that “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.”

Rails provides a set of tools to “DRY up your code” and avoid duplication, including partials and filters. Partials are used in views and filters in controllers.

From a REST architectural point of view, KISS and DRY translate at the API level to the constraints of  uniformity, modularity, and extendability. Both can be summarized into two fundamental principles of API design in particular and system development in general:

• Do simple syntax.
• Do simple semantics.

$ curl api.example.com/v1/products

$ curl api.example.com/v1/products/21345 

$ curl api.example.com/v1/products?type=sneakers&brand=sneak

Preparing the Database

Free copies of all available content on Wikipedia are available for users and multi-licensed under the Creative Commons Attribution-ShareAlike 3.0 License (CC-BY-SA) and the GNU Free Documentation License (GFDL).

Snapshots of Wikipedia categories and Wikipedia category links pre-prepared to be imported into this application can be found at the WikiCat GitHub repository.

Because the English category links table contains a large number of records, an extract of the whole dump has also been prepared for testing purposes, so that you do not have to wait several hours for the dump to be imported into your development database.

Wikipedia uses MySQL for its database, so although the dumps can be translated to another SQL dialect or NoSQL language, we are going to use MySQL for the WikiCat API.

The first step is making sure that MySQL is installed on your system. Run:

$ mysql --version
mysql Ver 14.14 Distrib 5.6.20, for osx10.9 (x86_64) using 
EditLine wrapper 

It may be the case that MySQL is not installed on your system. In this case, you will have to install it.

Mac OS X users can use brew and run:

$ brew install mysql

On Linux you will probably have to install the necessary libraries first:

$ apt-get install libmysqlclient18 libmysqlclient-dev

Then install MySQL Server:

$ apt-get install mysql-client-5.5 mysql-server-5.5

On Windows, you can use MySQL Installer. Instructions can be found in the MySQL documentation.

MySQL Installer simplifies the installation and updating process for a set of MySQL products. It allows the user to see which products have been installed, and configure, update, or remove them if needed. Through the installer it is also possible to install plug-ins, documentation, and tutorials; it includes a GUI and a command-line interface.

Once you’ve made sure MySQL is installed on your system and tested it is working with the command mysql --version, it is time to install the mysql gem for Rails, to allow our Rails app to interface with our database. Run:

$ gem install mysql

Now we are going to create a Rails app with MySQL support instead of SQLite3. Because we want to create an API-only application, we are going to use Rails::API. To install rails-api, run:

$ gem install rails-api

The application can then be generated by running:

$ rails-api new wikicat -d mysql

As before, the command will create our Rails application.

Now that our application has been created, we have to initialize the database:

$ cd wikicat
$ rake db:create

rake db:create creates our development and test databases. The command is run only the first time the database is created. Subsequently we will instruct our application to run the specified database migrations. We’ll discuss this in the next section.

Scaffolding the Models

Once our database has been prepared, we can start coding our application logic.

The categories table contains the following columns:

cat_id: <Integer>,
cat_title: <String>;,
cat_pages: <Integer>,
cat_subcats: <Integer>,
cat_files: <Integer>;

Each of these columns describes a property of a certain category. Let’s take a look at these in order:

cat_id

The unique category ID from Wikipedia

cat_title

The actual category title, as in “Sports” or “Science”

cat_pages

The number of pages belonging to that category

cat_subcats

The number of subcategories belonging to that category

cat_files

The number of files belonging to that category

We are going to generate the Category model accordingly:

$ rails generate model Category cat_id:integer cat_title:string 
  cat_pages:integer cat_subcats:integer cat_files:integer

The command will generate the Category model and a number of defaults:

invoke active_record
create db/migrate/20140821090455_create_categories.rb
create app/models/category.rb
invoke test_unit
create test/models/category_test.rb
create test/fixtures/categories.yml

The first file to be generated is the migration file to create the categories table. Migrations are a feature provided by Active Record to evolve the database schema over time, in a consistent and easy way.  The idea is that the developer doesn’t have to write schema modification code in SQL, but can use Ruby syntax to describe the changes he wants to apply to the database or a single table.

The database schema initially has nothing in it; consequently, each time a migration is run, tables, columns, or single entries are added to or removed from the database. Active Record will then update the schema accordingly, and modify the db/schema.rb file to match the up-to-date structure.

Migrations are stored in the db/migrate folder and sorted by time of creation. A timestamp is added to the filename.

We can now open our first migration file to check how it looks:

# db/migrate/<timestamp>_create_categories.rb
class CreateCategories > ActiveRecord::Migration
  def change
    create_table :categories do |t|
      t.integer :cat_id
      t.string :cat_title
      t.integer :cat_pages
      t.integer :cat_subcats
      t.integer :cat_files
      t.timestamps
    end
  end
end

If we want to add more columns to our tables, we can generate a migration file. Or, as we have not applied our migration yet, we can just edit our create_categories migration file.

What we want to do is use the Wikipedia category ID as the primary ID, without generating another ID field. Also, we want to add the created_at and updated_at columns. We will therefore modify the migration file:

# db/migrate/<timestamp>_create_categories.rb
class CreateCategories > ActiveRecord::Migration
  def change
    create_table :categories, {:id =&gt; false} do |t|
      t.integer :cat_id
      t.string :cat_title
      t.integer :cat_pages
      t.integer :cat_subcats
      t.integer :cat_files
      t.date :created_at
      t.date :updated_at
      t.timestamps
    end
    execute "ALTER TABLE categories ADD PRIMARY KEY (cat_id);"
  end
end

Now we are going to create the Link model. We again start with the Wikipedia SQL dump to see what the CategoryLinks table looks like:

cl_from: <Integer>,
cl_to: <String>,
cl_sortkey: <VarBinary>,
cl_timestamp: <Date>,
cl_sortkey_prefix: <VarBinary>
cl_collation: <VarBinary>
cl_type: <String>

Again, each of these columns describes a property of a certain category link:

cl_from

The page ID of the article where the link was placed.

cl_to

The name (excluding namespace prefix) of the desired category. This is the actual category title, as in “Sports” or “Science.”

cl_sortkey

The title by which the page should be sorted in a category list. For a subcategory this is equivalent to the category title.

cl_timestamp

The time at which that link was last updated in the table.

cl_sortkey_prefix

This is either the empty string if a page is using the default sortkey (aka the sortkey is unspecified), or it is the human-readable version of cl_sortkey.

cl_collation

What collation is in use.

cl_type

The type of link: page/subcategory/file.

Starting from the SQL dump schema of the categorylinks table, we are going to generate the Link model:

$ rails generate model Link          
            cl_from:integer          
            cl_to:string             
            cl_sortkey:binary        
            cl_timestamp:date        
            cl_sortkey_prefix:binary 
            cl_collation:binary      
            cl_type:string 

Let’s take a look at our new migration file:

# db/migrate/<timestamp>_create_links.rb

class CreateLinks > ActiveRecord::Migration
  def change
    create_table :links do |t|
      t.integer :id
      t.integer :cl_from
      t.string :cl_to
      t.binary :cl_sortkey
      t.date :cl_timestamp
      t.binary :cl_sortkey_prefix
      t.binary :cl_collation
      t.string :cl_type
      t.timestamps
    end
  end
end

Again, we want to slightly modify the create_links migration file before applying it. We will edit the file as follows:

# db/migrate/<timestamp>_create_links.rb
class CreateLinks > ActiveRecord::Migration
  def change
    create_table :links, {:id =&gt; false} do |t|
      t.integer :cl_from
      t.string :cl_to
      t.binary :cl_sortkey
      t.date :cl_timestamp
      t.binary :cl_sortkey_prefix
      t.binary :cl_collation
      t.string :cl_type
      t.date :created_at
      t.date :updated_at
      t.timestamps
    end
    execute "ALTER TABLE links ADD PRIMARY KEY (cl_from);"
  end
end

We can now run the migrations:

$ rake db:migrate

At this point we have two choices. The first one would be to import the two SQL dumps from Wikipedia. This will take a while, and you probably just want to go ahead with coding the WikiCat API at this point.

If you want to go ahead with the MySQL dumps, though, you can download the preprocessed zipped files.

Then unzip the file and run:

$ mysqld &
$ mysql -u root -p wikicat_development < 
  ~/Downloads/enwiki-latest-category.sql
$ mysql -u root -p wikicat_development < 
  ~/Downloads/enwiki-latest-categorylinks.sql

Alternatively, if you don’t want to wait several hours to have the whole Wikipedia dumps imported into your database, you can always populate it with seeds.rb.

To populate the database this way, run the command:

$ rake db:seed

Coding Controllers and Serializers

Once our database has been set up and our models generated, we can start coding our controllers.

We are going to use ActiveModel::Serializers to encapsulate the JSON serialization of objects.

A common way to generate a JSON API in Rails is to add a respond_to block to the controller action so that we can see the JSON representation of the returned object.

Suppose, though, we would like to have total control over the returned JSON. We could do this by overriding the as_json method, or by passing some options through the controller.

The problem with both these approaches is that they will not provide the level of customization and simplicity that ActiveModel::Serializers delivers. So, we are going to use it in our application by adding it to the Gemfile:

source 'https://rubygems.org'
# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '4.1.4'
gem 'rails-api'
# Use mysql as the database for Active Record
gem 'mysql2'
# Serializer for JSON
gem 'active_model_serializers'
group :doc do
  # bundle exec rake doc:rails generates the API under doc/api.
  gem 'sdoc', '~> 0.4.0'
end
group :development do
  # Spring speeds up development by keeping your application 
  # running in the background. Read more:
  # https://github.com/rails/spring
  gem 'spring'
end

and running Bundle to install it:

$ bundle install

We are now going to generate the serializer for our models by using the Serializers generator:

$ rails g serializer category
$ rails g serializer link

The generator will create a file in the folder serializers/ under app/. Here we can specify which attributes we want to serialize in our JSON replies.

For our category serializer we would like to include the category title and the number of subcategories. We are therefore going to specify the attributes cat_title and cat_subcats:

# serializers/category_serializer.rb
class CategorySerializer < ActiveModel::Serializer
  attributes :title, :sub_categories
end

We will now start coding our controllers. This is a good point to think about how we would like our API to be served and what URI structure we are going to use to identify our resources.

From a Rails perspective we are going to provide an api/ directory as our endpoint, followed by the API version, v1.

We are going to generate a category controller in the api/v1 namespace with one action:

$ rails generate controller api/v1/category show

This will generate an empty controller with a single empty action called show. We can modify the controller to return the category requested through a parameter:

class Api::V1::CategoryController < ApplicationController
  respond_to :json
  def show
    category = params[:category] ? params[:category] : "sports"
    @category = Category.where(:cat_title =&gt; 
                                category.capitalize).first
    render :json =&gt; @category, 
    serializer: CategorySerializer, 
    root: "category"
  end
end

Let us examine the controller bit by bit (not literally :).

First of all, we are instructing Rails to return only JSON by using the line respond_to :json.

Secondly, we are telling the controller that if the category param is not defined, the default category “sports” can be used.

Finally, we want the controller to return a category using the category serializer. We are also telling the serializer to use the string “category” as the root for the returned JSON.

Once the controller has been defined, we can define our routes. By default Rails fills the config/routes.rb file with some defaults and basic documentation on how to build routes for your application. Every time a controller is generated through the rails generate command, the routes file is modified with the default routes.

Therefore, we will go ahead and delete (or comment out, if you wish) all the defaults added to the routes file and replace them with the following:

Rails.application.routes.draw do
  namespace :api do
    namespace :v1 do
      get '/category/:category', :to => 'category#show'
    end
  end
end

The routes defined will instruct Rails to route the show action of the category controller within the api/v1 namespace. Furthermore, the portion of the path defined after /category/ will be considered as the category parameter to pass the controller.
Finally, it is time to test our serializer, controller, and routes:

$ curl http://0.0.0.0:3000/api/v1/category/science

If everything is going according to plan, we should receive the JSON string:

<p>{"category":{"cat_title":"SCIENCE","cat_subcats":34}}

We would like, though, to use different names in our JSON replies, and to add some text preprocessing.

We modify the serializer to force ISO-8859-1 and UTF-8 conversion and to replace any possible spaces left in the category titles with underscore characters.

Furthermore, we modify the attributes named cat_title and cat_subcats:

# serializers/category_serializer.rb
class CategorySerializer < ActiveModel::Serializer
  attributes :title, :sub_categories
  def title
    URI::encode(object.cat_title.force_encoding("ISO-8859-1")
        .encode("utf-8", replace: nil).downcase.tr(" ", "_"))
  end
  def sub_categories
    object.cat_subcats
  end
end

We can test again to see if the changes have been applied:

$ curl http://0.0.0.0:3000/api/v1/category/science
{"category":{"title":"science","sub_categories":34}}

Now that we can check single categories, we would like to display all subcategories of a given category passed as a parameter.

We are going to create a controller that will use the link serializer and build the category graph:

$ rails generate controller api/v1/graph show

Then we are going to modify it as follows:

class Api::V1::GraphController < ApplicationController
  respond_to :json
  def show
    cat = params[:category] || 'sports'
    @category = Category.where(:cat_title => cat.capitalize).first
    @links = Link.where(:cl_to => @category.cat_title, 
                        :cl_type => "subcat")
    render :json => @links, 
    each_serializer: LinkSerializer, 
    root: @category.cat_title.downcase
  end
  private
  def graph_params
    params.require(:category).permit(:cl_to, :cl_type)
  end
end

As for the category controller, we are asking Rails to respond only with JSON. Then we define an index action where we retrieve a category and find all the corresponding subcategories. We render the JSON using the link serializer defined earlier. The category title is our root.

We also define a private function where we specify the permitted attributes for the Link model and the required category parameter.

Next, we modify the link serializer to rename the cl_sortkey attribute to sub_category. We also enforce UTF-8 and ISO-8859-1 encoding while removing whitespaces:

class LinkSerializer < ActiveModel::Serializer
  attributes :sub_category
  def sub_category
    URI::encode(object.cl_sortkey.force_encoding("ISO-8859-1")
        .encode("utf-8", replace: nil).downcase.tr(" ", "_"))
  end
end

Then we test:

$ curl http://0.0.0.0:3000/api/v1/graph/science

And if everything was set up correctly, we receive the graph JSON:

{
    "science": [{
        "sub_category": "_%0Ascientific_disciplines"
    }, {
        "sub_category": "_%0Ascientists"
    }, ...]
}

Testing

Now we are at the boring and tedious step of setting up an app: testing.

We want to make sure that our controllers will behave according to our design even if something changes in the future. So, we are going to write two simple tests to assure the default behavior for both controllers:

# test/controllers/api/v1/category_controller_test.rb
require 'test_helper'
class Api::V1::CategoryControllerTest < ActionController::TestCase
  test "should get show" do
    get :show, {'category' => 'science'}
    assert_response :success
    assert_equal "{"category":
                    {"title":"science",
                    "sub_categories":1}}", 
                 @response.body
  end
end

require 'test_helper'
class Api::V1::GraphControllerTest < ActionController::TestCase
  test "should get show" do
    get :show, {'category' => "sports"}
    assert_response :success 
    assert_equal "{"sports":
                    [{"sub_category":
                        "culture_and_sports_culture"}]}", 
                 @response.body
  end
end

Before running the tests we also need to prepare the fixtures. This is fake data that will populate our test database in order for the tests to work.

We are going to create two links and two categories. The two fixtures were already generated when we scaffolded the models, but we are going to modify both as follows:

# test/fixtures/links.yml
one:
  cl_from: 1
  cl_to: Sports
  cl_sortkey: CULTURE AND SPORTS CULTURE
  cl_timestamp: 2005-04-29 10:32:42
  cl_sortkey_prefix: Culture
  cl_collation: uppercase
  cl_type: subcat
two:
  cl_from: 2
  cl_to: Science
  cl_sortkey: SCIENTIFIC DISCIPLINES
  cl_timestamp: 2013-09-02 22:56:59
  cl_sortkey_prefix: Sci
  cl_collation: uppercase
  cl_type: subcat

# test/fixtures/categories.yml
one:
  cat_id: 1
  cat_title: Sports
  cat_pages: 1
  cat_subcats: 1
  cat_files: 1
two:
  cat_id: 2
  cat_title: Science
  cat_pages: 1
  cat_subcats: 1
  cat_files: 1

We can then run the tests as usual:

$ rake test