In the previous chapter, we learned how to build a Todo MVC web app using Genie. This approach uses a JS web frontend for our users. But what if we want to make our data, which is stored in a database, available to other kinds of graphical frontends, such as a mobile app or a desktop application? We don’t want to rewrite everything from scratch for each new frontend. Besides, other companies may want to write these frontends to our data.
In this chapter, we’ll add a REST API to our existing app by organizing our code so as to allow other applications and software to access and interact with our data. Here are the topics we are going to discuss in this chapter:
The source code for this chapter can be found at https://github.com/PacktPublishing/Web-Development-with-Julia-and-Genie/tree/main/Chapter5. Please make sure you use Genie v5.11 or higher.
When writing this chapter, SwagUI v0.9.1 and Swagger Markdown v0.2.0 were used.
Just like the regular Julia libraries, REST APIs are meant to be used by developers in order to build applications that are interoperable. Let’s first lay some groundwork.
Each application changes, and it is common to introduce changes in our data model or API, such as different endpoints, different HTTP methods, different request parameters, different responses and response structures, and so on. These changes could cause other apps that access our data via the REST API to crash. That’s why we need versioning.
By introducing versioning in our API, we allow consumer apps to continue working while providing their developers the time to update to the latest version.
There are a few ways to version REST APIs, and we’ll use what’s arguably the most popular approach: versioning through the URI path. This means that we’ll place the major version of our API into the URL of the requests, as in the following link: mytodos.com/api/v1/todos. This is the preferred approach for Genie apps as it’s easy to use and understand. At the same time, this approach promotes a modular architecture, as we’re about to see in the next section.
Taking into account the versioning requirements, our API requests will be prefixed with /api/v1, indicating that the current major version of the API is v1. In the future, if we introduce breaking changes into our API, we’ll need to introduce a new major version of the API (v2, v3, and so on).
Under the hood, each part of the URI will be implemented into a distinct Julia module, making our code modular and composable for easier maintenance and extensibility. By encapsulating the API logic as well as the specific version into dedicated modules and submodules, we make our code future-proof and easy to maintain.
The REST API will expose similar endpoints as the web application itself. We want to allow the consumers of our API to create, retrieve, update, and delete todos. The only difference is that we will not include a dedicated toggle endpoint because REST APIs by convention have a different update mechanism. That being said, let’s define our routes by adding the following code at the end of our routes.jl file:
route("/api/v1/todos", TodosController.API.V1.list, method = GET) route("/api/v1/todos/:id::Int", TodosController.API.V1.item, method = GET) route("/api/v1/todos", TodosController.API.V1.create, method = POST) route("/api/v1/todos/:id::Int", TodosController.API.V1.update, method = PATCH) route("/api/v1/todos/:id::Int", TodosController.API.V1.delete, method = DELETE)
The code we just added defines five routes to handle the main four operations of the API: listing, creating, updating, and deleting todos. REST API best practices prescribe that we use dedicated HTTP methods for each of these operations. In addition, we’ll also use JSON to handle both requests and responses. Finally, to be good citizens of the web, we’ll also extend our current set of features to add support for pagination to the list of todos.
Let’s begin with the first operation: the retrieval and listing of to-do items. Add this at the bottom of the TodosController.jl file, right above the closing end:
### API module API module V1 using TodoMVC.Todos using Genie.Router using Genie.Renderers.Json using ....TodosController function list() all(Todo) |> json end function item() end function create() end function update() end function delete() end end # V1 end # API
In the preceding snippet, we define a new module called API and a submodule, V1. Inside V1, we declare references to various dependencies with using statements. More importantly, we bring into scope Genie.Renderers.Json, which will do all the heavy lifting for building JSON responses for our API. You can think of it as the counterpart of Genie.Renderers.Html, which we used in TodosController to generate HTML responses. And just like in the main controller, we’ll leverage the features in Genie.Router to handle the requests data. We have also included a reference to the main TodosController module, using a relative namespace two levels up (notice the four dots, ....).
Finally, we define placeholder functions for each of the operations, matching the handlers we defined in our routes file. The list function even includes a bit of logic, to allow us to check that everything is set up well. Try it out at http://localhost:8000/api/v1/todos, and you should get a JSON response with the list of todos.
Here is our output:
Figure 5.1 – List of todos as JSON response
Now it’s time to refine our todos listing. It’s important to keep our code DRY and reuse as much logic as possible between the HTML rendering in TodosController.jl and our API that outputs JSON. Right now, TodosController.index includes both the filtering and retrieval of the to-do items, as well as the rendering of the HTML. The filtering and retrieval operations can be reused by the API, so we should decouple them from the HTML rendering.
Replace the TodosController.index function with the following code:
function count_todos() notdonetodos = count(Todo, completed = false) donetodos = count(Todo, completed = true) ( notdonetodos = notdonetodos, donetodos = donetodos, alltodos = notdonetodos + donetodos ) end function todos() todos = if params(:filter, "") == "done" find(Todo, completed = true) elseif params(:filter, "") == "notdone" find(Todo, completed = false) else all(Todo) end end function index() html(:todos, :index; todos = todos(), count_todos()..., ViewHelper.active) end
Here, we refactor the index function to only handle the HTML rendering, while we prepare the data in two new functions: count_todos and todos. We’ll reuse these functions to prepare the JSON response for our API. It’s worth noticing the flexibility of Julia in the way we pass the keyword arguments to the html function inside index(): we explicitly pass the todos keyword argument with the todos() value, and we splat the NamedTuple received from count_todos() into three keyword arguments, and finally, we pass the ViewHelper.active value as the last implicit keyword argument.
Next, we can use these newly created functions in our API.V1.list function, to retrieve the actual data:
function list() TodosController.todos() |> json end
We can check that our refactoring hasn’t broken anything by checking some URLs for both the app and API:
We haven’t forgotten about our integration tests and you’re welcome to run those as well, but they will be more useful once we add tests for the API too.
I’m hoping that our to-do list will not be that long as to actually need pagination, but pagination is a common and very useful feature of REST APIs and it’s worth seeing how it can be implemented, especially as SearchLight makes it very straightforward. We want to accept two new optional query parameters, page and limit, to allow the consumer to paginate the list of todos. The page parameter will indicate the number of the page (starting with 1) and limit will indicate the number of todos per page.
As mentioned, the implementation is extremely simple; we only need to pass the extra arguments, with some reasonable defaults, to the SearchLight.all function that we use to get the todos. Update the else branch in the TodosController.todos function as follows:
function todos() todos = if params(:filter, "") == "done" find(Todo, completed = true) elseif params(:filter, "") == "notdone" find(Todo, completed = false) else # this line was: all(Todo) and is changed to: all(Todo; limit = params(:limit, SearchLight.SQLLimit_ALL) |> SQLLimit, offset = (parse(Int, params(:page, "1"))-1) * parse(Int, params(:limit, "0"))) end end
All we need to do is pass the limit and offset arguments to the all function, and we’re done. Given that these are optional (the users can make requests without pagination), we also set some good defaults: if there is no limit argument, we include all the todos by passing the SearchLight.SQLLimit_ALL constant to the limit argument. As for offset, this indicates how many items to skip, which are calculated by multiplying the page number by the number of items per page. If there is no page argument, we start with the first page by using 1 as the default, but do note that when we calculate the offset, we use page – 1; this way, on page 1 the offset is 0. This is because the offset argument in the database query represents the number of todos to skip, and for page 1, we want to skip 0 todos. As for limit, the default here is 0 (meaning that if no limit argument is passed, we’ll include all the todos with offset of 0).
We can test the new functionality by getting a couple of pages and limiting the number of todos per page to one:
Also, no pagination will return all the todos, as expected (http://localhost:8000/api/v1/todos).
Our web app does not support pagination yet. We’ll skip it, but if you want, you can do it as an exercise: add a new element to the UI to allow the users to navigate between pages of todos and set the limit to a reasonable constant value, such as 20.
We want to allow the consumer of our API to add new todos. We already have the route and the corresponding route handler, so it’s now time to add the actual code. Add the following code snippets in this section while changing the API.V1.create function, as follows:
using Genie.Requests using SearchLight.Validation using SearchLight
In the preceding code, we first declare that we’ll be using three extra modules. Genie.Requests provides a higher-level API to handle requests data, and we’ll rely on it to help us work with the JSON payloads. The other is SearchLight.Validation, which we’ve already seen in action and which helps us to validate the data we receive from the consumer of the API. SearchLight itself gives us access to the save! method:
function check_payload(payload = Requests.jsonpayload()) isnothing(payload) && throw(JSONException(status = BAD_REQUEST, message = "Invalid JSON message received")) payload end
Given that we expect a JSON payload, in the preceding check_payload, we verify whether the body of the request can be converted to JSON. We use the Requests.jsonpayload function to do that. If the payload is not valid JSON, the Requests.jsonpayload function will return nothing. In this case, we throw an exception, informing the user that the message received is not valid JSON.
Now we get to the create function:
function create() payload = try check_payload() catch ex return json(ex) end todo = Todo(todo = get(payload, "todo", ""), completed = get(payload, "completed", false)) persist(todo) end
Here, once we are sure that we have received a valid JSON payload, we parse it, looking for relevant data to create a new to-do item. We provide some good defaults and create a new instance of our Todo model, using the provided payload. We then attempt to persist the newly created model to the database, by passing it to the persist function, where we apply our model validations:
function persist(todo) validator = validate(todo) if haserrors(validator) return JSONException(status = BAD_REQUEST, message = errors_to_string(validator)) |> json end try save!(todo) json(todo, status = CREATED, headers = Dict("Location" => "/api/v1/todos/$(todo.id)")) catch ex JSONException(status = INTERNAL_ERROR, message = string(ex)) |> json end end
In the create function, we first check whether the request payload is valid by invoking the check_payload function.
If our persist function finds any validation errors, we return an exception with the error details. If the validations pass, we save the todo in the database and return a JSON response with the newly created todo and the CREATED status code. As a best practice, we also pass an additional location header, which is the URL of the newly created todo. If for some reason the todo could not be saved, we return an exception with the error details.
We can test the various scenarios using an HTTP client such as Postman (for an example of how to use Postman, see Chapter 2, Testing the ToDo Services with Postman) or Paw. But we’ll skip that for now and just add integration tests in the final section of this chapter.
Updating todos is a breeze, especially as we’ve already implemented our validation logic. First, we need to change the try/catch statement in the persist function to the following:
try if ispersisted(todo) save!(todo) json(todo, status = OK) else save!(todo) json(todo, status = CREATED, headers = Dict("Location" => "/api/v1/todos/$(todo.id)")) end catch ex JSONException(status = INTERNAL_ERROR, message = string(ex)) |> json end
This change allows us to detect whether the todo we’re attempting to save was already persisted to the database or not. If it was, we’ll update the todo in the database, otherwise, we’ll create a new one, and we need to return the correct response, based on the database operation we performed.
Now, fill in our empty API.V1.update function as follows:
function update() payload = try check_payload() catch ex return json(ex) end todo = findone(Todo, id = params(:id)) if todo === nothing return JSONException(status = NOT_FOUND, message = "Todo not found") |> json end todo.todo = get(payload, "todo", todo.todo) todo.completed = get(payload, "completed", todo.completed) persist(todo) end
We start by checking whether the payload is valid. If it is, we continue by retrieving the corresponding todo from the database, using the id passed as part of the URL. If the todo is not found, we return an exception. Otherwise, we update the todo with the provided data again, applying some good defaults (in this case, keeping the existing value if a new value was not provided). Finally, we attempt to persist the todo in the database.
The last operation that our API should support is the deletion of the to-do items. We’ll update the API.V1.delete function as follows:
function delete() todo = findone(Todo, id = params(:id)) if todo === nothing return JSONException(status = NOT_FOUND, message = "Todo not found") |> json end try SearchLight.delete(todo) |> json catch ex JSONException(status = INTERNAL_ERROR, message = string(ex)) |> json end end
The code attempts to retrieve the todo from the database, based on the id passed as part of the URL. If the todo is not found, we return an exception. Otherwise, we delete the todo from the database and return it.
For retrieving individual to-do items from the database, we only need to check that the corresponding to-do item exists by looking it up by id. If it does not exist, we return a 404 error. If it does, we return the todo. Here is the code for the item function:
function item() todo = findone(Todo, id = params(:id)) if todo === nothing return JSONException(status = NOT_FOUND, message = "Todo not found") |> json end todo |> json end
Note
The complete code listing for the tests in this section can be found at https://github.com/PacktPublishing/Web-Development-with-Julia-and-Genie/blob/main/Chapter5/TodoMVC/test/todos_API_test.jl.
It’s time to see our API in action by writing a test suite to check all the endpoints and the various scenarios we’ve implemented. Let’s start by adding a new test file for our API:
julia> touch("test/todos_API_test.jl")
Next, we’ll add the test suite to our newly created file:
using Test, SearchLight, Main.UserApp, Main.UserApp.Todos using Genie import Genie.HTTPUtils.HTTP import Genie.Renderers.Json.JSONParser.JSON3 try SearchLight.Migrations.init() catch end cd("..") SearchLight.Migrations.all_up!!() Genie.up() const API_URL = "http://localhost:8000/api/v1/todos" @testset "TodoMVC REST API tests" begin @testset "No todos by default" begin response = HTTP.get(API_URL) @test response.status == Genie.Router.OK @test isempty(JSON3.read(String(response.body))) == true end end Genie.down() SearchLight.Migrations.all_down!!(confirm = false) cd(@__DIR__)
Besides the declaration of the used dependencies, the first and the last parts of the file are the setup and teardown of the tests, just like we did in the integration tests in Chapter 4, Building an MVC ToDo App, in the Testing Genie apps section. This is where we set up the test database and the API server while at the end, we remove the test data and stop the web server.
All our tests will be placed inside a main testset called "TodoMVC REST API tests". And our first test simply checks that when initiating our test suite, our database does not contain any todos. We make a GET request to our /todos endpoint that lists the to-do items, and we verify that the response is a 200 OK status code and that the response body is empty.
Next, let’s add tests for to-do creation. These will verify all the assumptions related to to-do creation. Append these code snippets under the "No todos by default" testset:
@testset "Todo creation" begin @testset "Incorrect content-type should fail todo creation" begin response = HTTP.post(API_URL, ["Content-Type" => "text/plain"], JSON3.write(Dict("todo" => "Buy milk")); status_exception = false) @test response.status == Genie.Router.BAD_REQUEST @test JSON3.read(String(response.body)) == "Invalid JSON message received" end
This first test verifies that when we send a request with an incorrect content type, the response has a 400 BAD_REQUEST status code and that the response body equals the "Invalid JSON message received" error message:
@testset "Invalid JSON should fail todo creation" begin response = HTTP.post(API_URL, ["Content-Type" => "application/json"], "Surrender your data!"; status_exception = false) @test response.status == Genie.Router.BAD_REQUEST @test JSON3.read(String(response.body)) == "Invalid JSON message received" end
The second test checks that when we send a request with an invalid JSON payload, the API responds in the same manner, with a BAD_REQUEST status and the same error message:
@testset "Valid JSON with invalid data should fail todo creation" begin response = HTTP.post(API_URL, ["Content-Type" => "application/json"], JSON3.write(Dict("todo" => "", "completed" => true)); status_exception = false) @test response.status == Genie.Router.BAD_REQUEST @test JSON3.read(String(response.body)) == "Todo should not be empty" end
The third test checks that despite the valid content type and JSON payload, if the todo data is not valid, the request will fail with a BAD_REQUEST status and the "Todo should not be empty" error message.
Moving on to the next API test:
@testset "No todos should've been created so far" begin response = HTTP.get(API_URL) @test response.status == Genie.Router.OK @test isempty(JSON3.read(String(response.body))) == true end
The fourth test makes an extra check that because of the previous error responses, no todos have been created up to this point:
@testset "Valid payload should create todo" begin response = HTTP.post(API_URL, ["Content-Type" => "application/json"], JSON3.write(Dict("todo" => "Buy milk"))) @test response.status == Genie.Router.CREATED @test Dict(response.headers)["Location"] == "/api/v1/todos/1" @test JSON3.read(String(response.body))["todo"] == "Buy milk" end @testset "One todo should be created" begin response = HTTP.get(API_URL) @test response.status == Genie.Router.OK todos = JSON3.read(String(response.body)) @test isempty(todos) == false @test length(todos) == 1 @test todos[1]["todo"] == "Buy milk" response = HTTP.get("$API_URL/1") @test response.status == Genie.Router.OK todo = JSON3.read(String(response.body)) @test todo["todo"] == "Buy milk" end end # "Todo creation"
Finally, the last two tests confirm that when we send a valid payload, the API successfully creates a new todo, returns it with a 201 Created status code, and the location header is set to the new todo’s URL and that we can retrieve it.
Next, for the todo updating tests, use the following code (see the link for the complete code at the start of this section):
@testset "Todo updating" begin @testset "Incorrect content-type should fail todo update" begin response = HTTP.patch("$API_URL/1", ["Content-Type" => "text/plain"], JSON3.write(Dict("todo" => "Buy soy milk")); status_exception = false) ... end
The following tests are exactly the same as the tests for creating todos, except that the request is HTTP.patch("$API_URL/1",...) instead of HTTP.post(API_URL,…), because PATCH is needed for an update:
@testset "Invalid JSON should fail todo update" @testset "Valid JSON with invalid data should fail todo update"
The code for the following testset is the same as for the "One todo should be created" testset:
@testset "One existing todo should be unchanged"
The following testsets have almost the same code as their counterparts from the create todos testset:
@testset "Valid payload should update todo" @testset "One existing todo should be changed"
The following test is new:
@testset "Updating a non existing todo should fail" begin response = HTTP.patch("$API_URL/100", ["Content-Type" => "application/json"], JSON3.write(Dict("todo" => "Buy apples")); status_exception = false) @test response.status == Genie.Router.NOT_FOUND @test JSON3.read(String(response.body)) == "Todo not found" end end # "Todo updating"
These tests follow the logic of the todo creation testset, just adapted to the todo updating scenario, so we won’t get into details about these.
Now, let’s add the todo deletion tests (for the complete code, see the repository linked at the beginning of this section).
The most significant change is that the request is now of the HTTP.delete("$API_URL/1") form:
@testset "Todo deletion" begin @testset "Deleting a non existing todo should fail" begin response = HTTP.delete("$API_URL/100", ["Content-Type" => "application/json"]; status_exception = false) @test response.status == Genie.Router.NOT_FOUND @test JSON3.read(String(response.body)) == "Todo not found" end
The logic should be pretty clear by now. The first test checks that when we try to delete a non-existing todo, the API responds with a NOT_FOUND status and the "Todo not found" error message:
@testset "One existing todo should be deleted" begin response = HTTP.delete("$API_URL/1") @test response.status == Genie.Router.OK @test JSON3.read(String(response.body))["todo"] == "Buy vegan milk" @test HTTP.get("$API_URL/1"; status_exception = false).status == Genie.Router.NOT_FOUND end
The second test checks that when we delete an existing todo, the API responds with an OK status and the todo data.
At this point there should be no todos left in the database:
@testset "No todos should've been left" begin response = HTTP.get(API_URL) @test response.status == Genie.Router.OK @test isempty(JSON3.read(String(response.body))) == true end end # "Todo deletion"
The last test in this testset makes sure that no todos are left in the database.
And finally, to complete our test suite, we’ll add the pagination tests:
@testset "Todos pagination" begin todo_list = [ Dict("todo" => "Buy milk", "completed" => false), Dict("todo" => "Buy apples", "completed" => false), Dict("todo" => "Buy vegan milk", "completed" => true), Dict("todo" => "Buy vegan apples", "completed" => true), ] for todo in todo_list response = HTTP.post(API_URL, ["Content-Type" => "application/json"], JSON3.write(todo)) end
The preceding code is a bit more involved. First, we create a to-do list, which comprises some fake data to mock our tests. Next, we iterate over this list and use the API itself to create all the todos. Once our data is in, it’s time for the actual tests:
@testset "No pagination should return all todos" begin response = HTTP.get(API_URL) @test response.status == Genie.Router.OK todos = JSON3.read(String(response.body)) @test isempty(todos) == false @test length(todos) == length(todo_list) end
The first test checks that when we don’t specify any pagination parameters, the API returns all todos.
Now, we test the outputted data with various pagination scenarios, making sure that the data is split correctly between the various pages, according to the limit parameter:
@testset "One per page" begin index = 1 for page in 1:length(todo_list) response = HTTP.get("$API_URL?page=$(page)&limit=1") todos = JSON3.read(String(response.body)) @test length(todos) == 1 @test todos[1]["todo"] == todo_list[index]["todo"] index += 1 end end
For the last four testsets, see the code in the repository:
@testset "Two per page" @testset "Three per page" @testset "Four per page" @testset "Five per page" end # "Todos pagination"
Now, to verify these tests, go to your command-line interface, and within the test folder, issue the following command:
julia --project runtests.jl todos_API_test
You’ll see something like the following on your display:
todos_API_test: .............................................................................. Test Summary: | Pass Total Time TodoMVC tests | 78 78 23.6s
Now verify the complete testset as we have done in Chapter 4, in the Testing Genie apps section. You should have 98 tests that pass.
Having built our REST API and tested it, it is very important to document the API, because you can have a potentially unknown number of customers who will want to use it. Swagger (https://swagger.io/) is a very useful tool to help you design and document your APIs. Swagger UI employs the OpenAPI standard and allows us to document our API in code, and at the same time, to publish it via a web-based human-readable interface.
In order to add support for Swagger UI, we need to add two new packages to our project, SwagUI and SwaggerMarkdown:
pkg> add SwagUI, SwaggerMarkdown
We will set up the Swagger comments and the API documentation functionality in the routes.jl file. The routes for the web application remain the same, but the API routes are now augmented with swagger"..." annotations that are used to build the API documentation. Add using SwagUI, SwaggerMarkdown to the routes.jl file, and update the REST API routes as follows:
Note
Because it is quite lengthy, we are only showing the complete docs code for the GET request of the /apI/v1/todos route in this section. You can find the complete code at https://github.com/PacktPublishing/Web-Development-with-Julia-and-Genie/blob/main/Chapter5/TodoMVC/routes.jl.
swagger" /api/v1/todos: get: summary: Get todos description: Get the list of todos items with their status parameters: - in: query name: filter description: Todo completed filter with the values 'done' or 'notdone' schema: type: string example: 'done' - in: query name: page description: Page number used for paginating todo items schema: type: integer example: 2 - in: query name: limit description: Number of todo items to return per page schema: type: integer example: 10 responses: '200': description: A list of todos items post: " route("/api/v1/todos", TodosController.API.V1.list, method = GET) route("/api/v1/todos", TodosController.API.V1.create, method = POST) swagger" /api/v1/todos/{id}: get: patch: delete: " route("/api/v1/todos/:id::Int", TodosController.API.V1.item, method = GET) route("/api/v1/todos/:id::Int", TodosController.API.V1.update, method = PATCH) route("/api/v1/todos/:id::Int", TodosController.API.V1.delete, method = DELETE) ### Swagger UI route route("/api/v1/docs") do render_swagger( build( OpenAPI("3.0", Dict("title" => "TodoMVC API", "version" => "1.0.0")), ), options = Options( custom_favicon = "/favicon.ico", custom_site_title = "TodoMVC app with Genie", show_explorer = false ) ) end
In the preceding code, we have first grouped the routes by path, differentiating them by method. We have two distinct paths, /api/v1/todos and /api/v1/todos/:id. The first path accepts GET and POST requests to list and create todos, while the second path accepts GET, PATCH, and DELETE requests to retrieve, update, and delete a to-do item.
The Swagger documentation is built by annotating the individual paths and sub-differentiating them by method. Then, for each path and method combination, we detail the request and response information, including properties such as summary, description, requestBody, and responses.
In addition, at the end of the file, we now have a new route to render Swagger UI. This route invokes the render_swagger function, passing various configuration options to build the docs.
This was all! Restart your app by following the steps from the Pausing development subsection in Chapter 4. Our API is now documented and we can use Swagger UI to browse the API by accessing the /api/v1/docs route at http://localhost:8000/api/v1/docs. Not only that but the browser is fully interactive, allowing us to run queries against the API and see the results in real time.
Here is the main documentation screen:
Figure 5.2 – Swagger documentation screen
In this chapter, we added a REST API to our Todo application. You learned how to code the CRUD functions for this API and about the necessity of versioning your code. The REST API exposes your data to the world, that’s why testing and documenting are especially important. You learned how to write these specific tests and how to document the API using Swagger.
In the next chapter, we’ll discover the specific techniques to deploy our Genie app in production.
Another related and interesting topic is GraphQL, a new query language and runtime for APIs (see https://graphql.org/). There is a Genie example for GraphQL here: https://github.com/neomatrixcode/Diana.jl/tree/master/samples/genie.
18.222.196.175