Scaling is an important tenet of application design. Scaling not only provides application performance, but, if done right, scaling also provides application availability and fault tolerance. Developers must pay attention to effectively scaling the application. It can’t be a post-development thought. Previously, you learned that an application can be scaled vertically by allocating more resources to a running instance. Monolithic applications follow this principle. Chapter 1 explained how this is an ineffective approach.
Moreover, to provide availability, we often pick a hot-cold deployment pattern. This drives inefficiency because a cold instance is a standby instance. It is only activated when the primary application instance is down.
On the other hand, horizontal scaling allows you to run more than one instance of the application simultaneously. Each of these instances can run on the minimum required hardware and serve user requests. It is the preferred mechanism for deploying an application in cloud environments. It greatly improves application availability by using hot-hot deployments (see Figure 3-1).
Figure 3-1
Deployment types
Once there are multiple instances of an application, a load balancer must be configured to work with them effectively. A load balancer application like Traefik can run at Layer 4 and Layer 7 of the open systems interconnection (OSI) model. At Layer 4, it serves as a TCP/UDP proxy. It works on the base of host and port information. At Layer 7, the load balancer looks at many attributes (e.g., HTTP load balancing can be done based on host, port, request path, or request headers). Thus, Traefik can perform load balancing at both layers.
In Chapter 2, we configured HTTP services in Traefik. In this chapter, we configure load balancing of HTTP services. Traefik also provides TCP and UDP capabilities. We work with them, as well.
HTTP Load Balancer
To effectively use horizontal scaling, we need a load balancer. The load balancer is configured with all instances of the application. When the load balancer receives a request, it must delegate the request to one of the configured instances. Several load balancing algorithms can alter the behavior of request handling. Each algorithm has pros and cons and works better for some situations than others.
In the previous chapter, we configured Traefik using file type provider. We created an entrypoint for port 80. We also added routers and services to handle incoming requests. The services configuration points to the location of an application. The load balancing algorithm is also governed at the service level. Several services can use a particular algorithm, while others can use a different algorithm. When we look at the service configuration, it consists of the following blocks.
Service: Defines the logical grouping of servers so that common attributes can be applied
Server: Defines the actual location of the application
Either of the two blocks configures load balancing in Traefik. In the following section, we configure the different attributes to learn complete behavior.
Round Robin
A round robin (RR) is one of the simplest algorithms for load distribution. The algorithm delegates requests to each available instance in equal proportions (see Figure 3-2). It performs the operation in a circular manner without any notion of priority or preference. Round-robin load balancing works best when servers have roughly identical computing capabilities.
Figure 3-2
Request distribution
In Figure 3-2, there are four servers in the application. The graph depicts how the algorithm distributes incoming requests among them. Going further, we need an HTTP application for configuring the round robin. In the remaining sections, we work with a visitor log-keeping application. The application has the following behaviors.
Adds a guest name
Lists the latest guest name
Shows all guest names
The application is deployed on multiple boxes. Each application instance is given a name, which is shown in the UI (see Figure 3-3). The UI helps determine which instance serves the user request.
Figure 3-3
Visitor log screen
Traefik services configuration consists of a service and a server block. The round robin is configured by using a server block.
http :
routers :
guest-router :
entryPoints :
- web
rule : Host(`localhost`)
service : guestbook-service
services :
guestbook-service :
loadBalancer :
servers :
- url : "http://192.168.1.10:9090/"
- url : "http://192.168.1.11:9191/"
The following is configured in the preceding code.
Request routing is configured for the localhost domain. The rule matches all incoming requests. In the previous chapter, you saw the PATH rule, which validates request URL location. Here we are validating requests based on the hostname instead of the request path. Traefik uses the guestbook-service configuration to handle the request.
The server section lists the URLs of all available instances. It is configured as a list of values.
Let’s run the Traefik configuration and access http://localhost (see Figure 3-4) in the browser. Service configuration is also available on the Traefik dashboard (see Figure 3-5). It shows the complete status of a service.
Figure 3-4
Multiple instances of visitor
Figure 3-5
Traefik dashboard for round robin
If you refresh the browser a couple of times, you see that it is served from both instances. We can add a few entries to the application. This data is saved in the underlying database. The data is subsequently shown in both instances. In a nutshell, the application is not keeping any state. All states are persisted in the database. The classic round-robin algorithm is good enough when the complete application is stateless, like a visitor logbook.
On the other hand, the application can be stateful. This means each application has some data that is local to it. In web applications, the state is maintained using HTTP sessions. A session is created for every user. The session is an in-memory store. It remains associated with the user. There is no limit to what can be stored inside a session. Application developers can store user-centric data like id, latest transactions, and UI styling. When requests are routed from one instance to another, the session information is lost.
Sticky Session
Session stickiness ensures that all requests from the user during the session are sent to the same instance. The load balancer implements this by persisting cookies in the user request. The load balancer creates a cookie for the first user request. It then refers to that cookie for every subsequent request. In Traefik, the cookie is configured at the loadBalancer level.
# Removed for Brevity
services :
guestbook-service :
loadBalancer :
sticky :
cookie : {}
servers :
- url : "http://192.168.1.10:9090/"
- url : "http://192.168.1.9:9191/"
In the code, we added the sticky attribute for a defined guest-service loadBalancer. After the change, the requests can no longer toggle between the two application instances. It is served from only one instance. We can validate the instance details by looking-up cookie details in the browser (see Figure 3-6).
Figure 3-6
Browser cookie
The configuration added a cookie with a generated name. The sticky record provides the following optional attributes which can configure the behavior of the generated cookie.
Name: Specifies a cookie name instead of a generated one.
HttpOnly: The flag mitigates cookie access through client-side JavaScript.
Secure: The attribute sends a cookie over an HTTPS connection.
SameSite: The attribute restricts cookies within the same-site context. The context boundary is defined by the various values of the attribute.
The Traefik dashboard also shows the updated configuration (see Figure 3-7).
Figure 3-7
Traefik dashboard for sticky session
A cookie points to the server location that handled the original request. Traefik must be configured to monitor all these application servers. If the server specified in the cookie becomes unavailable, the request is forwarded to a new server. Traefik updates the cookie with details of the new server. This is achieved by configuring health checks for the instances.
Note
When using classic round robin routing Traefik keeps the unhealthy server until application health checks are configured.
Health Check
To work effectively, a load balancer should determine which backend servers are performing well. This is accomplished by sending requests to validate the instance status at periodic intervals. These requests are known as health checks. Traefik routes requests only to healthy instances. It keeps track of all active instances. Traefik drops an instance from the active instances pool when it determines the instance is unhealthy. It keeps monitoring unhealthy instances. Once the instance is restored, Traefik adds it back to the active instance pool. Only a response to the heath check request governs their status of the instance. Responses other than 2XX and 3XX are considered errors (see Figure 3-8).
Figure 3-8
Application errors
Traefik allows you to configure the server health using the health check attribute of the service.
services :
guestbook-service :
servers :
- url : "http://192.168.1.10:9090/"
- url : "http://192.168.1.11:9191/"
healthCheck:
path: /
interval: "10s"
timeout: "1s"
The following can be said about the preceding code.
The / path is configured for health status lookup.
The lookup is performed every 10 seconds. If the server changes its state, it is known after a maximum of 10 seconds.
Timeout configures the time interval for HTTP request-timeout
To test the configuration, you can either stop one of the servers or raise an error response (5XX,4XX) from the application. These health checks are also visible in the Traefik dashboard under the Services tab. (see Figure 3-9)
Figure 3-9
Traefik dashboard with application health
When working with sticky sessions, Traefik resets the cookie if the server becomes unable. The request is then routed to one of the healthy servers. Traefik updates the cookie with details of the new instance. All further requests are routed to the new server.
Weighted Round Robin
The weighted round robin (WRR) considers the resource capacities of the application instances (see Figure 3-10). An instance with higher hardware specifications than others can handle more requests. This is done by assigning a weight to each instance. There are no specific criteria to determine the weight. This is left to the system administrator. The node with the higher specs is apportioned a greater number of requests. The diagram in Figure 3-10 shows the request distribution for WRR.
Figure 3-10
Weighted distribution of requests
So far, you learned that weights are assigned to a different instance of the application. But in Traefik, this is denoted at the service level rather than the server level. In a previous section, we configured the loadbalancer type of service. The service had the location of each of the servers. But to work with WRR, we need to divide servers logically into different load capacities. Each of these capacity service instances are grouped into a weighted service instance with the associated weights (see Figure 3-11).
Figure 3-11
Weighted service hierarchy
# Removed for Brevity
services :
guestbook-service :
weighted:
services:
- name: guestv1
weight: 3
- name: guestv2
weight: 1
guestv1 :
loadBalancer :
servers :
- url : "http://192.168.1.10:9090/" -- host 1
- url : "http://192.168.1.11:9191/" -- host 2
guestv2 :
loadBalancer :
servers :
- url : "http://192.168.1.12:9292/" -- Host 3
The following can be said about the preceding code.
There are three hosts for above application. Host1 and Host2 are grouped together.
guestv1 defined configuration of grouped hosts. guestv2 defined configuration for host three instance.
guestbook-service configures both logical groups in ratio 3:1. Traefik sends every fourth request to h3 while the remaining requests are distributed in a round-robin manner within host2 and host3.
You can see the weighted distribution in the Traefik dashboard shown in Figure 3-12.
Figure 3-12
Weighted service
There is no health check for the weighted service. The health of the weighted service depends on the health configured for the underlying services.
Sticky Session
In the previous section, we enabled a sticky session to bind the user to a server. But when working with weighted services adding a sticky attribute to the loadbalancer service is not good enough. The weight service cannot recognize instance details from the cookie. To do so, we have to configure a cookie at the weighted service level. In summary, session stickiness is maintained at all levels of the service hierarchy. Thus, we need to add a cookie at the weighted service level and the load balancer level.
services :
guestbook-service :
weighted:
services:
- name: guestv1
weight: 3
- name: guestv2
weight: 1
sticky:
cookie:
httpOnly: true
guestv1 :
loadBalancer :
sticky:
cookie:
httpOnly: true
servers :
- url : "http://192.168.1.10:9090/"
- url : "http://192.168.1.9:9191/"
guestv2 :
loadBalancer :
servers :
- url : "http://192.168.1.11:9292/"
The following can be said about the preceding code.
It enables session stickiness for guestbook-service by adding the sticky attribute.
It enables session stickiness for guestbookv1 by adding the sticky attribute.
The configuration added a cookie with a generated name. The sticky record has the optional attributes that can configure the behavior of the generated cookies. We can validate the attributes in Browser cookie console (see Figure 3-13). The configured cookie details are also visible on Traefik dashboard (see Figure 3-14).
Figure 3-13
Browser cookies for all services
Figure 3-14
Sticky session for weighted service
Note
Traefik also supports dynamic request routing where weights are evaluated for every request. But the feature is available in 1.X and not in 2.X.
Mirroring
Traffic shadowing or Mirroring is a deployment pattern where production traffic is copied and sent to two places. The original production server gets the actual request and the same get duplicated to a test environment. This process helps validate regression issues in the new version of the application. If the testing version has the same request URLs and parameters, then mirroring can validate if the new version is as close to bug-free as possible.
Traffic mirroring is often done asynchronously. This makes sure that the original request processing is not impacted in any manner. Moreover, all mirrored requests are fire and forget. In summary, the response from the mirror is ignored. It is not propagated back to the client in any scenario.
As a practice, we do not duplicate all requests to the mirror service. If done so, it would require a testing infrastructure that is comparable to production. Thus, only a percentage of the requests are replicated to the mirror service. Traefik configures a mirror service as a different type of service. It does not limit you to have one mirror service. We can add as many mirror services as we required. We are only required to create a hierarchy of services as it was done in WRR.
services :
guestbook-service :
mirroring:
service: guestv1
mirrors:
- name: guestv2
percent: 10
guestv1 :
loadBalancer :
sticky:
cookie:
servers :
- url : "http://localhost:9090/"
healthCheck:
scheme : http
path: /
interval: "10s"
timeout: "1s"
guestv2 :
loadBalancer :
servers :
- url : "http://localhost:9191/"
healthCheck:
scheme : http
path: /
interval: "10s"
timeout: "1s"
In the code, we did the following to create a mirror for guest-service.
The top-level service (guest-service) is defined as a composite service consisting of two different services.
The mirroring attribute tells that the current service is a mirroring service. We added only one mirror service.
guestv2 is described as the mirror. It receives only 10 percent of the original request.
Next, we define two loadBalancer services.
Lastly, we added healthCheck for each application. But the service derives its health from the underlying original production service(s). The health of the mirror has no impact on the heath of the original service. The configured mirror service is also show on the service view of Traefik dashboard (see Figure 3-15).
Figure 3-15
Mirror service
Besides sending a subset of requests, we often do not want to send large requests. Again, here as well, the limit is based on the infrastructure available with the mirror. This is done by setting up the maxBodySize attribute.
services :
guestbook-service :
mirroring:
service: guestv1
maxBodySize : 1024
mirrors:
- name: guestv2
percent: 10
# Removed for brevity
The code defines a limit of 1024 as max body size. This send requests have a size less than 1024 for a mirror service.
TCP Service
Traefik can load balance TCP requests. TCP is the protocol for communication across many popular applications like LDAP, MySQL, and Mongo. The protocol creates a socket connection. All communication is done in a request-response manner. In the following section, we load the balance MongoDB server. MongoDB communicates over TCP protocol. The installation of the MongoDB server is beyond the scope of the book. Please refer to MongoDB documentation for the same.
Before we move ahead, we need to create an entrypoint for TCP in the static configuration. The entrypoint sends all incoming TCP requests to the mongo servers. The entrypoint is declared in the same manner as done for the HTTP service in Chapter 2.
entryPoints :
mongo :
address : ":80"
providers :
file :
directory : /Users/rahulsharma/traefik/ch03/code
watch : true
filename : config
api :
insecure : true
dashboard : true
Round Robin
We discussed the round-robin algorithm in the previous section. The algorithm distributes requests equally among the listed servers. Traefik allows you to load balance TCP services using the round-robin algorithm. As a prerequisite, you need to have MongoDB running on two servers.
tcp :
routers :
mongo-router :
entryPoints :
- mongo
rule : HostSNI(`*`)
service : mongo-tcp-service
services :
mongo-tcp-service:
loadBalancer :
servers :
- address : "192.168.1.10:27017"
- address : "192.168.1.11:27017"
The following can be said about the preceding code.
It describes a mongo-router for routing requests to mongo-tcp-service.
The TCP router has a single HostSNI rule. This enables you to operate a TCP and HTTP service on the same port. You see it when TLS support is enabled.
mongo-tcp-service has the same declaration as an HTTP service. It consists of a loadBalancer block.
The loadBalancer block contains a list of addresses, unlike the HTTP service, where the location was a URL. In TCP, this is a combination of an IP and a port.
The configured TCP services are shown in TCP service view of Traefik dashboard (see Figure 3-16).
The following command connects to Mongo servers using the Mongo shell. You can determine which server is connected using the db.hostInfo command.
Figure 3-16
Round-robin TCP service
$traefik:/# mongo -u root -p example --host localhost --port 80
MongoDB shell version v4.2.6
connecting to: mongodb:// localhost:80/?compressors=disabled&gssapiServiceName=mongodb
2020-05-23T13:41:25.742+0000 I STORAGE [initandlisten]
2020-05-23T13:41:25.742+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-05-23T13:41:25.742+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
> db.hostInfo()
Terminal Delay
A TCP client makes a connection only once. It then sends all requests over the open connection. This is opposite to an HTTP protocol, where every request is routed on a new connection. But this presents another challenge. A connection can be closed from either side. It can be closed from the server due to various reasons like business validations and service restart. It can be closed from the client as well. Since TCP works in duplex mode, every connection closed event must be acknowledged from both sides. If the other end does not handle the connection closer, the connection stays half-open. The open connection would lock Traefik resources.
Connection close handling can be improved by configuring a termination delay. The delay defines a timeout interval during which Traefik waits for connection close from both sides. After the delay, Traefik terminates the connection and recollects the allocated resources. The delay clock is set when either party sends a close event.
# removed for Brevity
services :
mongo-tcp-service:
terminationDelay: 50
loadBalancer :
servers :
- address : "192.168.1.10:27017"
- address : "192.168.1.11:27017"
The delay is configured at the service level. It applies to all servers under the service. The delay has a positive value specifying the interval in milliseconds. A negative value indicates that the connection is held until it is closed by both parties.
Note
There are no status codes or equivalent in TCP protocol. Thus there is no health check available for TCP services.
Weighted Round Robin
In the previous section, we discussed the weighted round-robin algorithm. The algorithm allows you to distribute incoming TCP requests based on a prescribed weight ratio. As seen in the HTTP service example, the weighted service is a higher-order service than the loadBalancer service. It has the same behavior for TCP services as well. Continuing with the MongoDB servers from the last section, let’s look at the weighted configuration.
# Removed for Brevity
services :
mongo-tcp-service :
weighted:
services:
- name: mongo-1-service
weight: 3
- name: mongo-2-service
weight: 1
mongo-1-service:
terminationDelay: 42
loadBalancer :
servers :
- address : "192.168.1.10:27017"
- address : "192.168.1.11:27017"
mongo-2-service:
terminationDelay: 42
loadBalancer :
servers :
- address : "192.168.1.12:27017"
The following can be said about the preceding code.
There are three hosts for above application. 192.168.1.10 and 192.168.1.11 are grouped together.
mongo-1-service defined configuration of grouped hosts. mongo-2-service defined configuration for the host3 instance.
Mongo-tcp-service configures both logical groups in ratio 3:1. Traefik sends every fourth connection request to 192.168.1.12 while the remaining requests are distributed in a round-robin manner within 192.168.1.10 and 192.168.1.11.
We can see the weighted distribution in the Traefik dashboard. (see Figure 3-17)
Figure 3-17
Weighted round-robin TCP service
UDP Service
Traefik can load balance UDP requests. UDP is the protocol for communication across many popular applications like IMAP, DNS, TFTP, and Memcache. UDP uses a connectionless communication model with a minimum of protocol mechanism. There are no handshakes, and there is no guaranteed delivery. Thus, the protocol has no overhead costs. Some applications require these attributes, like time-sensitive applications.
In the following section, we load balance TFTP servers. TFTP communicates over UDP protocol. The installation of a TFTP server is beyond the scope of this book. Please refer to TFTP/Unix documentation for more information.
Before we move ahead, we need to create an entrypoint, for UDP, in the static configuration. The entrypoint sends all incoming UDP traffic to the TFTP servers. The entrypoint declares in the same manner as for HTTP in Chapter 2.
We discussed the round-robin algorithm in previous sections. The algorithm distributes requests equally among the listed servers. Traefik allows you to load balance UDP services using the round-robin algorithm. As a prerequisite, you need to have TFTP running on two servers.
udp :
routers :
tftp-router :
entryPoints :
- tftp
service : tftp-service
services:
tftp-service:
loadBalancer :
servers :
- address : "192.168.1.10:69"
- address : "192.168.1.11:69"
The following can be said about the preceding code.
We described tftp-router to route requests to tftp-service.
A UDP router does not have any rules. It cannot perform hostname lookup. It can only be performed using a port.
tftp-service has the same declaration as an HTTP service. It consists of a loadBalancer block.
The loadBalancer block contains a list of addresses, unlike the HTTP service, where the location was a URL. In UDP, it is a combination of the IP and a port.
Traefik dashboard also provides a UDP service view to show the configured UDP services. (see Figure 3-18)
Figure 3-18
UDP service in round robin
Note
There is no health check available for UDP services.
The following command connects to TFTP servers by using the tftp command. You can transfer the sample files that we made available on the server.
traefik $ tftp 192.168.1.4
tftp> verbose
Verbose mode on.
tftp> get sample.md
getting from 192.168.1.4:sample.md to sample.md [netascii]
Received 682 bytes in 0.0 seconds [inf bits/sec]
Weighted Round Robin
The previous sections discussed the weighted round-robin algorithm. The algorithm allows you to distribute incoming UDP messages based on a prescribed weight ratio. As seen in the HTTP service example, the weighted service is a higher-order service than the loadBalancer service. It has the same behavior for UDP services as well. Continuing the TFTP servers from the last section, let’s look at the weighted configuration.
The following can be said about the preceding code.
There are three hosts for above application. 192.168.1.10 and 192.168.1.11 are grouped together.
tftp-1-service defines the configuration of grouped hosts. tftp-2-service defines the configuration for the host3 instance.
tftp-service configures both logical groups in ratio 3:1. Traefik sends every fourth connection request to 192.168.1.12 while the remaining requests are distributed in a round-robin manner within 192.168.1.10 and 192.168.1.11.
We can see the weighted distribution in the Traefik dashboard. (see Figure 3-19)
Figure 3-19
Weighted UDP service
Summary
In this chapter, you saw the different load balancing capabilities available in Traefik. We configured classic round robin and weighted round robin for HTTP, TCP, and UDP communication. You also worked with stick sessions and health checks for HTTP communication. Lastly, you saw mirroring capabilities that are used for canary deployments. In the next chapter, you look at TLS capabilities available in Traefik.
