Deploying Non-Mesh Services

To deploy frontend and backend into Kubernetes, you need to create a Deployment and Service resource for each service. For simplicity, use a separate file for each resource. In the same directory where you created your values.yaml file for the Helm installation, create a new directory called manifests and cd into it:

$ mkdir manifests
$ cd manifests

Inside the manifests/ directory, create four YAML files with the following contents:

# frontend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  labels:
    app: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: nicholasjackson/fake-service:v0.21.0
        env:
        - name: "LISTEN_ADDR"
          value: "0.0.0.0:80"
        - name: NAME
          value: "frontend"
        - name: MESSAGE
          value: "Hello from frontend"
        - name: UPSTREAM_URIS
          value: "http://backend"
        ports:
        - containerPort: 80

# frontend-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    app: frontend
spec:
  selector:
    app: frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

# backend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  labels:
    app: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: backend
        image: nicholasjackson/fake-service:v0.21.0
        env:
        - name: LISTEN_ADDR
          value: "0.0.0.0:80"
        - name: NAME
          value: "backend"
        - name: MESSAGE
          value: "Hello from backend"
        ports:
        - containerPort: 80

# backend-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: backend
  labels:
    app: backend
spec:
  selector:
    app: backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Use kubectl apply to apply these resources to Kubernetes:

$ kubectl apply -f ./
deployment.apps/backend created
service/backend created
deployment.apps/frontend created
service/frontend created

Use the kubectl get command with the --selector flag to view the frontend service’s Deployment and Service:

$ kubectl get deployment,service --selector app=frontend
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/frontend   1/1     1            1           1s

NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/frontend   ClusterIP   10.96.148.61   <none>        80/TCP    1s

deployment.apps/frontend should show 1/1 READY and 1 Available, meaning that there is one Pod running and the one container in that Pod is ready.

Use the same command with a difference selector to view the backend service’s Deployment and Service:

$ kubectl get deployment,service --selector app=backend
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/backend   1/1     1            1           1s

NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/backend   ClusterIP   10.96.38.122   <none>        80/TCP    1s

Now that your services are deployed, let’s see if frontend can call backend as expected. Use the kubectl port-forward command to forward your workstation’s localhost:8080 to the frontend application:

$ kubectl port-forward deployment/frontend 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

Now in another terminal, use curl to mimic a user’s call to frontend:

$ curl http://localhost:8080
{
  "name": "frontend",
  "uri": "/",
  "type": "HTTP",
  "ip_addresses": [
    "10.244.0.12"
  ],
  "start_time": "2021-04-04T20:29:54.940665",
  "end_time": "2021-04-04T20:29:54.943740",
  "duration": "3.07511ms",
  "body": "Hello from frontend", 
  "upstream_calls": {
    "http://backend": { 
      "name": "backend",
      "uri": "http://backend",
      "type": "HTTP",
      "ip_addresses": [
        "10.244.0.10"
      ],
      "start_time": "2021-04-04T20:29:54.943236",
      "end_time": "2021-04-04T20:29:54.943304",
      "duration": "67.339µs",
      "headers": {
        "Content-Length": "256",
        "Content-Type": "text/plain; charset=utf-8",
        "Date": "Sun, 04 Apr 2021 20:29:54 GMT"
      },
      "body": "Hello from backend", 
      "code": 200 
    }
  },
  "code": 200
}

The response from the frontend service.

The response from the backend service is included.

The body of the response from the backend service was “Hello from backend”.

The HTTP response code of the call to backend.

You’ve successfully deployed the frontend and backend applications into Kubernetes and verified that they are communicating as expected. However, these services aren’t yet part of the service mesh: they’re communicating insecurely over plaintext, there is no way to prevent other applications from calling them, and you can’t control where they’re routing their traffic. The next step is to add them to the service mesh so we can rectify these issues.

Adding Services to the Mesh

On Kubernetes it is easy to add existing services to the service mesh. All you need to do is add an annotation to the Deployment that tells Consul to inject its service mesh configuration: consul.hashicorp.com/connect-inject: true.

Let’s add this annotation to our two Deployments. First, edit frontend-deployment.yaml and add the annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  labels:
    app: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
      annotations: 
        "consul.hashicorp.com/connect-inject": "true" 
    spec:
      containers:
      - name: frontend
        image: nicholasjackson/fake-service:v0.21.0
        env:
        - name: "LISTEN_ADDR"
          value: "0.0.0.0:80"
        - name: NAME
          value: "frontend"
        - name: UPSTREAM_URIS
          value: "http://backend"
        ports:
        - containerPort: 80

Add annotations: under the spec.template.metadata key.

Add the consul.hashicorp.com/connect-inject annotation and set it to the string "true".

Now that you’ve modified your YAML, you’re ready to redeploy the frontend application. To do so, use kubectl apply:

$ kubectl apply -f ./
deployment.apps/backend unchanged
service/backend unchanged
deployment.apps/frontend configured
service/frontend unchanged

Kubernetes will handle starting a new Pod with the changes and spinning down the old Pod. You can use the kubectl rollout command to wait until the redeployment is complete:

$ kubectl rollout status --watch deploy/frontend
Waiting for deployment "frontend" rollout to finish...
deployment "frontend" successfully rolled out

Now that the Pod has been redeployed, when you open up the Consul UI, you should see the frontend service listed as part of the service mesh as shown in Figure 4-2.

Figure 3-2. The frontend service is now listed in the Consul UI.

Now add the backend service to the service mesh. You’ll add the same annotation to backend-deployment.yaml:

# backend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  labels:
    app: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
      annotations:
        "consul.hashicorp.com/connect-inject": "true"
    spec:
      containers:
      - name: backend
        image: nicholasjackson/fake-service:v0.21.0
        env:
        - name: LISTEN_ADDR
          value: "0.0.0.0:80"
        - name: NAME
          value: "backend"
        - name: MESSAGE
          value: "Hello from backend"
        ports:
        - containerPort: 80

Redeploy it with kubectl apply:

$ kubectl apply -f ./
deployment.apps/backend configured
service/backend unchanged
deployment.apps/frontend unchanged
service/frontend unchanged

Wait for the deployment to complete:

$ kubectl rollout status --watch deploy/backend
Waiting for deployment "backend" rollout to finish:
  1 old replicas are pending termination...
deployment "backend" successfully rolled out

Now when you view the UI (see “Consul UI” for how to access the UI), you should see both services listed as in Figure 3-3.

Figure 3-3. Both the frontend and backend services are now also listed in the Consul UI.

In order to test out the connection between frontend and backend, use the port-forward and curl combination. You will first need to stop the previous port-forward command with Ctrl-C if it’s still running because it’s pointing at a Pod that’s no longer running. Once you stop the old port-forward, start a new one:

$ kubectl port-forward deployment/frontend 8080:80

And then make your curl call again in another terminal:

$ curl http://localhost:8080
{
  "name": "frontend",
  "uri": "/",
  "type": "HTTP",
  "ip_addresses": [
    "10.244.0.15"
  ],
  "start_time": "2021-07-11T19:27:47.702816",
  "end_time": "2021-07-11T19:27:47.708736",
  "duration": "5.919404ms",
  "body": "Hello from frontend",
  "upstream_calls": {
    "http://backend": {
      "name": "backend",
      "uri": "http://backend",
      "type": "HTTP",
      "ip_addresses": [
        "10.244.0.16"
      ],
      "start_time": "2021-07-11T19:27:47.707571",
      "end_time": "2021-07-11T19:27:47.707688",
      "duration": "117.368µs",
      "headers": {
        "Content-Length": "257",
        "Content-Type": "text/plain; charset=utf-8",
        "Date": "Sun, 11 Jul 2021 19:27:47 GMT"
      },
      "body": "Hello from backend",
      "code": 200
    }
  },
  "code": 200
}

If everything is working as expected then the response will be exactly the same as before we added the services to the mesh. In order to verify that requests are going through the service mesh, you can use the Consul UI’s topology view that shows metrics emitted from the sidecar proxies.

To view the topology page for the backend service that is receiving traffic from frontend, click the backend service from the All Services view or navigate to http://localhost:8500/ui/dc1/services/backend/topology. You should see something similar to Figure 3-4.

Figure 3-4. The topology view for the backend service should show metrics from the incoming requests.

If you’re seeing metrics in the topology view that means you’ve successfully added your services to the mesh! Chapter 6 covers metrics in more detail.

In the next section, you’ll learn how to add services running on VMs into the mesh. If you’re only interested in Kubernetes, you can feel free to skip ahead to the next chapter where you’ll learn about service mesh security–specifically around encryption and authorization. In fact, even with these simple changes you’ve made here, your applications are already more secure because all the traffic flowing between frontend and backend is being automatically encrypted.

If you’re interested in also deploying onto VMs, read on as I cover how to add VM services into the service mesh.

Deploying Services

You’re going to use two pre-built example services so you can focus on the VM deployment rather than a particular service’s code. First, you need to get the service binaries onto your VM and then configure them to start with systemd.

$ vagrant ssh

$ wget https://github.com/nicholasjackson/fake-service 
  /releases/download/v0.21.0/fake_service_linux_amd64.zip
…
‘fake_service_linux_amd64.zip’ saved

$ sudo apt install unzip -y

$ unzip fake_service_linux_amd64.zip
Archive:  fake_service_linux_amd64.zip
  inflating: fake-service

$ chmod +x fake-service
$ sudo mv ./fake-service /usr/local/bin/fake-service

$ fake-service -h
Usage of fake-service:
  -help
    	--help to show env help

$ rm ./fake_service_linux_amd64.zip

Running services with systemd

Just like with how you are running Consul as a long-running background service under systemd, you want to do the same with your two applications frontend and backend. To do so, create two systemd unit files: frontend.service and backend.service in /etc/systemd/system/.

$ sudo touch /etc/systemd/system/frontend.service
$ sudo touch /etc/systemd/system/backend.service

Edit the frontend.service file and add the following:

[Unit]
Description="Frontend service"

# The service requires the VM's network
# to be configured, e.g. an IP address has been assigned.
Requires=network-online.target
After=network-online.target

[Service]
# ExecStart is the command to run.
ExecStart=/usr/local/bin/fake-service

# Environment sets environment variables.
# Set the frontend service to listen
# on port 8080.
Environment=LISTEN_ADDR=0.0.0.0:8080
Environment=NAME=frontend

# Set UPSTREAM_URIs to http://localhost:9090 because
# that's the port you'll run your backend service on.
Environment=UPSTREAM_URIS=http://localhost:9090

# The Install section configures this service to start
# automatically if the VM reboots.
[Install]
WantedBy=multi-user.target

Next edit the backend.service file and add the following:

[Unit]
Description="Backend service"
Requires=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/local/bin/fake-service

# Set the backend service to listen
# on port 9090.
Environment=LISTEN_ADDR=0.0.0.0:9090
Environment=NAME=backend
Environment=Message="Hello from backend"

[Install]
WantedBy=multi-user.target

Note

A unit in systemd means any resource managed by systemd, for example the long-running Consul service. A unit file is a systemd configuration file that tells systemd how to run and manage that unit.

With the unit files in place, you’re ready to enable and start your services. A service in systemd must first be enabled via systemctl:

$ sudo systemctl enable frontend.service
Created symlink ...

$ sudo systemctl enable backend.service
Created symlink ...

Now start the services:

$ sudo systemctl start frontend.service
$ sudo systemctl start backend.service

You can check their statuses with systemctl status:

$ sudo systemctl status frontend.service
● frontend.service - "Frontend service"
     Loaded: loaded...
     Active: active (running)...

$ sudo systemctl status backend.service
● backend.service - "Backend service"
     Loaded: loaded...
     Active: active (running)...

Finally, you can use curl to make a request to frontend which is listening on localhost:8080. Because frontend is configured to make a call to backend via its UPSTREAM_URIS environment variable, you should also see that call in the response:

$ curl http://localhost:8080
{
  "name": "frontend",
  "uri": "/",
  "type": "HTTP",
  "ip_addresses": [
    "172.31.56.8"
  ],
  "start_time": "2021-04-11T18:13:00.092110",
  "end_time": "2021-04-11T18:13:00.095565",
  "duration": "3.455289ms",
  "body": "Hello from frontend",
  "upstream_calls": {
    "http://localhost:9090": {
      "name": "Service",
      "uri": "http://localhost:9090",
      "type": "HTTP",
      "ip_addresses": [
        "172.31.56.8"
      ],
      "start_time": "2021-04-11T18:13:00.094039",
      "end_time": "2021-04-11T18:13:00.094256",
      "duration": "217.486µs",
      "headers": {
        "Content-Length": "257",
        "Content-Type": "text/plain; charset=utf-8",
        "Date": "Sun, 11 Apr 2021 18:13:00 GMT"
      },
      "body": "Hello from backend",
      "code": 200
    }
  },
  "code": 200
}

Success! In the response you can see the upstream_calls to http://localhost:9090 (the backend service) responding with success code 200. The current architecture is shown in Figure 3-5.

Figure 3-5. frontend listens on port 8080 and backend listens on port 9090. When you call frontend, it makes a request to backend.

Now you’re ready to register your services into Consul.

Registering Services in Consul

Services must be registered into Consul so Consul knows what’s running and on what IPs and ports. Services can be registered via configuration files or API. In dynamic systems like container orchestrators it makes sense to use the API, but on VMs it’s often simpler to use configuration files because you know ahead of time which services will be running where.

Create the service configuration files in Consul’s configuration directory /etc/consul.d:

$ sudo touch /etc/consul.d/frontend.hcl
$ sudo touch /etc/consul.d/backend.hcl

Next change the group and owner to be consul since Consul needs to read these files:

$ sudo chown consul:consul /etc/consul.d/frontend.hcl
$ sudo chown consul:consul /etc/consul.d/backend.hcl

Add the following contents to frontend.hcl:

service {
  name = "frontend"
  # frontend runs on port 8080.
  port = 8080

  # The "connect" stanza configures service mesh
  # features.
  connect {
    sidecar_service {
      proxy {
       # The "upstreams" stanza configures
       # which ports the sidecar proxy will expose
       # and what services they'll route to.
        upstreams = [
          {
            # Here you're configuring 9091
            # to route to the backend service.
            destination_name = "backend"
            local_bind_port  = 9091
          }
        ]
      }
    }
  }
}

And for backend.hcl:

service {
  name = "backend"
  # backend runs on port 9090.
  port = 9090

  # The backend service doesn't call
  # any other services so it doesn't
  # need an "upstreams" stanza.
  connect {
    sidecar_service {}
  }
}

You need to tell Consul to reload its configuration so it picks up the new files. To do so, use the consul reload command:

$ consul reload
Configuration reload triggered

You can use the consul catalog services command to check if your services were registered successfully:

$ consul catalog services
backend
backend-sidecar-proxy
consul
frontend
frontend-sidecar-proxy

You should see frontend and backend listed, along with their sidecar proxies.

In the UI, frontend and backend should be listed (Figure 3-6), though they will be shown as unhealthy. This is because their sidecar proxies are not yet running.

Figure 3-6. The frontend and backend services should be shown in the UI but they will be listed as unhealthy because their sidecar proxies are not yet running.

Now that your services are registered in Consul, it’s time to deploy their sidecar proxies.

Deploying Sidecar Proxies

For your services to become healthy, they require their sidecar proxies to be running. The sidecar proxies run Envoy so first you need to install Envoy. Then you will configure a systemd service to run each proxy.

$ wget https://archive.tetratelabs.io/envoy/download/  
  v1.18.3/envoy-v1.18.3-linux-amd64.tar.xz

$ tar -xvf envoy-v1.18.3-linux-amd64.tar.xz
$ sudo mv envoy-v1.18.3-linux-amd64/bin/envoy /usr/local/bin/envoy

$ envoy --version
envoy  version: .../1.18.3/Clean/RELEASE/BoringSSL

$ rm -rf envoy-v1.18.3-linux-amd64.tar.xz envoy-v1.18.3-linux-amd64

Configuring systemd to run sidecar proxies

Just like with the frontend and backend services, you will need to create systemd unit files. You’ll need one for frontend’s sidecar proxy and another for backend’s because you need to run a separate Envoy proxy process for each service.

Note

Consul doesn’t support running a single Envoy proxy for multiple services because each Envoy proxy must encode the identity of the source service into every request.

The command you’ll be using to run the sidecar proxies is:

consul connect envoy

This command accepts a flag, -sidecar-for, that is used to configure the Envoy proxy for a specific service.

First, create the two unit files:

$ sudo touch /etc/systemd/system/frontend-sidecar-proxy.service
$ sudo touch /etc/systemd/system/backend-sidecar-proxy.service

The frontend-sidecar-proxy.service file should have the following contents:

[Unit]
Description="Frontend sidecar proxy service"
Requires=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/consul connect envoy -sidecar-for frontend 
  -admin-bind 127.0.0.1:19000

[Install]
WantedBy=multi-user.target

And backend-sidecar-proxy.service should look like:

[Unit]
Description="Backend sidecar proxy service"
Requires=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/consul connect envoy -sidecar-for backend 
  -admin-bind 127.0.0.1:19001

[Install]
WantedBy=multi-user.target

Next enable the two services:

$ sudo systemctl enable frontend-sidecar-proxy.service
$ sudo systemctl enable backend-sidecar-proxy.service

And start them up!

$ sudo systemctl start frontend-sidecar-proxy.service
$ sudo systemctl start backend-sidecar-proxy.service

Check their statuses:

$ sudo systemctl status frontend-sidecar-proxy.service
 frontend-sidecar-proxy.service - "Frontend sidecar proxy service"
     Loaded: loaded...
     Active: active (running)...

$ sudo systemctl status backend-sidecar-proxy.service
 backend-sidecar-proxy.service - "Backend sidecar proxy service"
     Loaded: loaded...
     Active: active (running)...

With the proxies now running, the UI should show everything as healthy (see Figure 3-7).

Figure 3-7. The frontend and backend services are now healthy because their sidecar proxies are running.

Use curl again to see the request from frontend to backend:

$ curl http://localhost:8080
{
  "name": "frontend",
  "uri": "/",
  "type": "HTTP",
  "ip_addresses": [
    "172.31.56.8"
  ],
  "start_time": "2021-04-11T18:13:00.092110",
  "end_time": "2021-04-11T18:13:00.095565",
  "duration": "3.455289ms",
  "body": "Hello from frontend",
  "upstream_calls": {
    "http://localhost:9090": {
      "name": "backend",
      "uri": "http://localhost:9090",
      "type": "HTTP",
      "ip_addresses": [
        "172.31.56.8"
      ],
      "start_time": "2021-04-11T18:13:00.094039",
      "end_time": "2021-04-11T18:13:00.094256",
      "duration": "217.486µs",
      "headers": {
        "Content-Length": "257",
        "Content-Type": "text/plain; charset=utf-8",
        "Date": "Sun, 11 Apr 2021 18:13:00 GMT"
      },
      "body": "Hello from backend",
      "code": 200
    }
  },
  "code": 200
}

The response looks the same as before so how do you know it’s being routed through the service mesh? One thing you can try is to stop one of the proxies. If the proxy is stopped then we’d expect the request to fail.

Stop the backend sidecar proxy:

$ sudo systemctl stop backend-sidecar-proxy

And now try the request again:

$ curl http://localhost:8080
{
   ...
   "upstream_calls": {
    "http://localhost:9090": {
      "name": "backend",
      ...
       "body": "Hello from backend",
      "code": 200

The request succeeded with the sidecar proxy not even running so what’s going on?

Unlike in Kubernetes, when adding services to the service mesh on VMs you need to make some changes to your services so that they route traffic through their sidecar proxies. Right now, frontend is still calling backend directly and bypassing both its local proxy and backend’s proxy as shown in Figure 3-8.

Figure 3-8. frontend is calling backend directly–bypassing the sidecar proxies.

Configuring frontend to route through the mesh

You need to configure frontend to route its requests to backend through its sidecar proxy. Luckily, your frontend application exposes the environment variable UPSTREAM_URIS for configuring the URL to use to call backend. Currently in your frontend.service file this is set to http://localhost:9090 so all you need to do is set it to http://localhost:9091 which is where its sidecar Envoy proxy is listening.

Edit /etc/systemd/system/frontend.service (note, frontend.service, not frontend-sidecar-proxy.service) and change the UPSTREAM_URIS environment variable:

...
Environment=UPSTREAM_URIS=http://localhost:9091
...

Now run systemctl daemon-reload so that systemd loads the updated configuration and then use systemctl restart to restart the frontend service:

$ sudo systemctl daemon-reload
$ sudo systemctl restart frontend.service

If you run the curl command again, you should now see a failure because the backend proxy is not running:

$ curl localhost:8080
...
  "upstream_calls": {
    "http://localhost:9091": {
      "error": "Error communicating with upstream service:
       Get "http://localhost:9091/": read… connection reset by peer"
    }

Start the backend-sidecar-proxy service back up:

$ sudo systemctl start backend-sidecar-proxy

And run curl again. The error should go away (you may need to wait a couple of seconds for the proxy to be healthy):

$ curl localhost:8080
{
   ...
  "upstream_calls": {
    "http://localhost:9091": {
      "name": "backend",
       "body": "Hello from backend",
      "code": 200

Now you’ve confirmed that the requests between frontend and backend are being routed through the service mesh! Figure 3-9 shows what the architecture looks like now.

Figure 3-9. frontend is calling backend through the sidecar proxies.

Note

There are two sets of ports for a sidecar proxy: upstream ports and public ports. Upstream ports are used by the local service to route to its upstream dependencies. In this example, 9091 is an upstream port.

Public ports are the ports that incoming traffic from other sidecar proxies will be received on. In Figure 3-9 that port for backend’s sidecar proxy isn’t shown because it’s dynamically allocated by Consul. It can be hardcoded via the port setting: connect { sidecar_service { port = 21000 }}.