NodePort

A NodePort Service provides a simple way for external software, such as a load balancer, to route traffic to the pods . The software only needs to be aware of node IPs, and the Service’s port(s).A NodePort Service exposes a fixed port on all nodes, which routes to applicable pods. A NodePort Service uses the .spec.ports.[].nodePort field to specify the port to open on all nodes, for the corresponding port on pods.

apiVersion: v1
kind: Service
metadata:
  name: demo-service
spec:
  type: NodePort
  selector:
    app: demo
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30000

The nodePort field can be left blank, in which case Kubernetes will automatically select a unique port. The flag --service-node-port-range in kube-controller-manager sets the valid range for ports,30000-32767. Manually specified ports must be within this range.

Using a NodePort service external users can connect to the nodeport on any node, and be routed to a pod on a node that has a pod backing that service — Figure 5-4 demonstrates this. The service directs traffic to node 3 and Iptables rules forward the traffic to Node 2 hosting the pod This is a bit inefficient, as a typical connection will be routed to a pod on another node.

Figure 5-4. Node Port Traffic Flow

Figure 5-4 requires us to discuss an attribute of Services, externalTrafficPolicy. ExternalTrafficPolicy indicates how a Service will route external traffic to either node-local or cluster-wide endpoints. “Local” preserves the client source IP and avoids a second hop for LoadBalancer and NodePort type services but risks potentially imbalanced traffic spreading. “Cluster” obscures the client source IP and may cause a second hop to another node but should have good overall load-spreading. A “Cluster” value means that for each worker node, the kube-proxy iptable rules are set up to route the traffic to the pods backing the service anywhere in the cluster, just like we have shown in Figure 5-4.

A “Local” value means the kube-proxy iptable rules are set up only on the worker nodes with relevant pods running to route the traffic local to the worker node. Using Local also allows application developers to preserve the source IP of the user request. If you set externalTrafficPolicy to the value Local, kube-proxy will only proxies requests to node local endpoints and will not forward traffic to other nodes. If there are no local endpoints, packets sent to the node are dropped.

Let us scale up the Deployment of our web app for some more testing.

 kubectl scale deployment app --replicas 4
deployment.apps/app scaled

 kubectl get pods -l app=app -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE
app-9cc7d9df8-9d5t8   1/1     Running   0          43s   10.244.2.4   kind-worker
app-9cc7d9df8-ffsm6   1/1     Running   0          75m   10.244.1.4   kind-worker2
app-9cc7d9df8-srxk5   1/1     Running   0          45s   10.244.3.4   kind-worker3
app-9cc7d9df8-zrnvb   1/1     Running   0          43s   10.244.3.5   kind-worker3

With four pods running we will have one pod at every node in the cluster.

 kubectl get pods -o wide -l app=app
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE
app-5586fc9d77-7frts   1/1     Running   0          31s   10.244.1.5   kind-worker2
app-5586fc9d77-mxhgw   1/1     Running   0          31s   10.244.3.9   kind-worker3
app-5586fc9d77-qpxwk   1/1     Running   0          84s   10.244.2.7   kind-worker
app-5586fc9d77-tpz8q   1/1     Running   0          31s   10.244.2.8   kind-worker

Now let’s deploy our NodePort Service

kubectl apply -f services-nodeport.yaml
service/nodeport-service created

kubectl describe svc nodeport-service
Name:                     nodeport-service
Namespace:                default
Labels:                   <none>
Annotations:              Selector:  app=app
Type:                     NodePort
IP:                       10.101.85.57
Port:                     echo  8080/TCP
TargetPort:               8080/TCP
NodePort:                 echo  30040/TCP
Endpoints:                10.244.1.5:8080,10.244.2.7:8080,10.244.2.8:8080 + 1 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

In order to test the nodeport service we must retrieve the IP address of a worker node

kubectl get nodes -o wide
NAME                 STATUS   ROLES   INTERNAL-IP OS-IMAGE       KERNEL-VERSION
kind-control-plane   Ready    master  172.18.0.5  Ubuntu 19.10   4.19.121-linuxkit
kind-worker          Ready    <none>  172.18.0.3  Ubuntu 19.10   4.19.121-linuxkit
kind-worker2         Ready    <none>  172.18.0.4  Ubuntu 19.10   4.19.121-linuxkit
kind-worker3         Ready    <none>  172.18.0.2  Ubuntu 19.10   4.19.121-linuxkit

Communication external to the cluster will use the NodePort of 30040 opened on each worker and the node worker’s IP address.

We can see that our pods are reachable on each host in the cluster.

kubectl exec -it dnsutils -- wget -q -O-  172.18.0.5:30040/host
NODE: kind-worker2, POD IP:10.244.1.5

kubectl exec -it dnsutils -- wget -q -O-  172.18.0.3:30040/host
NODE: kind-worker, POD IP:10.244.2.8

kubectl exec -it dnsutils -- wget -q -O-  172.18.0.4:30040/host
NODE: kind-worker2, POD IP:10.244.1.5

It’s important to consider the limitations as well. A nodeport deployment will fail if it can not allocate the requested port. Also, ports must be tracked across all applications using a NodePort service. Using manually selected ports raises the issue of port collisions (especially when applying a workload to multiple clusters, which may not have the exact same nodeports free).

Another downside of using Nodeport service type is that the load balancer or client software must be aware of node IP addresses. A static configuration (e.g., an operator manually copying node IP addresses) may become too outdated over time, (especially on a cloud provider) as IP addresses change or nodes are replaced. A reliable system automatically populates node IP addresses, either by watching which machines have been allocated to the cluster, or listing nodes from the Kubernetes API itself.

NodePorts are the earliest form of services. We will see that other services types use node ports as a base structure in their architecture. Nodeports should not be used by themselves, as clients would need to know the IP addresses of hosts and the node for connections requests. We will see how nodeports are used to enable load balancers later in the chapter and when we discuss the cloud networks.

Next up is the Services default type, the ClusterIP.

ClusterIP

The IP address of pods share the lifecycle of it and thus are not reliable for clients to use for requests. Services help overcome this pod networking design. A ClusterIP Service provides an internal load balancer, with a single IP address that maps to all matching (and ready) pods.

The Service’s IP address must be within the CIDR set in service-cluster-ip-range, in the apiserver. You can specify a valid IP address manually, or leave .spec.clusterIP unset to have one assigned automatically. The ClusterIP address is a virtual IP address that is only routable internally.

kube-proxy is responsible for making the ClusterIP address route to all applicable pods. See the section on kube-proxy for more. In “normal” configurations, kube-proxy performs L4 load balancing, which may not be sufficient. For example, older pods may see more load, due to accumulating more long-lived connections from clients. Or, a few clients making many requests may cause load to be distributed unevenly.

A particular use case example for ClusterIP is when a workload requires a load balancer within the same cluster.

In Figure 5-5, we can see a ClusterIP service deployed. The Service name is app with a selector or App=App1 There are two pods powering this service. Pod 1 and Pod 5 match the selector for the service.

Figure 5-5. Cluster IP Example Service

Let us dig into an example on the command line with our kind cluster.

We will deploy a ClusterIP service for use with our Golang webserver.

kubectl apply -f service-clusterip.yaml
service/clusterip-service created

kubectl describe svc clusterip-service
Name:              clusterip-service
Namespace:         default
Labels:            app=app
Annotations:       Selector:  app=app
Type:              ClusterIP
IP:                10.98.252.195
Port:              <unset>  80/TCP
TargetPort:        8080/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

The clusterip service name is resolvable in the network.

kubectl exec dnsutils -- host clusterip-service
clusterip-service.default.svc.cluster.local has address 10.98.252.195

Now we can reach the Host API endpoint with The Cluster IP, 10.98.252.195, The Service Name, clusterip-service, or directly with the pod IP 10.244.1.4 and port 8080.

kubectl exec dnsutils -- wget -q -O- clusterip-service/host
NODE: kind-worker2, POD IP:10.244.1.4

kubectl exec dnsutils -- wget -q -O- 10.98.252.195/host
NODE: kind-worker2, POD IP:10.244.1.4

kubectl exec dnsutils -- wget -q -O- 10.244.1.4:8080/host
NODE: kind-worker2, POD IP:10.244.1.4

The clusterIP service is the default type for Services. With that default status, it is warranted that we should explore what the ClusterIP service abstracted for us. If you recall from Chapter 2 and 3, this list is similar to what is set up with Docker Network, but we now also have iptables for the service across all nodes.

• View veth pair and match with pod

• View network namespace and match with pod

• Verify pids on node match pods

• Match services with iptables rules

To explore this we need to know what Worker node the pod is deployed too, and that is kind-worker2

kubectl get pods -o wide --field-selector spec.nodeName=kind-worker2 -l app=app
NAME                  READY   STATUS    RESTARTS   AGE     IP           NODE
app-9cc7d9df8-ffsm6   1/1     Running   0          7m23s   10.244.1.4   kind-worker2

Since we are using kind we can use docker ps and docker exec to get information out of the running worker node kind-worker-2

docker ps
CONTAINER ID   COMMAND                   PORTS                       NAMES
df6df0736958   "/usr/local/bin/entr…"                                kind-worker2
e242f11d2d00   "/usr/local/bin/entr…"                                kind-worker
a76b32f37c0e   "/usr/local/bin/entr…"                                kind-worker3
07ccb63d870f   "/usr/local/bin/entr…"    0.0.0.0:80->80/tcp,         kind-control-plane
                                         0.0.0.0:443->443/tcp,
                                         127.0.0.1:52321->6443/tcp

kind-worker2 container id is df6df0736958, kind was kind enough to label each container with names, so we can reference each worker node with its name kind-worker2

Let’s see our Pod’s, app-9cc7d9df8-ffsm6, IP address and route table information.

kubectl exec app-9cc7d9df8-ffsm6 ip r
default via 10.244.1.1 dev eth0
10.244.1.0/24 via 10.244.1.1 dev eth0 src 10.244.1.4
10.244.1.1 dev eth0 scope link src 10.244.1.4

Our Pods IP Address is 10.244.1.4 running on interface [email protected] with 10.244.1.1 as it’s default route. That matches the interface 5 on the pod, [email protected]

kubectl exec app-9cc7d9df8-ffsm6 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: [email protected]: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: [email protected]: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
5: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 3e:57:42:6e:cd:45 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.1.4/24 brd 10.244.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3c57:42ff:fe6e:cd45/64 scope link
       valid_lft forever preferred_lft forever

Let’s check the network namespace as well, from the node ip a output.

 docker exec -it kind-worker2 ip a
<trimmerd>
5: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 3e:39:16:38:3f:23 brd <> link-netns cni-ec37f6e4-a1b5-9bc9-b324-59d612edb4d4
    inet 10.244.1.1/32 brd 10.244.1.1 scope global veth45d1f3e8
       valid_lft forever preferred_lft forever

netns list confirms that the network namespaces match our pods interface to the host interface, cni-ec37f6e4-a1b5-9bc9-b324-59d612edb4d4.

docker exec -it kind-worker2 /usr/sbin/ip netns list
cni-ec37f6e4-a1b5-9bc9-b324-59d612edb4d4 (id: 2)
cni-c18c44cb-6c3e-c48d-b783-e7850d40e01c (id: 1)

Let’s see what process/es run inside that network namespace. For that we will use docker exec to run commands inside the node kind-worker2 hosting the pod and its network namespace.

 docker exec -it kind-worker2 /usr/sbin/ip netns pid cni-ec37f6e4-a1b5-9bc9-b324-59d612edb4d4
4687
4737

Now we can grep for each process id and inspect what those are doing.

docker exec -it kind-worker2 ps aux | grep 4687
root      4687  0.0  0.0    968     4 ?        Ss   17:00   0:00 /pause

docker exec -it kind-worker2 ps aux | grep 4737
root      4737  0.0  0.0 708376  6368 ?        Ssl  17:00   0:00 /opt/web-server

4737 is the process id of our Web server container running on the kind-worker2

4687 is our pause container holding onto all our namespaces.

Now let’s see what will happen to the iptables on the worker node.

docker exec -it kind-worker2 iptables -L
Chain INPUT (policy ACCEPT)
target                  prot opt source     destination
/* kubernetes service portals */
KUBE-SERVICES           all  --  anywhere   anywhere    ctstate NEW
/* kubernetes externally-visible service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere   anywhere    ctstate NEW
KUBE-FIREWALL           all  --  anywhere   anywhere

Chain FORWARD (policy ACCEPT)
target        prot opt source     destination
/* kubernetes forwarding rules */
KUBE-FORWARD  all  --  anywhere   anywhere
/* kubernetes service portals */
KUBE-SERVICES all  --  anywhere   anywhere             ctstate NEW

Chain OUTPUT (policy ACCEPT)
target          prot opt source               destination
/* kubernetes service portals */
KUBE-SERVICES   all  --  anywhere             anywhere             ctstate NEW
KUBE-FIREWALL   all  --  anywhere             anywhere

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination

Chain KUBE-FIREWALL (2 references)
target     prot opt source    destination
/* kubernetes firewall for dropping marked packets */
DROP       all  --  anywhere  anywhere   mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
target  prot opt source    destination
DROP    all  --  anywhere  anywhere    ctstate INVALID
/*kubernetes forwarding rules*/
ACCEPT  all  --  anywhere  anywhere     mark match 0x4000/0x4000
/*kubernetes forwarding conntrack pod source rule*/
ACCEPT  all  --  anywhere  anywhere     ctstate RELATED,ESTABLISHED
/*kubernetes forwarding conntrack pod destination rule*/
ACCEPT  all  --  anywhere  anywhere     ctstate RELATED,ESTABLISHED

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-PROXY-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination

That is a lot of tables being managed by Kubernetes.

We can dive a little deeper to examine the iptables responsible for our services we deployed. Let us retrieve the IP Address of the clusterip-service deployed. We need this to find the matching iptables rules.

kubectl get svc clusterip-service
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
clusterip-service   ClusterIP   10.98.252.195    <none>        80/TCP     57m

Now use the cluster ip of the service, 10.98.252.195 , to find our iptables rule.

docker exec -it  kind-worker2 iptables -L -t nat | grep 10.98.252.195
/* default/clusterip-service: cluster IP */
KUBE-MARK-MASQ  tcp  -- !10.244.0.0/16        10.98.252.195 tcp dpt:80
/* default/clusterip-service: cluster IP */
KUBE-SVC-V7R3EVKW3DT43QQM  tcp  --  anywhere  10.98.252.195 tcp dpt:80

List out all the rules on the chain KUBE-SVC-V7R3EVKW3DT43QQM

docker exec -it  kind-worker2 iptables -t nat -L KUBE-SVC-V7R3EVKW3DT43QQM
Chain KUBE-SVC-V7R3EVKW3DT43QQM (1 references)
target     prot opt source               destination
/* default/clusterip-service: */
KUBE-SEP-THJR2P3Q4C2QAEPT  all  --  anywhere             anywhere

The KUBE-SEP- will container the endpoints for the services, KUBE-SEP-THJR2P3Q4C2QAEPT

Now we can see what the rules for this chain are in iptables.

docker exec -it kind-worker2 iptables -L KUBE-SEP-THJR2P3Q4C2QAEPT -t nat
Chain KUBE-SEP-THJR2P3Q4C2QAEPT (1 references)
target          prot opt source          destination
/* default/clusterip-service: */
KUBE-MARK-MASQ  all  --  10.244.1.4      anywhere
/* default/clusterip-service: */
DNAT            tcp  --  anywhere        anywhere    tcp to:10.244.1.4:8080

10.244.1.4:8080 is one of the services endpoints, aka a pod backing the service, which is confirmed with the output of kubectl get ep clusterip-service.

kubectl get ep clusterip-service
NAME                ENDPOINTS                         AGE
clusterip-service   10.244.1.4:8080                   62m

kubectl describe ep clusterip-service
Name:         clusterip-service
Namespace:    default
Labels:       app=app
Annotations:  <none>
Subsets:
  Addresses:          10.244.1.4
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    <unset>  8080  TCP

Events:  <none>

Now, let’s explore the limitations of ClusterIP. The ClusterIP is for internal traffic to the cluster. ClusterIP suffers the same issues as endpoints does. As the service size grows, updates to it will slow. In chapter two we discussed how to mitigate that by using IPVS over Iptables as the proxy mode for kubeproxy. We will discuss later in this chapter how to get traffic into the cluster using Ingress and the other service type Loadbalancer.

The ClusterIP is the default type of services, but there are several other specific types of services; headless and externalName. ExternalName is a specific type of services that helps with reaching services outside the cluster. Headless we briefly touched on with statefulsets, but let’s review those in depth now.

Headless

A Headless Service isn’t a formal type of service (i.e., there is no .spec.type: Headless. A Headless service is a service with .spec.clusterIP: "None". This is distinct from merely not setting a ClusterIP, which makes Kubernetes automatically assign a ClusterIP.

When ClusterIP is set to “None”, the Service does not support any load balancing functionality. Instead, it only provisions an Endpoints object, and points the service DNS record at all pods that are selected and ready.

A Headless service provides a generic way to watch Endpoints, without needing to interact with the Kubernetes API. Fetching DNS records is much simpler than integrating with the Kubernetes API, and it may not be possible with third party software.

Headless services allows developers to deploy multiple copies of a pod in a deployment. Instead of a single IP address returned, like with ClusterIP, all the IP addresses of the endpoint are returned in the query. It then up to to client to pick which one to use. To see this in action let us scale up the Deployment of our web app.

 kubectl scale deployment app --replicas 4
deployment.apps/app scaled

 kubectl get pods -l app=app -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE
app-9cc7d9df8-9d5t8   1/1     Running   0          43s   10.244.2.4   kind-worker
app-9cc7d9df8-ffsm6   1/1     Running   0          75m   10.244.1.4   kind-worker2
app-9cc7d9df8-srxk5   1/1     Running   0          45s   10.244.3.4   kind-worker3
app-9cc7d9df8-zrnvb   1/1     Running   0          43s   10.244.3.5   kind-worker3

Now let us deploy the headless service.

kubectl apply -f service-headless.yml
service/headless-service created

The DNS query will return all four of the Pod IP addresses. Using our dnsutils image we can verify that is the case.

kubectl exec dnsutils -- host -v -t a headless-service
Trying "headless-service.default.svc.cluster.local"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45294
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;headless-service.default.svc.cluster.local. IN A

;; ANSWER SECTION:
headless-service.default.svc.cluster.local. 30 IN A 10.244.2.4
headless-service.default.svc.cluster.local. 30 IN A 10.244.3.5
headless-service.default.svc.cluster.local. 30 IN A 10.244.1.4
headless-service.default.svc.cluster.local. 30 IN A 10.244.3.4

Received 292 bytes from 10.96.0.10#53 in 0 ms

That ip addresses returned from the query also matches the Endpoints for the service. kubectl describe for the endpoint object confirms that.

 kubectl describe endpoints headless-service
Name:         headless-service
Namespace:    default
Labels:       service.kubernetes.io/headless
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-01-30T18:16:09Z
Subsets:
  Addresses:          10.244.1.4,10.244.2.4,10.244.3.4,10.244.3.5
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    <unset>  8080  TCP

Events:  <none>

Headless has a very specific use case and is not typically used for deployments. As we mentioned in the Statefulset section, if developers need to let the client decide which endpoint to use, Headless is the appropriate type of service to deploy. Two examples of headless services are clustered databases and applications that have client-side load-balancing logic built-in to the code.

Our next example is ExternalName, which aids in migrations of services external to cluster. It also offers other DNS advantages inside cluster DNS.

ExternalName Service

ExternalName is a special case of Service that does not have selectors and uses DNS names instead.

When looking up the host ext-service.default.svc.cluster.local, the cluster DNS Service returns a CNAME record of database.mycompany.com

apiVersion: v1
kind: Service
metadata:
  name: ext-service
spec:
  type: ExternalName
  externalName: database.mycompany.com

If developers are migrating an application into Kubernetes but its dependencies are staying external to the cluster An ExternalName allows you to define a DNS record internal to the cluster no matter where the service actually runs.

DNS will try all the search as seen in the example below.

 kubectl exec -it dnsutils -- host -v -t a github.com
Trying "github.com.default.svc.cluster.local"
Trying "github.com.svc.cluster.local"
Trying "github.com.cluster.local"
Trying "github.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55908
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;github.com.                    IN      A

;; ANSWER SECTION:
github.com.             30      IN      A       140.82.112.3

Received 54 bytes from 10.96.0.10#53 in 18 ms

As an example, externalName service allows developers to map a Service to a DNS name.

Now if we deploy the External Service

kubectl apply -f service-external.yml
service/external-service created

The A record for github.com is return from the external-service query.

kubectl exec -it dnsutils -- host -v -t a external-service
Trying "external-service.default.svc.cluster.local"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11252
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;external-service.default.svc.cluster.local. IN A

;; ANSWER SECTION:
external-service.default.svc.cluster.local. 24 IN CNAME github.com.
github.com.             24      IN      A       140.82.112.3

Received 152 bytes from 10.96.0.10#53 in 0 ms

The CNAME for external service returns github.com.

kubectl exec -it dnsutils -- host -v -t cname external-service
Trying "external-service.default.svc.cluster.local"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36874
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;external-service.default.svc.cluster.local. IN CNAME

;; ANSWER SECTION:
external-service.default.svc.cluster.local. 30 IN CNAME github.com.

Received 126 bytes from 10.96.0.10#53 in 0 ms

Sending traffic to a Headless Service via DNS record is possible, but inadvisable. DNS is a notoriously poor way to load balancer, as software takes very different (and often simple or unintuitive) approaches to A or AAAA DNS records that return multiple IP addresses. For example, it is common for software to always choose the first IP address in the response, and/or caching reusing the same IP address indefinitely. If you need to be able to send traffic to the Service’s DNS address, consider a (standard) ClusterIP or LoadBalancer service.

The “correct” way to use a Headless service is to query the Service’s A/AAAA DNS record, and use that data in a serverside or clientside load balancer.

Most of the services we have been discussing are for internal traffic management for the cluster network. In our next sections will be reviewing how to route requests into the cluster with Service Type Loadbalancer and Ingresses.

LoadBalancer

LoadBalancer Services expose services external to the cluster network. They combine NodePort Service behavior with an external integration, such as a cloud provider’s load balancer. Notably, LoadBalancer services handle L4 traffic (unlike Ingress, which handles L7 traffic), so they will work for any TCP or UDP service, provided the load balancer selected supports L4 traffic.

Configuration and load balancer options are extremely dependent on the cloud provider. For example, some will support .spec.loadBalancerIP (with varying setup required), and some will ignore it.

apiVersion: v1
kind: Service
metadata:
  name: demo-service
spec:
  selector:
    app: demo
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  clusterIP: 10.0.5.1
  type: LoadBalancer

Once the load balancer has been provisioned, its IP address will be written to .status.loadBalancer.ingress.ip.

LoadBalancer Services are useful for exposing TCP or UDP services to the outside world. Traffic will come into to the load balancer on its public IP address address and TCP port 80, defined by spec.ports[*].port and routed to the clusterIP address,10.0.5.1, and then to container target port 8080,spec.ports[*].targetPort. Not shown in the example the .spec.ports[*].nodePort, if not specified Kubernetes will pick one for the service.

In Figure 5-6 we can see how Load Balancer builds on the other service types. The Cloud load balancer will determine how to distribute traffic; we will discuss that in depth in next chapter.

Figure 5-6. Load Balancer Service

Let us continue to extend our golang web server example with a LoadBalancer Service.

Since we are running on our local machine and not in a Service Provider like AWS, GCP or Azure, we can use MetalLB as an example for our Loadbalancer service. MetalLB project aims to allow users to deploy Bare Metal Loadbalancers for their clusters.

This example has been modified from Kind example deployment https://kind.sigs.k8s.io/docs/user/loadbalancer

Our first step is to deploy a separate namespace for MetalLB.

kubectl apply -f mlb-ns.yaml
namespace/metallb-system created

MetalLB Members also require a secret for joining the loadbalancer cluster, let us deploy one now for them to use in our cluster.

kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
secret/memberlist created

Now we can deploy MetalLB!

 kubectl apply -f ./metallb.yaml
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created

As you can see it deploys many objects, and now we wait for the deployment to finish. We can monitor the deployment of resources with --watch option in the metallb-system namespace.

kubectl get pods -n metallb-system --watch
NAME                          READY   STATUS              RESTARTS   AGE
controller-5df88bd85d-mvgqn   0/1     ContainerCreating   0          10s
speaker-5knqb                 1/1     Running             0          10s
speaker-k79c9                 1/1     Running             0          10s
speaker-pfs2p                 1/1     Running             0          10s
speaker-sl7fd                 1/1     Running             0          10s
controller-5df88bd85d-mvgqn   1/1     Running             0          12s

To complete configuration, we need to provide metallb a range of IP addresses it controls. This his range has to be on the docker kind network.

docker network inspect -f '{{.IPAM.Config}}' kind
[{172.18.0.0/16  172.18.0.1 map[]} {fc00:f853:ccd:e793::/64  fc00:f853:ccd:e793::1 map[]}]

172.18.0.0/16 is our docker network running locally.

We want our loadbalancer IP range to come from this subclass. We can configure metallb, for instance, to use 172.18.255.200 to 172.18.255.250 by creating the configmap.

The config map would look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 172.18.255.200-172.18.255.250

Let us deploy it so we can use MetalLB.

kubectl apply -f ./metallb-configmap.yaml

Now that we MetalLB deploy we deploy a loadbalancer for our Web app.

kubectl apply -f services-loadbalancer.yaml
service/loadbalancer-service created

For fun let us scale the web app deployment to 10, if you have the resources for it!

kubectl scale deployment app --replicas 10

 kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
app-7bdb9ffd6c-b5x7m    2/2     Running   0          26s   10.244.3.15   kind-worker    <none>           <none>
app-7bdb9ffd6c-bqtf8    2/2     Running   0          26s   10.244.2.13   kind-worker2   <none>           <none>
app-7bdb9ffd6c-fb9sf    2/2     Running   0          26s   10.244.3.14   kind-worker    <none>           <none>
app-7bdb9ffd6c-hrt7b    2/2     Running   0          26s   10.244.2.7   kind-worker2    <none>           <none>
app-7bdb9ffd6c-l2794    2/2     Running   0          26s   10.244.2.9   kind-worker2    <none>           <none>
app-7bdb9ffd6c-l4cfx    2/2     Running   0          26s   10.244.3.11   kind-worker2   <none>           <none>
app-7bdb9ffd6c-rr4kn    2/2     Running   0          23m   10.244.3.10   kind-worker    <none>           <none>
app-7bdb9ffd6c-s4k92    2/2     Running   0          26s   10.244.3.13   kind-worker    <none>           <none>
app-7bdb9ffd6c-shmdt    2/2     Running   0          26s   10.244.1.12   kind-worker3   <none>           <none>
app-7bdb9ffd6c-v87f9    2/2     Running   0          26s   10.244.1.11   kind-worker3   <none>           <none>
app2-658bcd97bd-4n888   1/1     Running   0          35m   10.244.2.6    kind-worker3   <none>           <none>
app2-658bcd97bd-mnpkp   1/1     Running   0          35m   10.244.3.7    kind-worker    <none>           <none>
app2-658bcd97bd-w2qkl   1/1     Running   0          35m   10.244.3.8    kind-worker    <none>           <none>
dnsutils                1/1     Running   1          75m   10.244.1.2    kind-worker3   <none>           <none>
postgres-0              1/1     Running   0          75m   10.244.1.4    kind-worker3   <none>           <none>
postgres-1              1/1     Running   0          75m   10.244.3.4    kind-worker    <none>           <none>

Now we can test the provisioned load balancer.

With more replicas deployed for our App behind the loadbalancer, we need the external IP of the Loadbalancer, 172.18.255.200.

kubectl get svc loadbalancer-service
NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
loadbalancer-service   LoadBalancer   10.99.24.220   172.18.255.200   80:31276/TCP   52s


kubectl get svc/loadbalancer-service -o=jsonpath='{.status.loadBalancer.ingress[0].ip}'
172.18.255.200

Since Docker for Mac or Windows does not expose the kind network to the host, we can not directly reach the 172.18.255.200 Loadbalancer IP on the Docker private network.

We can simulate it by attaching a docker container to the kind network and curling the Loadbalancer as a workaround.

We will use another great networking docker image called nicolaka/netshoot, to run locally, attach to the kind docker network, and send requests to our MetalLB Loadbalancer.

If we run it several times we can see the loadbalancer is doing its job of routing traffic to different Pods.

docker run --network kind -a stdin -a stdout -i -t nicolaka/netshoot curl 172.18.255.200/host
NODE: kind-worker, POD IP:10.244.2.7

docker run --network kind -a stdin -a stdout -i -t nicolaka/netshoot curl 172.18.255.200/host
NODE: kind-worker, POD IP:10.244.2.9

docker run --network kind -a stdin -a stdout -i -t nicolaka/netshoot curl 172.18.255.200/host
NODE: kind-worker3, POD IP:10.244.3.11

docker run --network kind -a stdin -a stdout -i -t nicolaka/netshoot curl 172.18.255.200/host
NODE: kind-worker2, POD IP:10.244.1.6

docker run --network kind -a stdin -a stdout -i -t nicolaka/netshoot curl 172.18.255.200/host
NODE: kind-worker, POD IP:10.244.2.9

With each new request the metalLB service is sending requests to different pods. Loadbalancer like other services uses selectors and labels for the pods, and we can see in the kubectl describe endpoints loadbalancer-service. The pod IP addresses match our results from the curl commands.

 kubectl describe endpoints loadbalancer-service
Name:         loadbalancer-service
Namespace:    default
Labels:       app=app
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-01-30T19:59:57Z
Subsets:
  Addresses:
  10.244.1.6,
  10.244.1.7,
  10.244.1.8,
  10.244.2.10,
  10.244.2.7,
  10.244.2.8,
  10.244.2.9,
  10.244.3.11,
  10.244.3.12,
  10.244.3.9
  NotReadyAddresses:  <none>
  Ports:
    Name          Port  Protocol
    ----          ----  --------
    service-port  8080  TCP

Events:  <none>

It is important to remember that LoadBalancer Services require specific integrations, and will not work without cloud provider support, or manually-installed software such as MetalLB.

They are not (normally) L7 load balancers, and therefore cannot intelligently handle HTTP(S) requests. There is a one to one mapping of load balancer to workload, which means that all requests sent to that load balancer must be handled by the same workload.

We can scale our Application to the demands of the Users, with no need for configuration changes on anyone part. Kubernetes and the Loadbalancer service takes care of all of that for developers, system, and network administrators.

We will see in the next chapter how we can take that even further using Cloud Services for autoscaling.

Services Conclusion

Here are some troubleshooting tips if issues arise with the endpoints or services.

Removing the label on the pod allows it to continue to run while also updating the endpoint and service. The endpoint controller will remove that unlabelled pod from the endpoints objects and the deployment will deploy another pod, this will allow you to troubleshoot issues with that specific unlabeled pod but not adverse effect the service to end customers. I’ve used this one countless times during development, and we did so in the previous sections examples.

• There are two probes that communicated the pod’s health to the Kubelet and the rest of the Kubernetes environment

• It is also very easy to mess YAML manifest, make sure to compare ports on the service and pods, and that they match.

• We discuss network Policies in Chapter 3 and those can also stop pods from communicating with each other and services. If you Cluster Network is using network policies ensure that they are set up appropriately for application traffic flow

• Also remember using diagnostic tools like the dnsutils pod, or the netshoot pods on the cluster network are helpful debugging tools.

• If endpoints are taking too long to come up in the cluster, there are several options on that can be configured on the kubelet to control how fast it responds to change in the Kubernetes environment

  • --kube-api-qps the Query Per Second rate at which the Kubelet will use when communicating with the Kubernetes apiserver, the default 5.

  • --kube-api-burst this will temporarily allow api queries to burst to this number, the default is 10.

  • --iptables-sync-period is the maximum interval of how often iptables rules are refreshed (e.g., 5s, 1m, 2h22m). Must be greater than 0 and the is default 30s.

  • --ipvs-sync-period duration is the maximum interval of how often ipvs rules are refreshed. Must be greater than 0; Default 30s.

• Increasing these options for larger clusters is recommended but also remember this increases the resources on both the Kubelet and the API server so keep that in mind.

These can help with alleviate issues and are good to be aware of as the number of services and pods grow in the cluster.

The various types of services exemplify how power the Network abstractions are in Kubernetes. We have dug deep into how these work for each layer of the tool chain. The developer looking to deploy applications to Kubernetes now has the knowledge to pick and choose which services are right for their use case. No longer will a Network Administrator have to manually update load balancers with IP address, with Kubernetes managing that for them.

We have just scratched the surface of what is possible with Services. With each new version of Kubernetes there are options to tune and configurations to run services. Test out each service for your use cases and ensure you are using the appropriate services to optimize your applications on the Kubernetes network.

The LoadBalancer service type is the only one that allows for traffic into the cluster. Exposing HTTP(S) services behind a load balancer, for external users to connect to. Ingresses support path-based routing, which allows different HTTP paths to be served by different services. In our next section will discuss Ingress and how it is an alternative to managing connectivity into the cluster resources

Ingress Controllers and Rules

We call ingress implementations ingress controllers. In Kubernetes, a controller is software that is responsible for managing a typical resource type, and making reality match the desired state.

There are two general kinds of controllers: external load balancer controllers, and internal load balancer controllers. External load balancer controllers create a load balancer that exists “outside” the cluster, such as a cloud provider product. Internal load balancer controllers deploy a load balancer that runs within the cluster, and do not directly solve the problem of routing consumers to the load balancer. There are a myriad of ways that cluster administrators run internal load balancers, such as running the load balancer on a subset of special nodes, and routing traffic somehow to those nodes. The primary motivation for choosing an internal load balancer is cost reduction. An internal load balancer for ingress can route traffic for multiple ingress objects, whereas an external load balancer controller typically needs one load balancer per ingress. As most cloud providers charge by load balancer, it is cheaper to support a single cloud load balancer that does fan-out within the cluster, rather than many cloud load balancers. Note that this incurs operational overhead, and increased latency and compute costs, so be sure the money you’re saving is worth it. Many companies have a bad habit of optimizing on inconsequential cloud spend line items.

Let’s look at the spec for an Ingress controller. Like LoadBalancer services, most of the spec is universal, but various ingress controllers have different features and accept different unique config. We’ll start with the basics.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: basic-ingress
spec:
  rules:
  - http:
      paths:
      # Send all /demo requests to demo-service.
      - path: /demo
        pathType: Prefix
        backend:
          service:
            name: demo-service
            port:
              number: 80
  # Send all other requests to main-service.
  defaultBackend:
    service:
      name: main-service
      port:
        number: 80

The above example is representative of a typical ingress. It sends traffic to /demo to one service, and all other traffic to another. Ingresses have a “default backend”, where requests are routed if no rule matches. This can be configured in many ingress controllers in the controller configuration itself (e.g., a generic 404 page), and many support the .spec.defaultBackend field. Ingresses support multiple ways to specify a path. There are currently three.

Exact

Matches the specific path and only the given path (including trailing / or lack thereof).

Prefix

Matches all paths that start with the given path.

ImplementationSpecific

Allows for custom semantics from the current ingress controller.

When a request matches multiple paths, the most specific match is chosen. For example, if there are rules for /first and /first/second, any request starting with /first/second will go to the backend for /first/second. If a path matches an exact path and a prefix path, the request will go to the backend for the exact rule.

Ingresses can also use hostnames in rules.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: multi-host-ingress
spec:
  rules:
  - host: a.example.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: service-a
            port:
              number: 80
  - host: b.example.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: service-b
            port:
              number: 80

In this example, we serve traffic to a.example.com from one service, and traffic to b.example.com from another. This is comparable to virtualhosts in webservers. You may want to use host rules to use a single load balancer and IP to serve multiple unique domains.

Ingresses have basic TLS support.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-ingress-secure
spec:
  tls:
  - hosts:
      - https-example.com
    secretName: demo-tls
  rules:
  - host: https-example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: demo-service
            port:
              number: 80

The TLS config references a Kubernetes Secret by name, in .spec.tls.[*].secretName. Ingress controllers expect the TLS certificate and key to be provided in .data."tls.crt" and .data."tls.key" respectively, as shown below.

apiVersion: v1
kind: Secret
metadata:
  name: demo-tls
type: kubernetes.io/tls
data:
  tls.crt: cert, encoded in base64
  tls.key: key, encoded in base64

We mentioned earlier that ingress is simply a spec, and drastically different implementations exist. It’s possible to use multiple ingress controllers in a single cluster, using IngressClass settings. An ingress class represents an ingress controller, and therefore a specific ingress implementation.

IngressClass was introduced in Kubernetes 1.18. Prior to 1.18, annotating ingresses with kubernetes.io/ingress.class was a common convention, but relied on all installed ingress controllers to support it. Ingresses can pick an ingress class by setting the class’s name in .spec.ingressClassName.

Ingress only supports HTTP(S) requests, which is insufficient if your service uses a different protocol (e.g., most databases use their own protocols). Some ingress controllers, such as the NGINX ingress controller, do support TCP and UDP, but this is not the norm.

Now onto deploying an Ingress Controller; so we can add ingress rules to our golang web server example.

When we deployed our kind cluster, we had to add several options to allow us to deploy an ingress controller.

• extraPortMappings allow the local host to make requests to the Ingress controller over ports 80/443

• node-labels only allow the ingress controller to run on a specific node(s) matching the label selector

There are many options to choose from with Ingress Controllers. The Kubernetes system does not start or have a default controller like it does with other pieces. The Kubernetes community does support an AWS, GCE and Nginx Ingress Controllers. Table 5-1 outlines several options for Ingress.

Some things to consider when deciding on the Ingress for your clusters.

• Protocol Support - Do you need more than TCP/UDP, for example gRPC integration or websocket?

• Commercial Support - Do you need a commercial support?

• Advanced Features - Is JWT/oAuth2 authentication or circuit breakers requirements for your applications?

• API Gateway Features - Do you need some API Gateway functionalities such as rate-limiting?

• Traffic distribution - Does your application require support for specialized traffic distribution like canary A/B testing or mirroring?

For our example we have chosen to use the Community version of the NGINX Ingress Controller.

Let’s Deploy the NGINX Ingress Controller into our kind cluster.

kubectl apply -f ingress.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created

As with all deployments, we must wait for the Controller to be ready before we can use it. With the below command we can verify if our Ingress controller is ready for use.

kubectl wait --namespace ingress-nginx 
>   --for=condition=ready pod 
>   --selector=app.kubernetes.io/component=controller 
>   --timeout=90s
pod/ingress-nginx-controller-76b5f89575-zps4k condition met

The Controller is deployed to the cluster, and now we’re ready to write Ingress rules for our application.

kubectl apply -f ingress-rule.yaml
ingress.extensions/ingress-resource created

kubectl get ingress
NAME               CLASS    HOSTS   ADDRESS   PORTS   AGE
ingress-resource   <none>   *                 80      4s

kubectl describe ingress
Name:             ingress-resource
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /host  clusterip-service:8080 (10.244.1.6:8080,10.244.1.7:8080,10.244.1.8:8080)
Annotations:  kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  Sync    17s   nginx-ingress-controller  Scheduled for sync

curl localhost/host
NODE: kind-worker2, POD IP:10.244.1.6
curl localhost/healthz

kubectl apply -f ingress-example-2.yaml
deployment.apps/app2 created
service/clusterip-service-2 configured
ingress.extensions/ingress-resource-2 configured

curl localhost/host
NODE: kind-worker2, POD IP:10.244.1.6

curl localhost/data
Database Connected