To verify the TLS configuration, follow these steps:

1. Run the istioctl command:

$ export RATING_POD=$(kubectl -n istio-lab get pods -l app=ratings -o jsonpath='{.items[0].metadata.name}') ; echo $RATING_POD
ratings-v1-79b6d99979-k2j7t

$ istioctl authn tls-check $RATING_POD.istio-lab ratings.istio-lab.svc.cluster.local
HOST:PORT                                    STATUS     SERVER  ---    
ratings.istio-lab.svc.cluster.local:9080     OK         mTLS    ---   

--- CLIENT     AUTHN POLICY          DESTINATION RULE
--- mTLS       default/istio-lab     ratings/istio-lab

Notice that the server and client communication between microservices is mTLS and that it is protected through strong identity—a standard that is progressing. You can find out more at https://spiffe.io/.

2. Run the istioctl describe pod command to check what type of traffic policy is being used. The output is useful for debugging/diagnostics purposes:

$ istioctl experimental describe pod $RATING_POD
Pod: ratings-v1-df666d977-l52gh
   Pod Ports: 9080 (ratings), 15090 (istio-proxy)
--------------------
Service: ratings
   Port: http 9080/HTTP
DestinationRule: ratings for "ratings"
   Matching subsets: v1
      (Non-matching subsets v2,v2-mysql,v2-mysql-vm)
   Traffic Policy TLS Mode: ISTIO_MUTUAL
Pilot reports that pod is STRICT (enforces mTLS) and clients speak mTLS
VirtualService: ratings
   1 HTTP route(s)

In the preceding section, we enabled mTLS at the istio-lab namespace level instead of enabling it at the global level. If we had enabled mTLS at the global level, we would have to allow the Kubernetes API server to communicate with Istio services without mTLS since there is no proxy sidecar running with the Kubernetes API server. 

3. The communication with mTLS between the Istio services and the Kubernetes API server can be disabled through a destination rule:

# Script : 08-disable-mtls-for-kube-apiserver.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: "api-server"
 namespace: istio-system
spec:
 host: "kubernetes.default.svc.cluster.local"
 trafficPolicy:
 tls:
 mode: DISABLE

4. Apply some destination rules so that communication between the Kubernetes API server and istio-system can happen when mTLS is set at the global level (note that we didn't set mTLS at the global level, so this step is optional and for information purposes):

$ kubectl -n istio-system apply -f 08-disable-mtls-for-kube-apiserver.yaml
destinationrule.networking.istio.io/api-server created

To recap, we covered the following security implementation topics: 

• • Through the Istio Ingress gateway, we enabled simple and mutual TLS so that TLS termination can occur at the Istio Ingress gateway. We kept the TLS mode as PERMISSIVE (through the MeshPolicy definition) to enable both text and TLS communication.
  • The downstream communication was still using a plaintext HTTP protocol until we enabled mTLS at the namespace level for the bookinfo application.
  • Now that mTLS has been implemented downstream of the Ingress gateway, service-to-service communication can be done through mTLS.

From your Chrome browser, from inside the VM, clear your local cache by pressing Ctrl + Shift + Del, click Clear Data, and close the Settings tab.

From your web browser within the VM, click on the second tab or go to https://bookinfo.istio.io. You will see a notice stating Your connection is not private, which is OK since our certificate is self-signed. Click Advanced and click to proceed with bookinfo.istio.io.

https://bookinfo.istio.io initiates a secure communication from the client (browser) to the Istio Ingress gateway. The client authenticated the server (the Istio Ingress gateway presented its X509 certificate to the browser), and downstream communication is done through mTLS. Between microservices, the client and server microservices will authenticate each other. Since the certificates are self-signed, the browser will complain about the connection being insecure, and that is OK. Normally, for the edge services (external-facing microservice), we will use a signed certificate instead of a self-generated certificate.

You will also notice that you can still run http://bookinfo.istio.io, where the communication between the client and the server is plaintext, but the server-side communication between microservices is using mTLS. The plain and TLS communication between the browser and Istio ingress gateway is allowed through the simple TLS mode, which is defined through the Istio primitive of a gateway. Run the kubectl -n istio-system get gw -o yaml command to confirm that the TLS mode for the bookinfo.istio.io host is SIMPLE.

5. Run https://httpbin.istio.io/ip from the Chrome browser. You will notice an error stating that the server could not prove that it is httpbin.istio.io. This is due to the fact that we had enabled TLS mode in the gateway as MUTUAL for httpbin.istio.io, which will require the client to present its key and certificate to the server so that a mutual authentication could occur. Previously, we used the following curl with the cacert, key, and cert parameters for mTLS to work for httpbin:

$ rm -fr ~/.pki

$ curl -HHost:httpbin.istio.io --resolve httpbin.istio.io:$INGRESS_PORT:$INGRESS_HOST --cacert $HOME/step/ca-chain.crt --cert $HOME/step/client.crt --key $HOME/step/client.key https://httpbin.istio.io/status/418

    -=[ teapot ]=-

       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`

If we want to enable mTLS for the browser, additional steps need to be performed.

The example given here is only for the Chrome browser in CentOS. Chrome uses shared NSS DB stored in $HOME/.pki/nssdb.

Make sure you have the certutil and pk12util utilities available. On CentOS, these can be installed using yum -y install nss-tools.

Exit from the root in the VM to get to the default user, which was used to log in to the system and the user. This will be running the Chrome browser from the Linux CentOS VM. Also, copy the contents of ~/httpbin.istio.io to a temporary location with proper permission for the regular user to import the root certificate and the client bundle. Let's get started:

1. Since we created a self-signed client certificate for httpbin.istio.io, we need to import the root certificate into the nss database:

$ certutil -d sql:$HOME/.pki/nssdb -A -n httpbin.istio.io -i $HOME/step/root-ca.crt -t "TC,,"

2. Create a client bundle using the client's key and a certificate in pk12 format:

$ openssl pkcs12 -export -clcerts  -inkey $HOME/step/client.key -in $HOME/step/client.crt -out httpbin.istio.io.p12 -passout pass:password -name "Key pair for httpbin.istio.io"

3. The password to create the client bundle is password, and the same must be used to import the client key bundle into the nss database using pk12util. You can choose a password of your choice:

$ pk12util -i httpbin.istio.io.p12 -d sql:$HOME/.pki/nssdb -W password
pk12util: PKCS12 IMPORT SUCCESSFUL

4. List the certificates in the nss database:

$ certutil -d sql:$HOME/.pki/nssdb -L
Certificate Nickname                      Trust Attributes
                                          SSL,S/MIME,JAR/XPI

httpbin.istio.io                          CT,,
Key pair for httpbin.istio.io             u,u,u

Run https://httpbin.istio.io/ip from the Chrome browser. A popup will appear where you can choose the certificate to authenticate to httpbin.istio.io and select httpbin.istio.io. Now, you'll be able to see the output.

This is an example of a secured authenticated communication in which both the client and the server authenticate each other. This is why this is known as mutual TLS as opposed to simple TLS. Mutual TLS is preferred for internal business applications such as bookinfo, where microservice-to-microservice communication is through mTLS. It shields the microservices, even in a zero-trust environment. It will be interesting to see the decline of VPN and firewall and the rise of secure authenticated communication in a zero-trust network.

The web client and server's mutual authentication is not typical in the internet world where a client (me) needs to validate whether the bank site that they are visiting is genuine, but the bank does not verify the authenticity of the client. This is how the internet works, and in one sense, it fails to establish the trust between a bank and its customer.  The preceding example shows mutual TLS between the client and the server, and the same is common for a business-to-business application.

With this, we have covered the first step of security authentication (who you are?). Next, we will cover the second step of security authorization (what can you do?).