Multiple PKS components need to communicate with the NSX-T manager. A PKS control plane VM using an NSX-T superuser principal identity certificate as an authentication mechanism is needed to create a T1/LS for each K8s cluster node network and an LB instance for each K8s cluster.

BOSH uses credentials as an authentication mechanism to tag all of a VM's logical ports with a special BOSH ID tag and NCP pod. It uses the NSX-T superuser principal identity certificate as an authentication mechanism to create T1/LS for each namespace, a SNAT rule on T0 for each namespace, and an LB virtual server for each K8s service of the type LB.

The following is a list of the NSX-T objects that are created, for each K8s cluster.

When a new K8s cluster is created, the following NSX-T objects are created by default:

• NSX-T LS:
  • One LS for K8s master and worker nodes
  • One LS for each K8s namespace, that is, kube-public, kube-system, and pks-infrastructure
  • One LS for the NSX-T LB associated with the K8s cluster
• NSX-T T1:
  • One T1 for K8s master and worker nodes (called cluster-router)
  • One T1 for each K8s namespace (default, kube-public, kube-system, and pks-infrastructure)
  • One T1 for the NSX-T LB associated with the K8s cluster
• NSX-T LB:
  • One NSX-T LB small instance, containing the following objects:
    • One virtual server to access the K8s control plane API (with port 8443)
    • One server pool containing the three K8s master nodes
    • One virtual server for the ingress controller (HTTP)
    • One virtual server for the ingress controller (HTTPS)
    • Each virtual server is allocated an IP address derived from the PKS Floating IP Pool

When a new K8s cluster is created, the following NSX-T objects are created, by default:

• NSX-T DDI/IPAM: A /24 subnet from the nodes IP block will be extracted and allocated for the K8s master and worker nodes.
• NSX-T DDI/IPAM: A /24 subnet from the PODs IP Block will be extracted and allocated for each K8s namespace (default, kube-public, kube-system, and pks-infrastructure).
  • NSX-T T0 router:
    • One SNAT rule created for each K8s namespace (default, kube-public, kube-system, pks-infrastructure), using one IP from the Floating IP Pool as the translated IP address.
    • One SNAT rule created for each K8s cluster (in the case that NAT topology is used), using 1 IP from the Floating IP Pool as the translated IP address. The K8s cluster subnet is derived from the nodes IP block, using a /24 netmask.
  • NSX-T DFW:
    • One DFW rule for kubernetes-dashboard: Source=K8s worker node (hosting the dashboard POD/Destination= dashboard POD IP/Port: TCP/8443/Action: allow
    • One DFW rule for kube-dns: Source=K8s worker node (hosting the DNS POD)/ Destination = DNS POD IP/Port: TCP/8081 and TCP/10054/Action: allow