Role based access control

Kubernetes role based access control (RBAC) is the primary tool for defining the scope of what individual users or groups of users are permitted to do in a Kubernetes cluster. RBAC permissions are defined using roles and granted to users or groups of users using role bindings. Each role includes a list of resources (specified by resource type, cluster wide, within a namespace, or even a specific resource instance) and the permissions for each of the resources (e.g. get, list, create, update, delete, etc).

Many Kubernetes resources are namespaced, including deployments, daemonsets, pods and Kubernetes network policies. This makes the namespace an ideal trust boundary between teams. There are no set rules for how to use namespaces but one common practice is to use a namespace per microservice. RBAC can then be used to grant permission to manage the resources in the namespace to the team responsible for operating the corresponding microservice.

If security has been shifted-left, this would normally include permissions to manage the network policies that apply to the microservice, but not to manage any network policies that apply to microservices they are not responsible for.

If default-deny style best-practices are being followed for both ingress and egress traffic, then the team cannot forget to write network policies because the microservice will not work without them. In addition, since other teams will have defined equivalent network policies covering both ingress and egress traffic for the microservices they are responsible for, traffic is only allowed between two microservices if both teams have to have specified network policy that says the traffic is allowed. This further controls the degree of trust being delegated to each team.

Of course, depending on the degree to which security has been shifted-left, the responsibility for defining network policies may fall to a different team than the team responsible for operating the microservice. Again, Kubernetes RBAC can be used to easily reflect this split of responsibilities.

Limitations with Kubernetes network policies

There are a couple of limitations it is worth being aware of when using RBAC with Kubernetes network policies in a shift-left environment.

• Default deny style policies need to be created per namespace at the time the namespace is provisioned. The team responsible for defining network policies for the microservice would also have the ability to modify or delete this default policy if they wanted to.

• Network Policies are IP based and you cannot use fully qualified domain names (FQDN). This can be a limitation especially when defining policies to resources external to the cluster.

• Kubernetes RBAC controls access to resources but does not constrain the contents of resources. Of particular relevance in the context of network policies is pod labels since these are used as the primary mechanism for identifying other microservices in network policy rules. So for example, if one team has written a network policy for their microservice with a rule allowing traffic to it from pods with a particular label, then in theory any team with permission to manage pods could add that label to their pods and get access to the microservice. This exposure can be reduced by always using namespace sectors within policy rules and being selective as to which teams have permissions to change namespace labels.

If standardized policy and label schemas have been defined and the teams are trusted to follow them then these limitations are more of a theoretical rather than practical issue. However, for some organizations, they may represent genuine issues not sufficient for their security needs.

These organizations may therefore want to leverage additional capabilities beyond Kubernetes RBAC and Kubernetes network policies. In particular:

• Richer network policy implementations that support additional network policy types, match criteria, and non-namespaced network policies which open up more options for how to split responsibilities and RBAC across teams.

• Admission controllers to enforce controls on a per field level within resources, for example to ensure a standardized network policy and label schemas as followed, including limiting teams to using particular labels.

We will now review network policy implementations that extend the Kubernetes network policy and how you can use the same to manage trust.

Richer network policy implementations

Some network policy implementations support both Kubernetes network policies and their own custom network policy resources that can be used alongside or instead of Kubernetes network policies. Depending on the implementation, these may open up additional options for how to split responsibilities and use RBAC across teams. There are vendors that support richer network policy implementations that support the Kubernetes network policy and add more features (e.g., Weave, Kube-router, Antrea, Calico). We encourage you to review these and choose the best one that meets your needs. In this section we will look at the concrete example using Calico as it is the most widely deployed container network plugin.

Calico supports the Kubernetes network policy feature set, plus its own Calico network policy resources that can be used alongside Kubernetes network policies. There are two types of Calico network policy, both under the projectcalico.org/v3 API group:

• NetworkPolicy - These policies are namespaced (just like Kubernetes network policies)

• GlobalNetworkPolicy - These policies apply across the whole of the cluster independent of namespace.

Both types of Calico network policy support a common set of capabilities beyond Kubernetes network policies, including:

• A richer set of match criteria than Kubernetes network policies, for example with the ability to match on Kubernetes Service Accounts.

• Explicit allow, deny or log actions for policy rules, rather than Kubernetes network policy actions which are implicitly always allow.

• Precedence ordering to define the evaluation order of the network policies if multiple policies apply to the same workload. (Note that if you are just using Kubernetes network policies, or Calico policies only with allow actions in them, then evaluation order doesn’t make any difference to the outcome of the policies. However, as soon as there are any policy rules with deny actions, ordering becomes important.)

We want to mention that there are other network policy implementations that extend the Kubernetes network policy like Antrea which offers ClusterNetworkPolicy (similar to GlobalNetworkPolicy)

The following sample shows how you can implement network policies using Kubernetes RBAC. In the example below you can control network access based on the labels assigned to a service account. In Kubernetes pods have service accounts associated with them and therefore pods can be identified by service accounts. You should use RBAC to control which users can assign labels to service accounts. The network policy in the example below uses the labels assigned to service accounts to control network access. In the following example, pods with an intern service account can communicate only with pods with service accounts labeled, role: intern:

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: restrict-intern-access
  namespace: prod-engineering
spec:
  serviceAccountSelector: 'role == "intern"'
  ingress:
    - action: Allow
      source:
        serviceAccounts:
          selector: 'role == "intern"'
  egress:
    - action: Allow
      destination:
        serviceAccounts:
          selector: 'role == "intern"'

This way you can extend the concept of RBAC which controls service accounts access to a Kuberntes resource to network access. It is a 2-step process, RBAC is used to control label assignment to service accounts and a label based service account selector is used to control network access. These additional capabilities can be leveraged alongside Kubernetes network policies to more cleanly split responsibilities between higher-level cluster ops or security teams, and individual microservice teams.

For example:

• Giving the cluster ops or security team RBAC permissions to manage Calico network policies at the cluster-wide scope so they can define basic higher-level rules that set the overall security posture of the cluster. For example, a default deny style app policy (as discussed earlier in the “Network policy best practices” section) and policies to restrict cluster egress to specific pods.

• Giving each microservice team RBAC permissions to define Kubernetes network policies in the microservice’s namespaces so they can define their own fine-grained constraints for the microservices they are responsible for.

On top of this basic split in network policy RBAC permissions, the cluster ops or security team can delegate different levels of trust to each microservice team by defining rules using namespaces or service accounts labels, rather than simplifying matching on pod labels. For example, defining policies to restrict cluster egress to specific pods using service account labels and giving the individual microservice teams to use, but not edit, any service accounts assigned to their namespace. Through this mechanism some microservice teams may be granted permission to selectively allow cluster egress from some of their pods, while not offering other teams the same permissions.

Figure 6-1 provides a simplified illustration of how these ideas can be combined together.

Figure 6-1. An example of implementing trust boundaries with network policy

While the above capabilities are reasonably powerful, in some organizations the required split of responsibilities across teams may be more complex, particularly where there are more layers of teams. For example, a compliance team, security team, cluster ops team, and individual microservice teams, all with different levels of responsibility. One way to more easily meet these requirements is to use a network policy implementation that supports the notion of hierarchical network policies.

There are some commercial implementations that support hierarchical network policy using policy tiers. A similar concept (Hierarchical namespaces and policies) is also being discussed in the Kubernetes community. RBAC for each tier can be defined to restrict who can interact with the tier. In this model, network policies are layered in tiers which are evaluated in a defined order, with as many tiers as required to match the organizational split of responsibilities. RBAC for each tier can be defined to restrict who can interact with the tier. The network policies in each tier can make allow or deny decisions (that terminate evaluation of any following policies), or can pass the decision on to the next tier in the hierarchy to be evaluated against the policies in that tier.

Figure 6-2 provides a simplified illustration of how these capabilities can be used to split responsibilities across three distinct layers of responsibility within an organization.

Figure 6-2. Implementing hierarchical network policies using tiers.

Admissions controllers

We have already discussed the value of defining and following standardised network policy and label schemas. The approaches for splitting responsibilities between teams discussed above are oriented around resource and namespace level RBAC, with teams having freedom to whatever they want within the resource and namespace scopes they are allowed to manage. As such they do not provide any guarantees that any such schemas are being followed by all teams.

Kubernetes itself does not have a built in ability to enforce restrictions at this granular level, but it does support an Admission Controller API which allows third-party admission controllers to be plugged into the Kubernetes API machinery to perform semantic validation of objects during create, update, and delete operations. You can also use admission controllers, known as mutating admission controllers for modifying objects that are admitted.

For example, in the context of network policy:

• Validate that network policies have both ingress and egress rules to comply with the best practices the organization is trying to follow.

• Ensure every pod has a specific set of labels to comply with the labelling standards the organization has defined.

• Restricting different groups of users to using specific label values.

But admission controllers have security use cases beyond network policy too. For example, Kubernetes Services include support for specifying an arbitrary IP address to be associated with the service using the Service’s ExternalIP field. Without some level of policing this is a very powerful feature which could be used maliciously to intercept pod traffic to an IP address and redirect it to the Kubernetes service by anyone with RBAC permissions to create and manage Kubernetes Services. Policing this with an admission controller might be essential depending on the level of trust within the teams involved.

There are a few options for admission controller implementations, depending on the skill sets and specific needs of the organization:

• Using a pre-existing third-party admission controller that specializes in the specific controls the organization needs, if one exists.

• Writing a custom admission controller optimized for the organization’s needs.

• Using a general purpose admission controller with a rich policy model that can map to a broad range of use cases.

For many scenarios choosing a general purpose admission controller gives a good balance of flexibility versus coding complexity. For example, Kyverno which has a policy engine specifically designed for Kubernetes, or an admission controller built around Open Policy Agent (OPA), where the policy model has flexible matching and language capabilities defined using Rego.

While admission controllers are very powerful, it is generally recommended to only implement them if you genuinely need them. For some organizations, using admission controllers in this way is overkill and not required given the levels of responsibility and trust across teams. For other organizations, they can be essential to meet internal compliance requirements, and the case for using them will be very clear.

Conclusion

Kuberentes security is different as it needs to be implemented by various teams and needs collaboration between teams. We covered the following key concepts:

• You should use RBAC and network policy to define boundaries that will help you manage activities across teams.

• You can extend the concept of RBAC to control network access by leveraging service accounts in network policy to help you manage trust

• How to use Admission controllers to help control access and implement trust boundaries across various teams.

• The importance of collaboration between the development, platform and the security teams, working together to implement security.