Tenancy structure

A tenant is an organizational structure that allows multiple teams to share the same Kubernetes cluster. Each tenant is:

• Allowed to create a certain number of namespaces in the cluster
• Limited by resource quota
• Configured with a secret store backed by Vault
• Provided with an OpenTelemetry instrumentation
• Optionally provided with an OpenTelemetry - Splunk instrumentation

We deploy Kyverno policies to manage the tenants. We will provide a short description below of each of the policies.

Create tenant-upgrade-user ServiceAccount and ClusterRoles

At first, the following resources are created:

Tenant-upgrade-user ServiceAccount

On top, you can see that our admin ServiceAccount is created:

• The ‘tenant-upgrade-user’ ServiceAccount
• The ‘flow:tenants:upgrader’ ClusterRole which grants permissions to perform CRUD operations for Kyverno ClusterPolicies
• The ‘tenants-upgrade-user’ ClusterRoleBinding which binds the ‘tenant-upgrade-user’ ServiceAccount to the ‘flow:tenants:upgrader’ ClusterRole

Default ClusterRoles

Next to the ServiceAccount with corresponding ClusterRole, we create five other ClusterRoles:

ClusterRole	Kubernetes resource	Permissions
flow:tenants:postgres:user	servicebindings serviceinstance servicemonitors	get, patch, create, delete
flow:tenants:namespace-creator	namespaces	create, list, watch
flow:tenants:namespace-remover	namespaces	get, delete
flow:tenants:argocd	applications applicationsets	get, list, create, patch
flow:tenants:traefik	middlewares ingressroutes	*

Delete existing ClusterPolicies

The second phase consists of the jobs that delete the existing ClusterPolicies:

These ClusterPolicies are deleted by the ‘tenants-upgrade-user’ ServiceAccount created in the first phase.

Create tenant RBAC, Gitlab variables and ClusterPolicies

The final phase consists of creating everything for the tenant:

Tenant RBAC

A service account is created for the tenant. Based on this service account, a secret is generated containing a token for that service account. By using a ClusterRoleBinding, the service account receives the permissions to create namespaces.

ClusterPolicies

The following ClusterPolicies are created:

• Allow the tenant ServiceAccount to create namespaces with a specific prefix (e.g. tenant1-*)
• Only allow that tenant/service account (and cluster admins) to delete namespaces that they created
• Only allow a certain configurable number of such namespaces to be created per tenant
• Add a (configurable) ResourceQuota to each such namespace
• Add a label to each such namespace to identify the tenant that created it
• Deploy generic OpenTelemetry Instrumentation policies
• Deploy Splunk OpenTelemetry Collector policies
• Deploy Splunk OpenTelemetry Instrumentation policies
• Deploy the allowed storage class policy, limiting access to storageClass objects in the cluster