Part 1: Consuming secrets from an Azure Kubernetes Service (AKS) container – Linux

Overview

When a Kubernetes cluster is set up in an AKS environment, you can associate that with an AAD service principal or an MSI (Managed Service Identity). Usually, you would use this identity to access “cluster-specific” resources, e.g. Container Registry, Key vault storing cluster secrets, Storage accounts with additional artifacts, etc. But suppose you are running this cluster to host multiple micro-services managed by different teams, and each of these micro-services is atomic. That is, they are required to have an independent identity. The micro-service you are running in the cluster would need to access different resources during its life cycle, and they need to have a specific identity they can use their resources, which in most cases would be hosted in their subscriptions. Assuming you are running tens of service, if not hundreds in your cluster, how do you make independent identities available to each of these applications. Also, how can each container access secrets stored in the Azure Key Vault using this identity?

In this article, I am covering how you can solve this for a set of Linux containers in Azure Kubernetes Service (AKS). I will write another article shortly, on how to achieve something similar to Windows containers.

Acquiring and application-specific identity inside the Container

The Azure AAD Pod Identity provides the capability to get a service-specific identity available to each container, even though running in the same machine.

There are two components in the AAD Pod identity solution:

1. MIC – which runs as a CRD in the controller and is responsible for provisioning the identities on the nodes the pod is running on.
2. NMI – a copy of this is running in each node, and when pods in that nodes request for a token, it gets the token on its behalf.

Essentially, in brief, this is how it works:

1. One-time Provisioning:
   1. The application service creates an AAD user-assigned MSI in its subscription.
   1. They provide “operator” access to the cluster identity on their AAD user-assigned MSI created above.
   1. They onboard their user-assigned MSI into the cluster, under their namespace.
2. Associate the application with a selector defined in the AzureIdentityBinding CRD. An example YAML to show this is here.
3. When deployment has happened under that namespace, and the pod is created, the MIC hosted as controller pushed the user-assigned MSI to the VM node(s) the pods are scheduled to run. If the pod is destroyed and moves to another node due to eviction or a new deployment, the MIC takes care of removing the identity from the machine it no longer is running on.
4. When an application pod makes a call to acquire a token, traditionally the call is routed to the IMDS, but in this case, the call is routed to the NMI component running in the same node as a privilege DaemonSet, and the NMI on-behalf of the pods looks up at the associated user-assigned MSI from the Kubernetes API, and get a token on behalf through the IMDS. It does the routing by applying a network route policy in the IPTable for the pods.

A lot of details can be found here – https://github.com/Azure/aad-pod-identity#components

To install the AAD Pod Identity in your cluster, please follow the instructions given here – https://github.com/Azure/aad-pod-identity#getting-started

Once you have that, you should have the AAD Pod Identity components running in your clusters, and it should look like this:

The NMIs would run as DaemonSet, which means a single copy of it would be running in each node of your cluster, and the MIC would be running as a controller in the master nodes. There would be two instances of MIC running, but only one instance would serve request at a time, because during startup they go through a leader election process, and the other one stays active as a backup. It uses a set of CRD to define the client identity and its binding to the application namespace.

Using the Token to Access Azure Key Vault

The token acquired during the above step can be used to perform authentication to any of the Azure resources, but in our scenario, we will access the Azure Key Vault. Once you have the bearer token, there are two ways to get a secret from the Azure Key Vault.

Make REST GET call to Azure Key Vault

You can make a standard GET call to rest Key Vault API with the bearer token in the authorization header. An example request would be:

GET http://mykeyvault.vault.azure.net/secrets/TestSecret?api-version=7.0 HTTP/1.1
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtp……..….
Accept: */*
Cache-Control: no-cache
Host: mykeyvault.vault.azure.net
Accept-Encoding: gzip, deflate, br
Connection: keep-alive

Key Vault Flex Volume

The Key Vault flex volume has a simple job, it downloads the specified secrets from your Key Vault into a volume, specifically a tempfs volume and then mount that volume into your container. From then on, you can access all your secrets as file system objects in your mount. It can utilize a variety of identities to access the Key Vault resource, including standard service principals, MSIs of the VMSS/cluster or the best option is the user-assigned MSI which provides namespace level isolation.

To install the Key Vault flex volume driver, please follow the instructions here – https://github.com/Azure/kubernetes-keyvault-flexvol#getting-started

Once you install, it should look like this in your cluster:

Once the driver is installed, the application deployments can mount the flex volume and specify the driver, and the desired mount name. An example YAML for your pod deployment is given here.

I have a full working Golang application here, which utilized the AAD pod identity and Key Vault flex volume using user-assigned MSI.

General Guidelines to Follow while using these Solutions:

Namespace Validation

The current implementation of the AAD Pod identity does not enforce namespace validation. This means that if two applications in two different namespaces re-use the application selector, they might be able to request a token. So, it is recommended that you enable namespace validation for AAD Pod identity deployment in your cluster. You can enforce this at the MIC component level or the AzureIdentity CRD level. Read this for instructions on how to do this.

Choice of Identity Type

The Key Vault flex volume supports multiple types of identities, but based on my use, I believe the user-assigned MSI is the best choice. Here is why I think:

1. Using standard service principal, where credentials of the service principal would be created as Kubernetes secret and referenced in the deployment YAML. The disadvantage of this is – the secret must be manually added to the cluster, and this presents some challenges in managing the secret. The secret would have to be generated and download to the desktop by someone and added to the cluster, and there is no automation around this. The secret would expire someday and without manual action, it could cause an outage.
   1. Using VMSS system-assigned MSI is a better option then standard service principal as the credential management is taken care for you by Azure, and you do not have to worry about it. But the identity is common for the nodes in the same VMSS and all the containers running in those nodes would share the identity. Sharing identity is something that is not recommended.
   1. Using AAD Pod Identity with user-assigned MSI is the best approach that combines the advantages of the two above options, i.e. no credentials to manage and isolation of identities for each application.

Use of Default Namespace

The current deployment of AAD Pod Identity deploys everything in the default namespace. It is not a good idea to deploy pods in the default namespace, as eventually, it turns out to be a dumping ground for everything, and namespaces provide reasonable isolation for applications in terms of security and RBAC, etc.

Reference Image by Digest rather than by Tag

In the current deployment YAML of both AAD Pod identity and Key Vault flex volume, the images are referred with tags. When you are consuming an image from an external or public registry, it is recommended that you consume it by its digest, rather than the tag. Tags are easily replaceable. If the registry is breached and a malicious actor tries to replace the image you are referring to with a newer one with the same tag, you would start to consume it without realizing it. But if you reference it by the digest, the changed image will invalidate the digest, which is more secure. For example, instead of referring the component images as:

spec:
      containers:
      - name: mic
        image: "mcr.microsoft.com/k8s/aad-pod-identity/mic:1.3"

please refer images with the digest as:

spec:
      containers:
      - name: mic
        image: " mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:3317b3802ab12578b11e5af6be8901738b9790068044f45d6e2a277168085"

Identity Sharing

It is not recommended to share identities between applications (micro-services), even though they share resources. This ensures the atomic behavior of the application, without any cross dependencies. If an application is in a bad state or breached, the other applications are not affected, thus reducing the attack surface. It also ensures assigning minimum-privilege on each of the resources they share. So, plan to have an independent set of identities for all your applications and between rings (prod and pre-prod).