Secrets Management in Kubernetes

Date : 2021-10-13

Sam Pecsek

Secrets management is a necessary task for all applications and platforms that deal with confidential information. This includes Kubernetes, which can deploy apps along with all the secrets needed for proper operation. The deployment of secrets into apps is trivial, but making sure the secrets are securely stored and encrypted is a more complex undertaking.

The official Kubernetes documentation defines a 'Secret' as:

An object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret means that you don't need to include confidential data in your application code.¹

Separating the private data from the application code is a great start, and Kubernetes takes care of this step completely. The catch is that

Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd)²

This means that anyone with access to a Kubernetes cluster’s API can read all the Secrets with minimal effort.

The following command will display the Secret "production-db-password" with base-64 encoded contents:

kubectl get secret production-db-password -o jsonpath='{.data}'
> {"password":"dW5lbmNyeXB0ZWRQYXNzd29yZA==","username":"cm9vdA=="}

Unhash the values and the username and password are displayed in plain text:

echo 'dW5lbmNyeXB0ZWRQYXNzd29yZA==' | base64 --decode
> unencryptedPassword

echo 'cm9vdA==' | base64 --decode
> root

While RBAC can help secure the API, parties who gain access to etcd can read all the cluster’s secrets, and users who have permission to create a pod can read all secrets in the namespace. So, secrets need to be encrypted.

Secrets need to be encrypted

Kubernetes provides a native method of locally encrypting secrets. However, it is more profitable to skip this step and simply use an external or third party solution because by the documentation’s own admission, the local encryption mechanism only provides a moderate improvement over no encryption at all!

“Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption.” ³

Fortunately, there are numerous solutions available which proficiently fill the role of securely storing and encrypting Kubernetes Secrets.

Cloud solutions

Many cloud providers maintain Kubernetes encryption provider plugins to store Secrets with encryption at rest as well as mechanisms to store secrets directly in the cloud provider's own secrets management service. For example, users of AWS can choose between integrating AWS' Key Management Service (KMS)⁴ via the AWS Encryption Provider for Kubernetes or AWS' Secrets Manager via the AWS Secrets and Config Provider (ASCP)⁵

• When integrating a KMS plugin for a specific cloud provider, Kubernetes will use the KMS service in that cloud provider to encrypt Secrets in the Kubernetes cluster. The Secrets are still present and readable via API, but cannot be decrypted without KMS keys. When a new Pod is deployed, an API call is made to the cloud provider to decrypt the Secret for the Pod to read.
• Using a cloud secrets manager removes Secret storage from the cluster and simply fetches the Secrets from the cloud provider's secrets manager when a Pod is deployed. When a new Pod is deployed, the cloud provider's secrets manager is queried for the specific Secret in the Kubernetes manifest and the Secret is inserted in a specified location within the Pod's filesystem.

Many cloud providers maintain similar tools to accomplish this encryption or external storage of Kubernetes Secrets. Google Cloud KMS and Azure Key Vault follow similar principles to AWS KMS with regards to their integration with the Kubernetes KMS provider.

Self-hosted solutions

Administrators who want a self-hosted solution or a solution which can also be used for non-Kubernetes purposes will want to look into Hashicorp's Vault⁶ or Bitnami's Sealed Secrets⁷.

Hashicorp Vault is a free, open-source tool for managing and delegating secrets to any application or user who can query its API. Vault can be used in a hybrid environment where Kubernetes is not the only client in need of secrets management. There are multiple convenient methods of integrating Vault secrets into a Kubernetes cluster. Vault can be compared to AWS Secrets and Config Provider in that when a Pod requests a Secret, it is fetched from Vault.

Bitnami's Sealed Secrets is a tool exclusively for Kubernetes. A controller is installed in the Kubernetes cluster which manages the encryption of Secrets in the cluster. Since the Secrets can only be decrypted by the controller, the attack surface for the Secrets becomes much smaller to the point where the Secrets can be stored in a public git repository and be considered safe. Sealed Secrets can be compared in some ways to the AWS KMS encryption provider since the Kubernetes Secrets are stored in etcd, but in a secure and encrypted form.

Knowledge is power (and peace of mind)

There are many excellent secrets management tools available for Kubernetes, but all of them must be understood and used properly within the scope of their intended use cases to be used effectively.