Kubernetes and Vault integration

Keeping secrets entirely secure has always been a challenging task for any organisation, whether its account credentials or application related secrets. It's not a best practice to store secrets in plain text files or embed them into source code. This is to avoid accidentally exposing any application token or  credentials into source code which leads to other security risks.  Every organisation is focused on securing their infrastructure and application assets and ensure that they do not expose any credentials, tokens or application specific secrets, even accidentally.

There are several techniques to enable the secure storage of secrets, without the threat of exposure, one such simplest approach being storing the secrets in environment variables. However, it does come with its own set of pitfalls and is not a viable solution for a production scale environment on Kubernetes. One of the most common and recommended approach for storing secrets in a secure manner, is to use a secure secrets store such as Vault, coupled with Kubernetes integration.

We are already using a vault server to store the secrets, but wanted to improve upon the current approach in the best manner possible.

Vault agent Injector

We need a solution, especially for secrets that are dynamically getting updated, to securely integrate secrets without a deployment. Hashicorp provides a vault-agent as a sidecar solution which fits into this requirement.

The Vault Agent Injector alters pod specifications to include Vault Agent containers that render Vault secrets to a shared memory volume using Vault Agent Template. By rendering secrets to a shared volume, containers within a pod can consume Vault secrets without being Vault aware.

The injector is a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the pod if a particular annotation (“vault.hashicorp.com/agent-inject: "true"”) exists within the request. This functionality is provided by the vault-k8s project and can be automatically installed and configured using the vault-helm chart. If you are configuring the vault for the very first time, you can proceed with the vault helm chart.

Since we already have a vault HA cluster that is EC2 based and did not want to migrate to Kubernetes as of now. Our idea is to use our existing vault server and integrate a vault agent container with a vault running on EC2.

Workflow:

1. User sends the deployment creation request, which goes to kube-api server.
2. Deployment object resides in API-Server Worker Queue.
3. The Vault-injector mutation webhook controller receives the deployment object request from worker queue.
4. The Controller mutates the deployment with init and sidecar containers.
5. The init container sends the request to vault server to pre-populate secrets from secret path given in annotations.
6. The vault server then verifies the service account of k8s deployment from specific k8s cluster configured in vault agent.
7. After successful verification of service account, Vault generates the child token to read the secrets.
8. Init-container uses the child token vault to fetch the secrets and places in shared pod volume /vault/secret/
9. The sidecar container keeps a vigil on the child token expiration and renews the same as needed , and refreshes the secrets.

Implementation Walk Through

The complete solution can be implemented in four steps detailed below:

1. Vault agent integration with external vault server
2. Enabling Kubernetes vault authentication
3. Injecting secrets into application
4. Integration with helm chart

Step 1: Vault agent integration with external vault server

We only  need to configure vault-agent with our vault server and don’t need to install the complete helm chart of vault:

If the vault server is behind nginx or reverse proxy use the nginx url with port configured in reverse proxy. For example: if my vault nginx url is www.vault-server.com, vault-agent configuration will like this:

You will able to see vault-agent-injector pod running in your default namespace:

Vault agent injector works at pod create and update events of Kubernetes. Kubernetes controller looks for pod annotation vault.hashicorp.com/agent-inject: true and if finds it, it will alter pod with other annotations available for the pod.

Since vault-agent is a webhook which works with Kubernetes mutation webhook controller, at minimum Hashicorp vault has provided volume mounted at /vault/secrets and will be used by the Vault Agent containers for sharing secrets with the other containers in the pod.

Step 2: Enable Kubernetes Vault Authentication

Vault provides authentication via a Service account. We at Halodoc configured each microservice/application with their own service account to avoid accessing secrets with a single service account. But first, we needed to configure Kubernetes vault authentication. Vault provided a Kubernetes authentication method to enable vault communication with Kubernetes using a service account.

Enabling Kubernetes vault authentication

We are using a default /config endpoint for the vault to communicate with Kubernetes.

Create a service account vault_auth and enable RBAC using clusterRoleBinding to grant access cluster-wide

Get service account value created earlier

Get the JWT_TOKEN value to access TokenReview API

Get the Kubernetes certificate value

Configure vault auth on /config endpoint using the above values.

Step 3: Injecting secrets into application

Now we need that application should be able to read the vault secrets using a vault-agent as a sidecar container. Hashicorp vault provides two methods to use vault-agent integration as sidecar to read secrets:

• the vault.hashicorp.com/agent-inject-secret annotation, or
• a configuration map containing Vault Agent configuration files.
  - (Note: Only one of these methods can be used at a time).

We have configured our application to access secrets using vault.hashicorp.com/agent-inject-secret annotation.

First create a vault secret path to store all credentials.

Put secrets into secrets/ path

Create vault policy to read secrets from defined path

And then create service account in desired namespace

Last vault role to bound service account and policy for authentication.

Note: if the application is running in a default/different namespace, create your service account in respective namespace and change namespace value in above vault role definition.

Create deployment manifest

Get list of pods running n myapp-ns namespace

Currently there is no annotation applied to pod for vault injector and if we look for /vault/secrets volume in pod we won’t get anything

Let’s apply the vault annotation and apply the same to our running application myapp

Let’s run the command again to get the content from /vault/secrets path:

So as we can see the secrets are available in the application pod in the shared volume mount, the secrets can be templated based on the application need. Like the secrets can be called in the environment variable in application. Below is the example to source the /vault/secrets/myapp.txt file in command “args” in pod container definition.

Step4: Integration in Helm Chart

By following the above steps, we have deployed applications on the Kubernetes cluster. However if there is an existing Helm chart and application is deployed on K8S. In that case other than injecting the vault-agent file to deployment, we can also modify the existing helm chart. Below are the changes required:

Update values.yaml with below lines of code:

Integrating this pod metadata annotation in deployment.yaml:

Conclusion

By doing this POC and integrating in our K8S environment, we conclude that vault agent as a sidecar injector is the best approach we have identified so far. This also ensures to keep the environment separate and by using the vault policy and restricting service account bound for the respective application.

References:

https://learn.hashicorp.com/vault/identity-access-management/vault-agent-k8s
https://learn.hashicorp.com/vault/kubernetes/sidecar
https://www.vaultproject.io/docs/platform/k8s/injector/examples

Join us

We are always looking out for top engineering talent across all roles for our tech team. If challenging problems that drive big impact enthral you, do reach out to us at careers.india@halodoc.com

About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek and many more. We recently closed our Series B round and In total have raised USD$100million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.