VAULT Auto Unseal via AWS KMS

A resilient, highly-automated and highly available infrastructure are the quint-essential expectations of Site Reliability Engineering. In this blog, we will be discussing about Vault auto unseal feature using AWS KMS.

When we configure the Vault service, the Vault server will be in a sealed state. Vault knows where and how to access the storage but does not know how to decrypt any of the data from the storage. Unsealing is the process of obtaining the master key required to decrypt and read the data from storage. The unsealing is done using a series of three secret keys. Unfortunately if the server is in sealed state, Vault will not be able to decrypt the secrets, since it is in a sealed state.

Manual Unseal Process

By default, Vault is configured with Shamir secret algorithm. It is a form of secret sharing, where the secret is divided into shards and a certain threshold of shard is required to recreate the secret again. When the Vault server is initialised, the master key is divided into three keys by using the Shamir algorithm. Any two keys from the list of three keys are necessary every-time the Vault server needs to be unsealed. Once the Vault is unsealed it will be in the same state until:

• Vault service/Vault server is restarted

Even if we run multi-node Vault cluster, if the service is stopped on the active node, we will need to manually unseal the Vault server on the standby node to be able to make it work. To unseal Vault, we can use below command:

vault operator unseal

Lets discuss few scenarios where auto unseal will be very efficient:

• If the infrastructure is hosted on cloud (AWS/Azure/Google Cloud etc.) there will be possibility that server will be restarted as part of cloud maintenance activity or any other issue due to outage in the region and Vault will move to sealed state.

What will happen if the server is in sealed state:

• None of the new deployment will able to read the secrets from the Vault
• Application hosted on EC2 instances or pods, autoscaling will be blocked because it will not be able to read the secrets from Vault.

To avoid any such issue, we need the Vault service to be unsealed immediately either manually or automatically.

Auto Unseal

Hashicorp introduced auto unseal feature in Vault 1.0 open source in Dec 2018. Earlier this feature was supported only in Vault enterprise version. Auto unseal reduce the operational overhead of unsealing the Vault while keeping the master key secure. This feature delegates the responsibility of securing the master key from operator to trusted device or service (in this scenario we will use AWS KMS). Instead of only constructing the key in memory, the master key is encrypted with cloud based managed service.

Configuration

Step 1: Creating AWS KMS for Vault configuration in AWS console

Step 2: Create a new role in AWS and attach it to the Vault EC2 instance

Step 3: Configure vault.hcl

Step 4: Initialise the Vault server for the first time configuration by running below command

vault operator init

Notice that the vault status is already unsealed state.

If there is existing Vault server which is configured and initialised, we need to migrate it to auto unseal by following the steps below:

• Stop vault service

service vault start

• Update vault seal configuration

• Start vault service

service vault start

• Run vault unseal with migrate option

vault operator unseal -migrate

Once the migration process is completed, unseal key will be migrated to recovery keys and auto unseal will be enabled. To verify if the auto unseal is working fine, stop the vault service on active node and run vault status command on the standby node and verify that the Vault unseal state is set to false state.

To revert back repeat all the above steps with enabling disabled = true in Vault seal configuration step.

Conclusion

By using the auto-unseal feature of Vault, we can ensure that none of the applications are impacted when the Vault service goes into an unsealed state and this process is completely automated requiring no human intervention.

Join us

We are always looking out for top engineering talent across all roles for our tech team. If challenging problems that drive big impact enthral you, do reach out to us at careers.india@halodoc.com

About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek and many more. We recently closed our Series B round and In total have raised USD$100million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.