Building Infrastructure as Code (IaC) using Terraform at Halodoc

At Halodoc, our company's cloud service provider is AWS. Terraform is used to provision all of the AWS cloud resources using IaC (Infrastructure as Code). We use the GitOps workflow to manage all of our infrastructure and settings, giving developers the freedom to run the pipeline automatically.

In this blog, you will learn to create Terraform projects to gain an understanding of the IAC approach and how it is used in practise to accelerate the creation and deployment of reusable and scalable infrastructure architectures.

Difficulties encountered when manually provisioning cloud resources through the AWS console

The difficulty is that SRE engineers are, well, too human. They commit human errors, lose focus, grow weary, and make mistakes due to fat fingers. They become unmotivated when performing the same tasks repeatedly and are also pricey.

Here are some of the most key challenges we've seen when configuring AWS resources via the console.

• There are chances of human errors while configuring the cloud resources.
• Creating the resource manually is a time consuming task.
• It is difficult to keep track of the Cloud resources you have created.
• Replicating the entire infra from one region to another will be painful.
• With manual AWS resource provisioning, no version control strategy.
• Replicating resources for creating a separate environment would be a painful job.
• It is difficult to find the changes made to existing infrastructure in order to roll them back if they were provisioned via the AWS console.

We implemented IaC at Halodoc to combat this. What is Infrastructure as Code?

IaC, or infrastructure as code, is the phrase I just used. This means that, as opposed to setting up the infrastructure manually, you set up the entire infrastructure using some coding language.

IaC enables users to codify their infrastructure. It allows users to:

• Automate the infrastructure management.
• Create repeatable, version-controlled configurations.
• Integrating them as a part of the CI/CD pipeline.
• Assists users in reviewing changes prior to generating or making changes to cloud resources.

Using a source control system like Git, IaC enables the automated integration of infrastructure changes into the CI/CD pipeline. It provides automated infrastructure modifications, quick rollbacks of changes, and auditable updates.

Benefits of IaC

To someone who is constantly focused on the end users of the product rather than the underlying infrastructure, the advantages of investing that time and resources may appear hazy when using infrastructure as code.

Here's why we should utilise IaC using Terraform to its full potential

• Documented Architecture:
  As resources are defined in code, we can keep track of what we are creating.
• Consistent Environments:
  Since there is no human meddling, the infra that has been created is constant.
• Automated deployment:
  Creation of these resources are quick and automated.
• Source Controlled:
  Versioning is easy because the infrastructure code is source controlled.
• Reusable components:
  Since the scripts have already been written, replicating the resources to create a new environment may be done very quickly.

What is Terraform?

HashiCorp's Terraform is an open-source infrastructure as a code tool. Users can specify cloud resources in human-readable configuration files that are simple to reuse, distribute, and version.

By allowing users to express the expected state of resources without having to specify the precise methods to get there, Terraform, a declarative tool, further simplifies the user experience. Terraform controls the changes that must be made to the infrastructure in order to accomplish the intended outcome.

Terraform is a platform-agnostic technology, it can be used with any cloud resource provider that supports it. This is done via Terraform interacting with cloud provider APIs. When a configuration is made using Terraform, it will use the API to connect to the required platform and make sure the specified modifications are applied to the intended platform. Users may configure resources from top cloud providers like Azure, AWS, GCP, and Oracle Cloud to more niche platforms like Kubernetes.

Terraform blocks and fundamentals

Variables

These are used as inputs for Terraform Resources.

Provider

Terraform relies on plugins called “providers” to interact with remote systems.

Data Sources

In Terraform configuration, data sources enable data to be fetched or computed for usage elsewhere. Utilizing data sources enables a Terraform configuration to leverage data defined by another Terraform configuration or by data sources outside of Terraform.

Resources

The most significant component of the Terraform language is resources. One or more infrastructure items are described by each resource block.

Output

Outputs resemble the return values for a Terraform module.

Working with Terraform

One of the simplest workflows for managing any type of infrastructure is Terraform's, which just has three phases. It gives users the freedom to modify the workflow to suit their own implementation requirements. Let us take an example of simple SNS resource creation to understand the workflow.

1. Write

The first stage of the workflow is where users create the configurations to define or modify the underlying resources. This writing part can be facilitated either through HasiCorp Configuration Language (HCL). The configuration below creates an AWS SNS resource, and the terraform init command allows you to initialise the resource.

2. Plan

In the next step of the procedure, Terraform examines the configuration files and develops an execution strategy. It allows users to view the precise changes that will be made to the underlying infrastructure as a result of the creation, resource allocation, modification and deletion of new resources. This command terraform plan is used to ease the creation of the SNS resource plan.

3. Apply

After the user has approved the modifications, the workflow moves on to this stage, which only takes place if the plan is satisfactory. Terraform will implement the modifications in a precise order while keeping in mind all resource dependencies in order to achieve the desired state. Whether or not you specify dependencies in the configuration, it will still occur. The platform's resource dependencies will be automatically identified by Terraform, and the modifications will be implemented without any problems. The command terraform apply makes applying simple.

Terraform uses the state to monitor all infrastructure changes and identify config drifts. Upon initial execution, a state file will be created, and as new changes are made, the state file will be updated. This state file may be kept locally or in a system with remote backup, such as an S3 bucket. To identify the resources it is in charge of managing and to keep track of changes to the infrastructure, Terraform constantly refers to this state file. The terraform state is a JSON file, the below state file holds the configuration of created SNS.

4. Destroy

To destroy the created resource on cloud, we can make use of terraform destroy, the below image refers to the destruction of single SNS resource

Terraform vs. CloudFormation, Heat, etc.

The specifics of an infrastructure can be formalised into a configuration file using tools like CloudFormation, Heat, etc. The configuration files enable the infrastructure to be established, updated, and deleted elastically. The issues they resolve are what motivate Terraform.

Similar configuration files are used by Terraform to describe the infrastructure setup, but it goes a step further by being cloud-independent and allowing the combining and composition of many providers and services. Terraform, for instance, can be used to coordinate the simultaneous operation of an AWS and OpenStack cluster while allowing the integration of third-party service providers like Cloudflare and DNSimple to offer CDN and DNS services. Instead of just the portion that is contained within a single provider, Terraform is now able to represent and manage the full infrastructure together with its auxiliary services. Instead of requiring operators to use separate, incompatible tools for each platform and service, it provides a single, uniform syntax.

Terraform utilises the idea of an execution plan to further distinguish between the planning and execution phases. The configuration is consulted to create an action plan while the present state is refreshed by terraform plan. All actions, including which resources will be created, destroyed, or modified, are outlined in the plan. To make sure it is exactly what is intended, operators can inspect it. The plan's dependent ordering can be seen by using a terraform graph to display it. After the plan has been written down, the execution phase can only include the acts in the plan. Other tools combine the planning and execution phases, which forces operators to think through the ramifications of a change in a way that, in large infrastructures, is quickly intractable. Terraform enables operators to make changes with confidence since they are aware of every step along the way.

Let’s look at why so many people appreciate Terraform

• Easy rollbacks
  The majority of infrastructure configurations may be simply and safely rolled back by users without requiring laborious reconfigurations because all configurations are version controlled and the state is managed.
• Integration to CI/CD
  While Terraform offers a straightforward three-step workflow that can be quickly put into any CI/CD pipeline, IaC can be integrated into any pipeline. The management of the infrastructure benefits from being fully automated.
• Declarative nature
  Users can declare the desired end state using a declarative tool, and the IaC tools will automatically complete the necessary processes to fulfil the user configuration. In contrast to other imperative IaC tools, which require users to specify the precise steps needed to obtain the desired state, this one does not demand this.
• Platform agnostics
  Platform-specific IaC tools include AWS CloudFormation and Azure Resource templates. However, Terraform enables users to manage infrastructure across platforms with applications that leverage numerous tools, platforms, and multi-cloud architectures with just one tool.
• Reusable configurations
  Terraform promotes the development of reusable configurations, which may be applied to several settings at once. Additionally, Terraform supports the use of modules to create reusable components within configuration files.
• Managed state
  Since state files record every change made to the environment, any unauthorised alterations won't take place unless the user specifically instructs them to. To ensure that the target state is met at all times, it can be further automated to recognise any configuration drifts and to automatically correct the drift.

Want to use terraform?

Just brew install terraform !!
Simply installing Terraform in your local environment will allow you to use it right away. Windows, Linux, and macOS environments are supported by Terraform. Users have the choice of manually installing it using a pre-compiled binary or using a package management like Homebrew on a Mac, Chocolatey on Windows, or Apt/Yum on Linux.

Halodoc's adherence to terraform best practises

• Modular approach
  writing modules for each resource type(ec2, rds...) and using them to create resources across different aws accounts.
• Variabilize the configuration of resource
  with the variable set being created for a resource, it can be reused to create similar resource types under same infra.
  For eg, to create lambda the variable set looks like

• Storing secrets in vault
  anyone who have the access to source repository can view the secrets of the resources like rds. Storing the secrets in vault and fetching it by using the data block in tf helps in secrets management.

• Storing the state in s3 and having the dynamodb locks
  Store the state file in s3 bucket instead of storing them in local(git) which helps avoid losing tfstate and doing manual changes on tfstate. Adding the locks in the backend will lock your state for all operations that could write tfstate. This prevents others from acquiring the lock and potentially corrupting your state
• Separate state files for each resources
  having separate state files for each resource types(rds, sns..) helps in reducing the time of execution if there are more number of resources.
  Automated job to create resources across each AWS account with having separate state looks like

Summary

Why use clicks and manual processes to provision infrastructure when writing application code? Infrastructure configuration can be integrated into the same development processes with the help of infrastructure as code technologies like Terraform, allowing for testing, standardisation, and scalability. A wide range of learning and getting started resources are available in the contemporary Infrastructure as Code environment.

We began utilising this tool at Halodoc a year ago, and it has aided in faster resource creation as well as resource migration or replication from one AWS account or region to another. Because it's code, all the wonderful advantages of the CI/CD pipeline and code reviews apply. IaC is written by one engineer, while peer reviews are done by another engineer. In this instance, a merge request ought to include a Terraform plan as well as HashiCorp Configuration Language (tf code). The inclusion of Terraform in CI/CD pipelines frees the engineer from having to constantly check the status of the Terraform apply in a terminal window and allows them to concentrate on other intellectually taxing tasks.