Reverse Terraforming Using Terraformer

At Halodoc, we manage Infrastructure as code (IAC) by writing code rather than manually configuring individual servers and infrastructure components. This approach allows for greater consistency, reliability, and scalability, as well as easier management of complex systems.

Unfortunately managing existing resources is slightly complicated with Terraform. You will have to do a terraform-import and define a resource block in terraform configuration with complete resource metadata. This however is a complex process where you have to manually apply the import command for all resources that belong to a project.

Terraformer, a CLI tool developed by Waze SRE and made open-source soon after, became quickly popular because of its capabilities of overcoming these challenges. Learn how to generate your infrastructure as code from pre-existing AWS resources using the terraformer tool in this blog.

What is Terraformer?

Terraformer is an open-source tool that helps generate Terraform code from existing infrastructure. Terraform is a popular infrastructure-as-code tool that allows us to define and manage infrastructure resources in a declarative way. Terraformer can help us quickly and easily create Terraform code for our existing infrastructure, allowing you to easily manage and update it in the future.

Why use Terraformer?

When managing infrastructure as code, Terraformer offers a number of advantages.

• Faster migration to IAC

Terraformer allows you to quickly and easily generate Terraform code for your existing infrastructure, speeding up the process of migrating to IAC.

• Read only permissions

Terraformer requires just the RO permissions. So there is no risk of changing existing infrastructure configurations

• Greater consistency

By managing your infrastructure as code, you can ensure that all your resources are configured in a consistent way. This helps to reduce errors and makes it easier to troubleshoot issues.

• Error free

The use of ‘Terraform import’ can lead to minor issues and downtime because of manual intervention. The use of Terraformer (IaC) automates manual tasks and reduces the risk of error to a great extent

• Better collaboration

Infrastructure as code allows teams to collaborate more effectively, as everyone can work from the same codebase. Terraformer makes it easy to get started with IAC, even if you have an existing infrastructure that was not originally designed with IAC in mind.

• Easier management

Managing infrastructure as code allows you to easily make changes to your infrastructure and track those changes over time. Terraformer makes it easy to generate the code you need to manage your infrastructure in this way.

• Cost

The use of Terraformer can help teams spend less time migrating existing infrastructure to their cloud environment, and more time to focus on critical tasks. This reduces the overall cost significantly.

Ok, so how does terraformer work under the hood?

Terraformer works by using provider APIs to fetch the current state of the infrastructure resources and then transforming that state into Terraform code. The process can be broken down into the following steps:

1. Authentication: Terraformer requires authentication credentials for the cloud provider(s) that are being used. These credentials are used to authenticate with the provider API and retrieve information about the existing resources.
2. Discovery: Terraformer discovers the resources that exist within the provider's infrastructure. It uses the provider's API to fetch information about the existing resources and their configurations.
3. Transformation: Terraformer transforms the information it retrieves from the provider into Terraform code. It generates the necessary HCL code to recreate the resources in Terraform.
4. Output: Terraformer outputs the generated Terraform code to the user's preferred output format. Users can choose to output the code as a file or as standard output to be piped into another command or process.

Terraformer supports a variety of cloud providers, including AWS, Google Cloud Platform, Microsoft Azure, and more. It enables users to generate Terraform code for existing resources in these cloud providers, making it easier for them to manage their infrastructure as code.

Let’s play with it a bit by running brew install terraformer !!

Using Terraformer

• To use Terraformer, we need to configure it to work with our infrastructure provider. For example, if we want to generate Terraform code for AWS resources, we need to configure Terraformer to use the AWS provider API.

• Authenticate with AWS by exporting the AWS credentials of the account for which you are trying to generate IaC code.
• Let’s import all DynamoDB and databases (RDS) from Singapore (ap-southeast-1) region from AWS cloud. In my example, We have multiple AWS accounts, so here we use profiles feature, terraformer supports it as well.

Command to import :

terraformer import aws --resources=dynamodb,rds --regions=ap-southeast-1 --profile=""

• Terraformer generated many files, let’s see the files and directories structure

Here we can see, that we get terraform files with tfstate in every resource’s directory. Terraformer supports different resource structures, more information about that you can find here.

• Let’s test what terraform plan shows for us:

Here we can see that terraform code and tfstate are synchronized. Cool.

• By default generated code is for Terraform 0.12. I use Terraform 1.2.6, so I had issues with init like here:

I fixed it by updating the state:

This command updated tfstate version from 3 to 4.

• Let's avoid keeping the state file in our local/git where it could not be secure. Let's configure dynamoDB lock and push the state to S3. We will be able to push the state to s3 and have it locked by specifying the s3 location to store the state and dynamoDB to lock in terraform backend configurations.

• Import Single resource

Start off easy. In the AWS dashboard, manually create an EC2 instance. Once you've done so, write down the instance's instance ID. For my demonstration, that instance ID is i-0e1b122eb8d11ee22.

terraformer import aws --resources=ec2_instance --filter=”Name=id;Value=i-0e1b122eb8d11ee22"

In the above command, aws is the provider. --resources passes in the type of resource we want to generate Terraform code from. --filter is used to filter the resources by specific criteria, in our case, we are filtering the EC2 instance by its id i-0e1b122eb8d11ee22.

• Import one type of resources

The following command can be used to import just one kind of resource, such as all of our Lambda functions:

terraformer import aws --resources=lambda

This command will generate Terraform code for all our Lambda functions for the particular account and region we have configured.

• Import more than one type of resources

If we want to import more than one type of resources, simply separate the resources with comma, such as the following command, which imports for all our Lambda functions as well as DynamoDB tables:

terraformer import aws --resources=lambda,dynamodb

• Import multiple resources within the same resource type

If we need to import multiple resources, we can do so by using terraformer import as well. For example, if we need to import two Lambda functions, lambda-test1 and lambda-test2, let’s try this command:

terraformer import aws --resources=lambda --filter="Name=function_name;Value=lambda-test1:lambda-test2"

Notice the filter Name is function_name, and Value has two function names separated by colon.

• Import resources using filters by tag

We briefly touched upon filters in the “Import one resource” section above. Let’s dive deeper here to explore how best to use filters to import code for the resources we want. We can filter resources by their tags, for example, the following command filters DynamoDB resource by its tag name with value test-table.

Notice the filter name, tags.name, this is case sensitive, if you enter upper case tags.Name, the filtered number of service may be 0 if you don’t have another DynamoDB table with that tag Name defined.

terraformer import aws --resources=dynamodb --filter="Name=tags.name;Value=test-table"

• Import resource with specific tag key

To filter resources with a specific tag key, say we want to filter DynamoDB with tags having tag key name defined, we can run the command like this, notice we don’t need to specify Value:

terraformer import aws --resources=dynamodb --filter="Name=tags.name"

• Import all resources

What if you would like to generate Terraform code for your whole infrastructure stack? The command is actually really simple:

terraformer import aws --resources=*

Yes, use wildcard to pass in to resources will do the trick. However, depending on your infrastructure size, this may take some time, and the amount of directories created under generated\aws may be astounding.

• Import resource with excludes keyword

If you need to exclude certain resource from being imported, you can use --excludes keyword, sample command:

terraformer import aws --resources=* --excludes="dynamodb"

My Experience

At Halodoc, I successfully imported over 1000 AWS resources from various resource types in each AWS accounts. Throughout the process, I encountered no issues with updating tfstate versions, converting the code to 1.2, or pushing the tfstate to the S3 backend.

Summary

In our exploration, we delved into Terraformer, its importance, and how to utilize it to import existing infrastructure into Terraform code. Terraformer is a remarkable tool for reverse-terraforming that boasts an easily extendable framework for quickly incorporating new resources and providers. Additionally, the Terraformer project is actively maintained by a dedicated community that diligently works to improve it. Although there are minor issues such as the old tfstate version or 0.12 generated code, these can be addressed manually or, possibly, through automation in the future.

We are grateful to the Terraformer community for their efforts and support. Long live and prosper 🖖, Terraformer!