Automate HashiCorp Terraform Continuous Testing

Continuous testing is the process of testing early, often, and in an automated way — usually triggered by any change to code. Continuous testing reduces the time to obtain feedback on code quality, thereby reducing business risk. To achieve continuous testing, we rely on automation, which is not to be confused with test automation that is focused on simple pass/fail hurdles. Instead, with continuous testing, we shift our thinking to focus on acceptable levels of business risk and whether or not code can be released given that lens. 

We have been working with an R&D department at a global manufacturer to encapsulate its infrastructure as code (IaC). In doing so, we built several re-usable HashiCorp Terraform modules to deploy and provision infrastructure to its environments. While sharing these modules across multiple environments, it was difficult to make sure that a new change would not cause any issues for those entities. Should there be an issue, the first question is naturally, ‘what changed?’ To avoid this question and the resulting resources needed to identify, troubleshoot and resolve the issue, the infrastructure team spent a great deal of time debugging code before actually deploying it. 

Interested in streamlining this process as well, the R&D team asked our DevOps consultants if we could help. The goals included: 

• automating the process and ensuring that Terraform module code changes are compatible across all its environments; 
• implementing quality assurance, and; 
• saving time while learning how to manage and extend the solution moving forward.

Working hand-in-hand with the team, we helped automate the process of testing Terraform modules. We wrote test cases in the Go programming language, aided by helper scripts provided by Terratest. We implemented a new multi-branch pipeline that is triggered whenever a new pull request is raised. As an example, the Terraform module to deploy Netbox, an IP address management (IPAM) and data center infrastructure management (DCIM) solution, was used to test and assert if the configuration is set and performs as expected. 

Before we look at the details of the testing solution, let’s take a quick look at Terratest. 

Terratest

Terratest is implemented as a Go library. It provides a collection of helper functions and patterns for common infrastructure testing tasks, like running shell commands, making HTTP requests, executing commands on servers over SSH and more. Terratest helps maintain IaC by making it easier to write automated tests for it, ultimately giving teams greater confidence in the changes they make to code.

For more information, refer to the Terratest Documentation.

The Test Automation Process

Using our Netbox example, the Terraform automation process involved the following steps:

1. Terratest based test case files ending with _test.go are created in Go to automate infrastructure testing while deploying.
2. A Jenkins multi-branch pipeline is created to run automated tests on raising pull requests to master branch.
3. Netbox is deployed to validate that the infrastructure is created and works correctly.

Testing Workflow

Continuing with the Netbox example, at a high level, the testing workflow is as follows:

1. Deploy the Netbox application into a test environment based on the input specified in the Jenkinsfile.
2. Trigger the helper methods in Terratest to validate that the infrastructure was created and works correctly.
3. Clean up and destroy the Netbox application in the test environment.

Continuous Testing Benefits

Terraform continuous testing has the following benefits:

• Faster feedback helps engineers gain greater confidence in handling the risks related to a build. 
• Saves engineering time by stopping them from advancing to the next stages, like unnecessary deployments/testing rounds/debugging/troubleshooting.
• Serves as a model to engineers in deploying similar tests in other Terraform modules with minimal changes.

By creating Terratest based test cases and a Jenkins multi-branch pipeline, we automated the process of testing Terraform modules, ensuring that code changes are compatible across multiple environments. Moreover, the solution has increased quality while giving engineers time to focus on more strategic outcomes.