Author: Christopher Markieta


Terraform

This is a walkthrough of my custom Terraform provider that creates/updates file content inside a GitHub repository. The official Terraform GitHub provider may only control the existence of certain resources (such as repositories) but does not implement content manipulation of repositories. You can find my custom Terraform provider here on GitHub. I wrote my custom provider using the go-github Go library for accessing the GitHub API. You will notice that the official GitHub provider also utilizes this library.

It’s easier than it sounds

Before I begin, I recommend following the Writing Custom Providers guide to get started. Once that’s up and running, elaborate on functionality and explore a new horizon of opportunities! I am grateful to HashiCorp’s attention to detail involved in their documentation. In addition to being very thorough, you also have the entire collection of official Terraform providers openly accessible on GitHub. This is especially useful when attempting to follow best practices and discover how particular scenarios are achieved.

After exercising the tutorial, I became almost immediately aware of how I can approach my problem. However, it was a challenge to design these functions. I had to keep in mind how they will be interfacing with Terraform modules in a declarative fashion. In other words, it was tough to avoid writing the provider like any other shell script I have written in the past.

Room for improvement

My custom provider is far from complete. As of writing this, it only supports the creation of resources. I have not implemented terraform destroy, nor does it handle the updating of existing resources. However, it would not be too complicated to implement this logic. Much of the logic would be reused from the Create function. Given more time, some areas I would have liked to improve:

  • Move the organization and GitHub token parameters from resource to provider-scope. Right now, all Terraform/environment variables are referenced via the resource. This means the organization and token need to be provided for each resource defined in .tf files, as opposed to the provider block. This is also a security concern as the GitHub token will be exposed in plaintext during terraform runs and in state files.
  • Move OAuth2 client declaration to provider-scope. While the OAuth2 client is only being created once for the Create function, it would need to be generated upon update and destroy as well. Initializing the client during the provider construction would be a bit more efficient.
  • Implement branch creation and pull requests. Currently, this provider allows for directly writing to files in an existing branch (even master, if it is not protected). An ideal approach would be to generate a pull request with a dynamic branch, awaiting an authorized approver’s review before applying the merge.

Code Walkthrough

Setup, usage details, and examples given here can be found in my repo.

Here’s a high-level explanation of what’s going on in the repository:

File Description
main.go Main entry point for the Go code of the custom Terraform provider.
main.tf Contains the example usage of the demo_repo_content resource.
provider.go Root of the Terraform provider code.
resource_demo_repo_content.go Defines the manufacturing process of the demo_repo_content resource.

Let’s go into more detail one-by-one. Please note that I will be using snippets in this blog article, but the full code is available on GitHub!

main.go

This will be a main Go package that contains the same boilerplate main as the official tutorial:

package main

import (
    "github.com/hashicorp/terraform-plugin-sdk/plugin"
    "github.com/hashicorp/terraform-plugin-sdk/terraform"
)

func main() {
    plugin.Serve(&plugin.ServeOpts{
        ProviderFunc: func() terraform.ResourceProvider {
            return Provider()
        },
    })
}

These 2 imports are necessary when creating a Terraform provider. The Provider constructor being called is defined in provider.go.

main.tf

This Terraform module contains 1 example resource creating a README.md in the root of a GitHub repository.

resource "demo_repo_content" "main" {
  organization = "Markieta-Inc"
  repo = "repo1"
  branch = "master"
  file_path = "README.md"
  file_content = <<EOF
# Repository Numba 1

Chris, we really need to be more descriptive with these docs...
EOF
}

Notice that I am not providing the GitHub token in the resource declaration. Because this is sensitive data, I’d much rather export it as an environment variable than having it present in my publicly accessible repository. :grimacing:

We could also specify the provider block (for demo), but at the moment, there are no arguments to be passed at the provider-level.

provider.go

As with main.go, this is mostly boilerplate code from the custom Terraform provider tutorial. This is the definition of the Provider constructor referenced by main.go:

package main

import (
    "github.com/hashicorp/terraform-plugin-sdk/helper/schema"
)

func Provider() *schema.Provider {
    return &schema.Provider{
        ResourcesMap: map[string]*schema.Resource{
            "demo_repo_content": resourceRepoContent(),
        },
    }
}

Lastly, this is the mapping between the Terraform resource name (referenced in .tf files) and its implementation in Go (create, update, destroy, etc). This implementation is defined in resource_demo_repo_content.go:

"demo_repo_content": resourceRepoContent(),

resource_demo_repo_content.go

Here is where all of the logic comes into play, expect a bit of verbosity.

...
func resourceRepoContent() *schema.Resource {
    return &schema.Resource{
        Create: resourceRepoContentCreate,
        Read:   resourceRepoContentRead,
        Update: resourceRepoContentUpdate,
        Delete: resourceRepoContentDelete,
...

The above snippet demonstrates what components are typically required for defining a Terraform resource. Terraform will be using this function to determine what actions to perform when we reference demo_repo_content in .tf modules.

Create, Read, Update, and Delete (CRUD) are all referencing Go functions within this file to be called when appropriate. Currently, only the Create function has been implemented, the others simply return nil and do nothing. While this does not break functionality when using terraform commands, it will certainly cause inconsistencies. For example, after running terraform destroy, Terraform will successfully destroy the resource(s) from its state file(s). However, according to Terraform’s state, this will make the resource available to be created again, when in reality, it was never destroyed in the first place.

See the Schema definition that follows the CRUD references:

...
        Schema: map[string]*schema.Schema{
            "token": &schema.Schema{
                Type:        schema.TypeString,
                Required:    true,
                DefaultFunc: schema.EnvDefaultFunc("GITHUB_TOKEN", nil),
                Description: "The OAuth token used to connect to GitHub.",
            },
...

This is where we can plug in the resource’s configuration parameters. For example, the organization and token can be included as arguments within the resource declaration in the .tf file, or by exporting the GITHUB_ORGANIZATION and GITHUB_TOKEN environment variables, respectively.

These get referenced in the Create function like so:

...
func resourceRepoContentCreate(d *schema.ResourceData, m interface{}) error {
    TOKEN := d.Get("token").(string)
    org := d.Get("organization").(string)
...

Next, we create an OAuth2 client to authenticate and perform actions against the GitHub API:

...
// Authenticate and create OAuth2 client
func CreateOauth2Client(token string) (*http.Client, error) {
    if len(token) == 0 {
        return nil, errors.New("GitHub access token not defined.")
    }

    ts := oauth2.StaticTokenSource(
        &oauth2.Token{AccessToken: token},
    )

    tc := oauth2.NewClient(context.Background(), ts)

    return tc, nil
}
...

I have tried being meticulous with error-handling throughout my code as Terraform doesn’t give the most helpful message when a provider fails ungracefully.

Here’s where the magic happens. Using our OAuth2 client, acquire the repo and branch in question, and apply the file content:

...
// Retrieve branch reference and write (commit) file
func UpdateFile(client *github.Client, org string, repo string, branch string, file_path string, file_content string) error {
    // Retrieve branch reference
    baseRef, _, err := client.Git.GetRef(context.Background(), org, repo, REFS_PREFIX+branch)
    if err != nil {
        return err
    }

    encodedContent := []byte(file_content)
    commitMessage := COMMIT_MESSAGE

    // Apply new commit with file updates
    _, _, err = client.Repositories.UpdateFile(context.Background(), org, repo, file_path, &github.RepositoryContentFileOptions{
        Message: &commitMessage,
        Content: encodedContent,
        SHA:     baseRef.Object.SHA,
    })
    if err != nil {
        return err
    }

    return nil
}

With client.Git.GetRef we receive the branch reference (for its SHA) to make a new commit. Then, client.Repositories.UpdateFile performs the commit to that same branch to write the file. encodedContent is a []byte-encoded version of the file_content argument passed into our Terraform resource configuration.

To keep things simple, these actions are all performed after running terraform apply and confirming the outcome. To set the state of this particular resource, a unique identifier must be set:

...
    d.SetId(org + "/" + repo + "/" + branch + "/" + file_path)
...

So long as the resource configuration remains the same, Terraform will not attempt to run Create again. To handle the other scenarios a resource’s life cycle includes, we can implement the remaining CRUD functions.

Want to learn more Terraform? We’d be happy to hear from you.

//take the first step

Tagged:



//comments


//blog search


//other topics