Author: Marc LeBlanc


Intro

When I am talking with customers about using Terraform to achieve Infrastructure as Code goals, one topic that always comes up is how to handle Service Account keys. What are they, what do they have access to, how do we manage them, and so forth. It is always a sensitive subject due to the risks associated if not appropriately handled. This post is a look at how we can use Vault’s GCP Secrets Engine to dynamically generate service account keys and provide a central location to manage key rotation and access rights.

Traditional

GCP SA Request flow

I may have taken some artistic liberties here, but generally, the flow is correct. A team requests a Service Account, the request goes through an approval process, and the request is granted. Once a developer has this key, off they go and can write some cool automation with Terraform. The problems here are many.

  1. Way too many touchpoints in the process, inefficient.
  2. An operator has to manually (or programmatically) make the changes in GCP - applying IAM roles to various levels of your GCP implementation
  3. A key file is generated, someone has a physical copy of it. This key needs to be managed and rotated

The Vault Way

So no comic this time, but the flow with Vault is thus.

Vault has a Group with permission to a path with a defined roleset - say gcp/roleset/app-team-xyz. This roleset is associated with a GCP project and a set of bindings granting IAM roles to specific GCP Resources. This roleset creates a Service Account in the named GCP project and applies the requested IAM to the specified GCP projects, but does not create a Service Account key. A user with the correct Vault policy can call upon this roleset and be issued a Service Account key. Additionally, a TTL can be set with the roleset, meaning any time the roleset is called upon, the generated key will be automatically cleared out after the TTL expires. Better yet, we can code this up in a way that Terraform can directly request the key from Vault, meaning no user is ever in direct possession of a key. There is still an approval process here, the roleset defined in Vault should still be audited and approved accordingly, but this removes some of the pain points that exist in a more “traditional” approach.

Setting it Up - Vault

1) Enable GCP Secrets Engine

 $ vault secrets enable gcp
Success! Enabled the gcp secrets engine at: gcp/

2) Create a Service Account/Key in GCP and save it to the Vault GCP Config

This is a one-time thing. Vault requires a Service Account to use - this can be restricted as you need, but it will require (at a minimum):

  • Folder IAM Admin to be able to apply IAM at a folder level
  • Project IAM Admin to apply IAM at a Project level - at a minimum. Adjust the IAM permissions based on the needs of the rolesets you need to create.
$ vault write gcp/config credentials=@vault-sa.json
Success! Data written to: gcp/config

3) Create your Roleset bindings file

This example grants compute.admin, container.admin and serviceAccountUser roles on GCP project app-team-xyz. Save this as bindings.hcl

    resource = "//cloudresourcemanager.googleapis.com/projects/app-team-xyz" {
        roles = ["roles/compute.admin", "roles/container.admin", "roles/iam.serviceAccountUser"]
  }

4) Write the roleset to Vault

$ vault write gcp/roleset/project-factory-roleset  /                  
        project=app-team-xyz /  # Service Account Will be created in this Project
        secret-type="service_account_key" / 
        bindings=@bindings.hcl 

So that’s the Vault side of things, but this post is about consuming this via Terraform Enterprise, so now let’s take a look at that.

Setting it Up - Terraform Enterprise

We’ll make some assumptions that you are somewhat familiar with Terraform Enterprise, but generally in Terraform Enterprise you create a Workspace, a Workspace is associated with a repo (in this case, GitHub), and in a Workspace you can set Terraform and Environment Variables. For Terraform to fetch a dynamically generated Service Account key, it has to have access to Vault. We achieve this using the Terraform Vault Provider and setting Environment variables for the Vault address and a Token. So the Vault provider says, “Hey, I need a key from gcp/keys/roleset”. The response it receives is the Service Account key. This key is valid for whatever the TTL is set to, for example, 5 minutes. This return value is then passed onto the Terraform Google Provider and then onto the rest of your code.

Here’s what that looks like.

Workspace Variables

TFE Workspace Vars

Terraform Provider Code

# VAULT_ADDR and VAULT_TOKEN read in from Environment variables on the workspace
provider "vault" {}

# What roleset are we looking for - read in from Terraform variable on the workspace
data "vault_generic_secret" "gcp_auth" {
  path = "gcp/key/${var.roleset}"
}

provider "google" {
    # Pass the key from Vault to the Google provider
    credentials = base64decode(data.vault_generic_secret.gcp_auth.data.private_key_data) 
    project = var.project_name
}

Summary

Ok, so what did we solve? What happened here?

  • First there is a self-service component. No longer do developers need to ask for a new key file, and no longer does an admin need to manage or rotate that key. Vault will expire and manage the keys per the TTL set.

  • Second is that no user has direct access to the key; the key is accessed and consumed programmatically. This also makes developing and testing code much easier, with no requirement to stage a key file and no possibility to mishandle that key file. Even in the event that a user does get a hold of the key file, that key is only valid for the TTL value.

  • Lastly is increased velocity; because this is being done programmatically and the approval should only need to happen once, this can all happen a lot faster which will reduce the temptation for teams to forgo automation in favor of deploying manually.

We solved these problems using Vault’s GCP Secrets Engine and Terraform Enterprise Workspaces. Sidenote, you can do this with Terraform Cloud as well.

Finally - this concept has been recorded as a demo. Have a watch and see how to make use of the GCP Secrets engine to automate provisioning GCP projects and GKE clusters via Terraform. You can see it here

Happy Terraforming!

If you are interested in learning more about TFE variables with GitHub source control, we would love to hear from you //take the first step

Tagged:



//comments


//blog search


//other topics