Author: Mohsin Raza


Deploying Vault with integrated storage on GKE

Hashicorp has recently shipped v1.4 of its Vault secrets management and ID management tool, which comes with lots of new and exciting features, but the one that we will focus in this blog is integrated storage. The previous Vault release integrated storage was in beta, but now with with v1.4 it is in general availability. Although Vault supports many storage backends, such as Consul, MySQL, PostgreSQL, Google Cloud Storage etc., the internal storage option eliminates the need to manage a separate storage backend. It also provide high availability, supports Enterprise Replication and provides backup/restore workflows. This blog will walk you through best practices for the new Vault deployment with integrated storage using Helm templates.

Raft Protcol

Vault’s integrated storage uses a consensus protocol based on Raft to provide consistency. There’s a very nice visual explaination of Raft available here. To briefly explain, Raft nodes are always in one of three states: follower, candidate, or leader. All nodes initially start out as a follower. In this state, nodes can accept log entries from a leader and can cast votes. If no entries are received for some time, nodes self-promote to the candidate state. In the candidate state, nodes request votes from their peers. If a candidate receives a quorum of votes, then it is promoted to a leader. The leader must accept new log entries and replicate to all the other followers.

When Vault is started, a single server is initialized which makes the cluster size 1 so the server self-promotes and becomes leader. Then as new nodes are added to the Vault cluster, the process involves an encrypted challenge/answer workflow. To accomplish this, all nodes in a single raft cluster must share the same seal configuration. If you are using Auto Unseal then the join process can use the configured seal configuration to automatically decrypt the challenge and can provide the answer. In case of Shamir seal the unseal keys must be provided to the nodes before it can decrypt the challenge and respond with the answer.

Deploying Vault

Now we are going to deploy Vault on Google Kubernetes Engine (GKE) using the Vault Helm chart available here. We are going to use Terraform to deploy Vault on GKE. The complete terraform code is available here. This code will create a GKE cluster, generates self-signed TLS certificates for Vault, and will deploy Vault using Terraform Helm provider on the GKE cluster.

First, clone the Terraform code in any location and then also clone the Vault helm chart.

$ git clone https://github.com/mohsinrz/vault-gke-raft
$ cd vault-gke-raft
$ git clone https://github.com/hashicorp/vault-helm 

Setup gcloud authentication for your account, create a service account that has access to use resources in your project and store the JSON for this account in the creds folder. After that modify the terraform.tfvars accordingly and deploy the stack.

$ terraform init
$ terraform apply

After completion Terraform code will output the URL for accessing the newly deployed Vault. We can check the status of the pods using kubectl but for that first get the credentials for the GKE cluster.

$ gcloud container clusters get-credentials demo-cluster
Fetching cluster endpoint and auth data.
kubeconfig entry generated for demo-cluster.
$ 
$ kubectl get pods -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          24m
vault-1                                 0/1     Running   0          24m
vault-2                                 0/1     Running   0          24m
vault-agent-injector-7d4cccc866-7qfkx   1/1     Running   0          24m

The pods will not become ready until they are bootstrapped and unsealed which will involve making one of the pods as Raft leader and joining others to this pod. First we will make vault-0 as the leader by running the following commands to initialize Vault and unseal it.

$ kubectl exec -ti vault-0 -n vault -- vault operator init

We will use the unseal keys from the output of above command to unseal Vault.

$ kubectl exec -ti vault-0 -n vault -- vault operator unseal

After the Vault is unsealed the pod will become ready and will be elected as leader (you can verify this by checking logs kubectl logs vault-0 -n vault).

$ kubectl exec -ti vault-0 -n vault -- vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.4.0
Cluster Name    vault-cluster-5d640d05
Cluster ID      4d9f4351-52ac-4819-b0a0-ce9e2949aa87
HA Enabled      true
HA Cluster      https://vault-0.vault-internal:8201
HA Mode         active
$ 
$ kubectl get pods -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          2m18s
vault-1                                 0/1     Running   0          2m18s
vault-2                                 0/1     Running   0          2m18s
vault-agent-injector-7d4cccc866-fbmz4   1/1     Running   0          2m19s

Now we will make other pods join the cluster leader Vault-0 so they can become part of the Raft cluster before we unseal them. This step will be done for all the remaining pods.

$ kubectl exec -ti vault-1 -n vault --  vault operator raft join -leader-ca-cert="$(cat ./tls/ca-certificate.cert)" --address "https://vault-1.vault-internal:8200" "https://vault-0.vault-internal:8200" 
Key       Value
---       -----
Joined    true 

# Unseal vault-1
kubectl exec -ti vault-1 -n vault -- vault operator unseal
$ kubectl get pods -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          12m
vault-1                                 1/1     Running   0          12m
vault-2                                 1/1     Running   0          12m
vault-agent-injector-7d4cccc866-5vs9w   1/1     Running   0          12m

After all the pods become part of the Raft cluster and unsealed, all pods will become ready and the Vault cluster will be ready to serve requests.

Conclusion

Integrated storage in HashiCorp Vault 1.4 will simplify the deployment and operation of production Vault clusters significantly. The Vault Helm chart greatly reduces the complexity of running Vault on Kubernetes, providing a repeatable deployment process. If you are already using Vault with Consul as a storage backend and want to migrate to integrated storage then you can also take a look at this guide provided by Hashicorp.

//take the first step, and reach out to the ArctiqTeam if you would like our support and guidance on your Vault projects.

Tagged:



//comments


//blog search


//other topics