This is a follow up blog to my previous blog which talked about using the Vault PKI engine to create dynamic certs for webapps running on GKE. A lot of Arctiq’s customers use OpenShift as their choice of Kubernetes flavor so a similar blog for OpenShift was in high demand.

For those of our readers that haven’t read the previous blog, I’ll go over the workflow again.

What is Vault?

Vault is a tool for securely accessing secrets. Vault allows users to store, manage and control access to tokens, username password and database credentials. There are many secrets management tools out there but Vault has gained a lot of popularity thanks to it’s flexible API and providing encryption at rest and in flight.

There have been cases of applications leaking credentials through logs, diagnostic output, external logging to a logging stack. Vault can provide dynamic secrets which help provide short-lived ephemeral credentials to applications instead of long live credentials. With dynamic secrets, even if an application writes it to an external system, it is valid for a specific period of time.

I’m not going to get into how to install Vault but going to cover more around the integration with apps running on OpenShift.

Vault PKI Secrets Engine

Vault enables you to deal with a whole Public Key Infrastructure (PKI) to guarantee secure correspondence among various applications. This enables organizations to effectively setup their own certificate authority (CA), revoke or issue new certs using basic API calls.. and dumping the agonizing procedure of always creating self signed certificates.

In this guide, I am going to clarify how Vault PKI engine functions and how you can utilize it to make your very own Root CA.

Cert-Manager

Cert-manager is a fantastic open-source project which can be used to request dynamic TLS certificates from Vault, Letsencrypt etc. Cert-manager runs as a pod on OpenShift and grabs TLS names from ingress objects.

Using Ingress on OpenShift

OpenShift ships with an HA Proxy router which acts a front end load balancer for wildcard application URLs which supports the Route API end point. Cert-manager does not support the Route object yet. So as a work around we will use NGINX reverse proxy to create ingress objects for OpenShift. There is a really good doc written for installing NGINX reverse proxy on kubernetes here. I won’t be getting into how to install the reverse proxy on OpenShift in this blog but I will link our public repo which will have the manifests at the end of this blog.

A lot of moving parts again but let’s get started:

Deploy the application

The application I’m going to be using is going to be the weather app again.

• Create a new project in OpenShift

oc new-project weather

• Create the deployment config

$ cat weather-dc.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: weather
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: weather
    spec:
      containers:
      - image: gcr.io/container-secrets/weather:latest
        imagePullPolicy: Always
        name: weather
        ports:
        - containerPort: 8080

$ oc create -f weather-dc.yaml

• Create the service for the weather app

$ cat weather-svc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: weather
spec:
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
  selector:
    app: weather

$ oc create -f weather-svc.yaml

• Assign DNS

At Arctiq’s production environment we use Terraform to dynamically update DNS for applications for lab and customer testing. I won’t get into too much about how to accomplish that with Terraform. It’s fairly straight forward by following Google’s and Hashicorp’s guides.

To make things simple I assigned the public IP of the node where the NGINX reverse proxy pod is running.

• Create the ingress for the weather app

$ cat weather-ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: weather
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - <YOUR-DNS-NAME>
    secretName: weather-tls
  rules:
  - host: <YOUR-DNS-NAME>
    http:
      paths:
      - path: /
        backend:
          serviceName: weather
          servicePort: 8080

$ oc create -f weather-ingress.yaml

Install cert-manager

At this stage, cert-manager can be installed on the OpenShift cluster.

• Clone the JetStack repo.

$ git clone https://github.com/jetstack/cert-manager.git

• Create the objects for the cert-manager deployment:

$ cd deploy/manifests/
$ oc create -f 00-crds.yaml
$ oc create -f 01-namespace.yaml
$ oc create -f cert-manager.yaml

• Verify cert-manager is running:

$ oc get pods -n cert-manager
NAME                                      READY     STATUS             RESTARTS   AGE
cert-manager-589678351-vd599              1/1       Running            0          1h

Configure Vault PKI back end

• Create policy

I won’t get into how to install Vault. Cert-manager requires issue and sign on the PKI back end:

path "pki/issue/cert-manager" {
  capabilities = ["read", "list", "create", "update"]
}

path "pki/sign/cert-manager" {
  capabilities = ["read", "list", "create", "update"]
}

• Create approle for cert-manager

Save your role-id and secret-id:

$ vault write auth/approle/role/cert-manager-role policies=cert-manager-pki
$ vault read auth/approle/role/cert-manager-role/role-id
$ vault write -f auth/approle/role/cert-manager-role/secret-id

• Enable Vault PKI backend to act as a private CA

$ vault secrets enable pki vault secrets tune -default-lease-ttl=2160h -max-lease-ttl=87600h pki
$ vault write pki/root/generate/internal common_name=arctiq.ca ttl=87600h
$ vault write pki/roles/cert-manager \ allowed_domains=weirdscience.labs.arctiq.ca \ allow_subdomains=true max_ttl=8760h

• Create kubernetes secret containing secret-id (saved in the previous steps)

cat cert-manager-vault-approle.yaml 
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cert-manager-vault-approle
  namespace: weather
data:
  secretId: "<base-64-encoded>" ### your secret-id

• Create the vault issuer for cert-manager

$ cat issuer.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: vault-issuer
spec:
  vault:
    path: pki/sign/cert-manager
    server: https://vault.weirdscience.labs.arctiq.ca:8200
    auth:
      appRole:
        path: approle
        roleId: "d37b79df-3d0f-f089-4121-be4c6887b367" ### your role-id
        secretRef:
          name: cert-manager-vault-approle
          key: secretId

• Create the certificate for cert-manager

$ cat certificate.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: weather
spec:
  secretName: weather-tls ### should be the same as in ingress manifest
  issuerRef:
    name: vault-issuer
  commonName: <YOUR-DNS-NAME>
  dnsNames:
  - www.<YOUR-DNS-NAME>

At this point, test if the weather-tls secret is dynamically created:

$ oc get secrets -n weather
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-5k6fv    kubernetes.io/dockercfg               1         1h
builder-token-2cpzw        kubernetes.io/service-account-token   4         1h
builder-token-7zcjw        kubernetes.io/service-account-token   4         1h
cert-manager-vault-token   Opaque                                1         1h
default-dockercfg-r648w    kubernetes.io/dockercfg               1         1h
default-token-mclnr        kubernetes.io/service-account-token   4         1h
default-token-tkpx5        kubernetes.io/service-account-token   4         1h
deployer-dockercfg-bkb2j   kubernetes.io/dockercfg               1         1h
deployer-token-b52qz       kubernetes.io/service-account-token   4         1h
deployer-token-srqgn       kubernetes.io/service-account-token   4         1h
weather-tls                kubernetes.io/tls                   

Test the application:

And done! I have committed the code for the guide here

This demo was also recorded live at one of Arctiq’s event. A recap can be seen here.

Interested in learning more about this demo? Feel free to contact us by click on the link below

//take the first step