The OpenShift platform make extensive use of SSL certificates. The most common deployments leverage the automatically generated SSL certificates for back-end (inter-cluster) communication, but will almost always use a public CA signed certificate for the console access, as well as the wildcard certificate applied to application routes that require TLS.
While it may not be a frequent task, updating SSL certificates is necessary from time to time. A few of these reasons include:
- Changing providers (cost, value, features, etc.)
- Changing server/service/organizational details
- Lifecycle management (expired certificates, rotation policies, etc)
- Lost, stolen, or compromised keys
In this post, we will illustrate one method of changing these certificates with minimal impact to the production cluster.
In a standard OpenShift deployment, a wildcard certificate is created and deployed into the router pods. This default certificate is used for any TLS enabled route that does not specify a specific certificate. The process to add a default certificate during deployment is well documented here.
In a Production Environment
Once the OpenShift PaaS is running in production, updating the certificate requires a different process than the one followed during the install. This process will help to ensure that services remain online during the change, and will use the native OpenShift blue-green deployment strategy to ensure that the router pods operate properly before cutting over users/services.
In order to change the default SSL certificate, we will simply modify the certificate contents that are loaded into the router and then re-deploy the router. The certificate contents are stored in OpenShift as a secret, which means that the secret can be updated with the new contents, and will take effect upon router redeploy. This script will:
- Back up the existing secret (sometimes you just need a safety net)
- Create a new .pem file out of the new certificate files
- Delete the existing secret
- Create the new secret
- Start a new router deployment
- Clean up the .pem file
- Display pods in the default project (new containers should be spawning)
This script assumes you have cluster-admin privileges, and are running this from one of the master hosts (likely where you maintain your SSL certificates).
The script is maintained in the ArctiqTeam public GitHub repository, and can be found here: https://github.com/ArctiqTeam/openshift/tree/master/ssl_management
#!/bin/bash ## This script is intended to replace the default SSL certificate assigned to the OpenShift router pods ## This process can be peformed without service disruption, rather than re-deploying the routers (which has a service impact) ## The SSL certificate contents are mounted as a secret inside of the router pods, which this script will replace and re-deploy ## You must have cluster-admin privileges, and this assumes the router deployment config has the default name ## Pass the new certificate file names in this order: NEW_CERT NEW_PRIVATE_KEY NEW_CA_CERT ## ie. ./change_router_default_cert.sh apps_cert.crt apps_key.key ca_cert.crt NEW_CERT="$1" NEW_PRIVATE_KEY="$2" NEW_CA_CERT="$3" NEW_PEM=new_cert.pem date=`date +%y%m%d%H%M` #Select the project oc project default #Backup existing secret data oc export secret router-certs -o yaml > router-certs.backup.$date.yaml # Create a pem file with the new files cat $NEW_CERT $NEW_CA_CERT > $NEW_PEM # Remove existing secret oc delete secret router-certs # Create new secret oc secrets new router-certs tls.crt=$NEW_PEM tls.key=$NEW_PRIVATE_KEY --type='kubernetes.io/tls' --confirm # Redeploy router pods oc deploy dc/router --latest # Clean up temp file rm $NEW_PEM # Display running pods for verificaiton oc get pods
This has been tested in environments with the default HAProxy router deployment that has previously used the --default-cert flag, with and without the ipfailover deployment options (outlined here). No changes are required to the ipfailover deployment configuration for this to take effect. If the current router config does not use the --default-cert flag, it will need to be redeployed with the oadm command as outlined in the installation instructions.