In this post, I will be demonstrating SCM-based configuration and IaC using Anthos Config Management. Centralizing our infrastructure &configuration files in source-code repositories provides benefits such as history tracking, branching, versioning, policy management as well as reducing & eliminating any snowflakes in our application environments. While I intend to demonstrate this up with GKE On-Prem, the approach should be quite similar when applied to other cloud services such as GCP.

To start, we will be using the nomos binary provided by Google (available on Linux, Mac, and Windows) which provides us with a simple method to construct and validate our working Config Management repository.

Visit Anthos Config Management downloads to configure the Config Management Operator CustomResourceDefinition (CRD) and download the nomos binary.

Starting from scratch as a regular Linux user in my home directory, let’s create a new directory, change into it, and use nomos to initialize the directory structure:

mkdir acmd # Anthos Configuration Management Directory
cd acmd/
nomos init

This should produce the following directory structure:

$ tree -F
.
├── cluster/
├── clusterregistry/
├── namespaces/
├── README.md
└── system/
    ├── README.md
    └── repo.yaml

Here are some details on what has just been created:

See Using the Anthos Config Management repo for further details.

I will be manipulating a MachineDeployment to demonstrate how my cluster will scale with Config Management. A MachineDeployment is like a Deployment for governing the machines (nodes) of a cluster. The MachineDeployment lives in the default namespace of a cluster. To manage the MachineDeployment via code, we will create a directory under namespaces/ for default and specify its configuration. The contents of namespaces/default/namespace.yaml may look something like this:

apiVersion: v1
kind: Namespace
metadata:
  name: default

A good starting point for the MachineDeployment configuration file would be to use the existing MachineDeployment of your user cluster. That way, once the Config Management directory is initially synced with your cluster, the MachineDeployment will remain the same.

Assign the KUBECONFIG environment variable to the absolute pathname of your user cluster’s kubeconfig, so that you don’t have to specify it during each kubectl command:

KUBECONFIG=<user_cluster_kubeconfig_pathname>

Export the MachineDeployment of your user cluster in YAML format:

kubectl get machinedeployment <user_cluster_name> -n default -o yaml

Copy the output into a new file: namespaces/default/machinedeployment.yaml. Some information needs to be stripped out of the YAML, such as creationTimestamp, generation, resourceVersion, selfLink, uid, status, and any other metadata specific to the current environment.

To ensure this MachineDeployment isn’t applied to every cluster, define your specific cluster in: clusterregistry/<user_cluster_name>.yaml, like so:

kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
  name: <user_cluster_name>
  labels:
    <label_name>: <label_value>

Where the <label_name> and <label_value> can be defined however you like, but the kicker is that the ClusterSelector that matches this label: clusterregistry/<user_cluster_selector_name>.yaml, as below:

kind: ClusterSelector
apiVersion: configmanagement.gke.io/v1
metadata:
  name: <user_cluster_selector_name>
spec:
  selector:
    matchLabels:
      <label_name>: <label_value>

This will imply that configs referencing the above ClusterSelector will only apply to your Cluster. To make the MachineDeployment take advantage of this, include the following in the MachineDeployment:

metadata:
  annotations:
    configmanagement.gke.io/cluster-selector: <user_cluster_selector_name>

Now, when changes are applied to the MachineDeployment, it will only be updated against your specific user cluster.

As a quick sanity check, execute the following command to vet that your Anthos Config Management directory is in working condition:

nomos vet # No output means it's valid

However, this is simply a directory with some configuration files living on your local workstation. To put these configuration files in action, the Config Management directory must live as a Git repository accessible by your cluster.

Proceed with the steps to initialize and connect your Config Management Directory as a Git repository, respective to your Git provider. You can also find all of the examples given here in my repository on GitHub.

To configure the ConfigManagement operator in your user cluster, adding the Git repository details to be read from, create the config-management.yaml file locally:

apiVersion: addons.sigs.k8s.io/v1alpha1
kind: ConfigManagement
metadata:
  name: config-management
  namespace: config-management-system
spec:
  # clusterName is required and must be unique among all managed clusters
  clusterName: <user_cluster_name>
  git:
    syncRepo: <repo_address>
    syncBranch: master
    secretType: none

Applying the update will configure Config Management to frequently poll your repository for updates:

kubectl apply -f config-management.yaml

See Anthos Config Management Quickstart for more details on how this is done.

Now that everything is hooked up, whenever you make changes to the master branch, your cluster will import that updated change and adjust itself to satisfy the MachineDeployment, and any other configurations you’ve included. Since the syncer is only listening on the master branch, less privileged developers can continue working in their separate branches without affecting the live cluster, enforcing the good habit of submitting pull requests and performing code reviews before adopting a change.

Go ahead and try scaling the number of nodes, CPUs, and size of memory. Once changes are merged in the master branch, you can watch as the changes occur to your cluster in a matter of seconds:

watch kubectl get nodes -o wide

Arctiq recently teamed up with Google and Sysdig for a live customer event demonstrating many compelling capabilities Anthos enables. Here’s my 12-minute clip from that event that demonstrates all of this code live:

You can find more videos like this on our YouTube channel.

Interested in learning more about Google Anthos, Config Management, or GKE On-Prem? We would love to hear from you.

//take the first step