In this post, I will be demonstrating SCM-based configuration and IaC using Anthos Config Management. Centralizing our infrastructure &configuration files in source-code repositories provides benefits such as history tracking, branching, versioning, policy management as well as reducing & eliminating any snowflakes in our application environments. While I intend to demonstrate this up with GKE On-Prem, the approach should be quite similar when applied to other cloud services such as GCP.
To start, we will be using the nomos binary provided by Google (available on Linux, Mac, and Windows) which provides us with a simple method to construct and validate our working Config Management repository.
Visit Anthos Config Management downloads to configure the Config Management Operator CustomResourceDefinition (CRD) and download the nomos
binary.
Starting from scratch as a regular Linux user in my home directory, let’s create a new directory, change into it, and use nomos
to initialize the directory structure:
mkdir acmd # Anthos Configuration Management Directory
cd acmd/
nomos init
This should produce the following directory structure:
$ tree -F
.
├── cluster/
├── clusterregistry/
├── namespaces/
├── README.md
└── system/
├── README.md
└── repo.yaml
Here are some details on what has just been created:
Directories | Details | Subdirectories Allowed? |
---|---|---|
cluster/ |
Contains configs that apply to entire clusters. All configurations are applied to every cluster enrolled in Anthos Config Management unless otherwise dictated by clusterregistry/ configs. |
No |
clusterregistry/ |
Contains configs for ClusterSelectors which are referenced by other configs in clusters/ and/or namespaces/ to confine the cluster(s) they are applied to. |
No |
namespaces/ |
Contains configs for namespaces and namespace-scoped objects. | Yes |
system/ |
Contains system configs such as the repo version and how resources are synced. system/repo.yaml is generated automatically and specifies 1.0.0 as the initial version of the repo. |
No |
See Using the Anthos Config Management repo for further details.
I will be manipulating a MachineDeployment
to demonstrate how my cluster will scale with Config Management. A MachineDeployment
is like a Deployment for governing the machines (nodes) of a cluster. The MachineDeployment
lives in the default
namespace of a cluster. To manage the MachineDeployment
via code, we will create a directory under namespaces/
for default
and specify its configuration. The contents of namespaces/default/namespace.yaml
may look something like this:
apiVersion: v1
kind: Namespace
metadata:
name: default
A good starting point for the MachineDeployment
configuration file would be to use the existing MachineDeployment
of your user cluster. That way, once the Config Management directory is initially synced with your cluster, the MachineDeployment
will remain the same.
Assign the KUBECONFIG
environment variable to the absolute pathname of your user cluster’s kubeconfig, so that you don’t have to specify it during each kubectl
command:
KUBECONFIG=<user_cluster_kubeconfig_pathname>
Export the MachineDeployment
of your user cluster in YAML format:
kubectl get machinedeployment <user_cluster_name> -n default -o yaml
Copy the output into a new file: namespaces/default/machinedeployment.yaml
. Some information needs to be stripped out of the YAML, such as creationTimestamp
, generation
, resourceVersion
, selfLink
, uid
, status
, and any other metadata specific to the current environment.
To ensure this MachineDeployment
isn’t applied to every cluster, define your specific cluster in: clusterregistry/<user_cluster_name>.yaml
, like so:
kind: Cluster
apiVersion: clusterregistry.k8s.io/v1alpha1
metadata:
name: <user_cluster_name>
labels:
<label_name>: <label_value>
Where the <label_name>
and <label_value>
can be defined however you like, but the kicker is that the ClusterSelector
that matches this label: clusterregistry/<user_cluster_selector_name>.yaml
, as below:
kind: ClusterSelector
apiVersion: configmanagement.gke.io/v1
metadata:
name: <user_cluster_selector_name>
spec:
selector:
matchLabels:
<label_name>: <label_value>
This will imply that configs referencing the above ClusterSelector
will only apply to your Cluster
. To make the MachineDeployment
take advantage of this, include the following in the MachineDeployment
:
metadata:
annotations:
configmanagement.gke.io/cluster-selector: <user_cluster_selector_name>
Now, when changes are applied to the MachineDeployment
, it will only be updated against your specific user cluster.
As a quick sanity check, execute the following command to vet that your Anthos Config Management directory is in working condition:
nomos vet # No output means it's valid
However, this is simply a directory with some configuration files living on your local workstation. To put these configuration files in action, the Config Management directory must live as a Git repository accessible by your cluster.
Proceed with the steps to initialize and connect your Config Management Directory as a Git repository, respective to your Git provider. You can also find all of the examples given here in my repository on GitHub.
To configure the ConfigManagement
operator in your user cluster, adding the Git repository details to be read from, create the config-management.yaml
file locally:
apiVersion: addons.sigs.k8s.io/v1alpha1
kind: ConfigManagement
metadata:
name: config-management
namespace: config-management-system
spec:
# clusterName is required and must be unique among all managed clusters
clusterName: <user_cluster_name>
git:
syncRepo: <repo_address>
syncBranch: master
secretType: none
Applying the update will configure Config Management to frequently poll your repository for updates:
kubectl apply -f config-management.yaml
See Anthos Config Management Quickstart for more details on how this is done.
Now that everything is hooked up, whenever you make changes to the master branch, your cluster will import that updated change and adjust itself to satisfy the MachineDeployment
, and any other configurations you’ve included. Since the syncer is only listening on the master branch, less privileged developers can continue working in their separate branches without affecting the live cluster, enforcing the good habit of submitting pull requests and performing code reviews before adopting a change.
Go ahead and try scaling the number of nodes, CPUs, and size of memory. Once changes are merged in the master branch, you can watch as the changes occur to your cluster in a matter of seconds:
watch kubectl get nodes -o wide
Arctiq recently teamed up with Google and Sysdig for a live customer event demonstrating many compelling capabilities Anthos enables. Here’s my 12-minute clip from that event that demonstrates all of this code live:
You can find more videos like this on our YouTube channel.
Interested in learning more about Google Anthos, Config Management, or GKE On-Prem? We would love to hear from you.