Author: Greg Reboul


Managing multiple Kubernetes clusters with kubectl

Since the beginning of Kubernetes the community had a strong interest on spanning clusters across regions. Unfortunately the latency does not really permit it at the node level.

There was a first attempt to implement multi-cluster management by the core Kubernetes team. The first version was introduced with Kubernetes 1.3 released on July 2016, but was not as successful as expected. The project was named Kubernetes Cluster Federation (nickname “Ubernetes”).

This release suffered a number of limitations:

  • problem 1: Limited info on federated deployments, only showing the pod count, for example.
  • problem 2: It was introducing an overarching Federation Control plane, turning the cluster into a massive meta-cluster which was hard to troubleshoot
  • problem 3: Very sensitive to what version each cluster was running, making the maintenance challenging

Maybe the reason was that the scope was too wide and lacked depth in features. The project was then delegated to a SIG. The group went back to the drawing board and came back last summer with an entirely redesigned system. The name changed in the process, and Kubernetes Federation v2 is now called KubeFed.

The major improvements since the first version:

  • The project is now using standard Kubernetes components and is allowing it to “Federate” any kind of resource, even custom ones.
  • More details are visible on federated deployments status, the first version was only showing how many pods were running
  • New CRD based implementation, which feels more integrated and consistent.

In this post we will see how setup and deploy a workload on a Kubernetes federation. We will start by creating three clusters, one host (master) and two members. We will enable the propagation of a namespace. Then a FederatedService and a FederatedDeployment will be applied to the host, translating into a corresponding Service and Deployment on each Member. The full process should take you about half an hour.

Big picture

Step 1 - Clusters creation

To keep it simple, in this example we will be using three Minikubes using VirtualBox as Hypervisor. To install Minikube and Virtualbox, please follow the instructions there

Gregs-MacBook-Pro:~ greg$ minikube start --vm-driver=virtualbox -p federation-host-cluster
[...]
🏄  Done! kubectl is now configured to use "federation-host-cluster"

Gregs-MacBook-Pro:~ greg$ minikube start --vm-driver=virtualbox -p federation-member-1
[...]
🏄  Done! kubectl is now configured to use "federation-member-1"

Gregs-MacBook-Pro:~ greg$ minikube start --vm-driver=virtualbox -p federation-member-2
[...]
🏄  Done! kubectl is now configured to use "federation-member-2"

Gregs-MacBook-Pro:~ greg$ kubectl config get-contexts
CURRENT   NAME                      CLUSTER                   AUTHINFO                  NAMESPACE
          federation-host-cluster   federation-host-cluster   federation-host-cluster
          federation-member-1       federation-member-1       federation-member-1
*         federation-member-2       federation-member-2       federation-member-2

Step 2 - Dependencies

We need to install Helm2 and a CLI tool called kubefedctl.

Helm 2 installation

Gregs-MacBook-Pro:~ greg$ brew install [email protected]
[...]
🍺  /usr/local/Cellar/[email protected]/2.16.5: 51 files, 87.5MB

We need permissions for tiller. Let’s create this tiller-rbac.yml file :

cat > tiller-rbac.yml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
EOF

Gregs-MacBook-Pro:~ greg$ kubectl config use-context federation-host-cluster
Switched to context "federation-host-cluster".

Gregs-MacBook-Pro:~ greg$ kubectl apply -f tiller-rbac.yml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created

Gregs-MacBook-Pro:~ greg$ helm init --service-account tiller --wait
$HELM_HOME has been configured at /Users/greg/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://v2.helm.sh/docs/securing_installation/

Kubefedctl installation

You can download the latest versions under “Assets” on the release page https://github.com/kubernetes-sigs/kubefed/releases

wget https://github.com/kubernetes-sigs/kubefed/releases/download/v0.2.0-alpha.1/kubefedctl-0.2.0-alpha.1-darwin-amd64.tgz
tar -zxvf kubefedctl-0.2.0-alpha.1-darwin-amd64.tgz

Step 3 - Installing kubefed chart

Pleasantly, you only have to install the federation system on the federation host. No specific configuration has to be done on any of the members.

commands:

helm repo add kubefed-charts https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts
helm repo update
helm install  kubefed-charts/kubefed --name kubefed --version=0.2.0-alpha.1 

output:

Gregs-MacBook-Pro:~ greg$ helm repo add kubefed-charts https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts
"kubefed-charts" has been added to your repositories

Gregs-MacBook-Pro:~ greg$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "kubefed-charts" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.

Gregs-MacBook-Pro:~ greg$ helm install  kubefed-charts/kubefed --name kubefed --version=0.2.0-alpha.1 --namespace kube-federation-system --wait
NAME:   kubefed
LAST DEPLOYED: Wed Apr  8 10:17:45 2020
NAMESPACE: kube-federation-system
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                                AGE
kubefed-admin                       15s
kubefed-edit                        15s
kubefed-role                        15s
kubefed-view                        15s
system:kubefed:admission-requester  15s

==> v1/ClusterRoleBinding
NAME                                      AGE
kubefed-admission-webhook:anonymous-auth  15s
kubefed-admission-webhook:auth-delegator  15s
kubefed-rolebinding                       15s

==> v1/Deployment
NAME                        READY  UP-TO-DATE  AVAILABLE  AGE
kubefed-admission-webhook   1/1    1           1          14s
kubefed-controller-manager  2/2    2           2          14s

==> v1/Pod(related)
NAME                                         READY  STATUS   RESTARTS  AGE
kubefed-controller-manager-54bbc759c8-77kkx  1/1    Running  0         15s
kubefed-controller-manager-54bbc759c8-hn995  1/1    Running  0         15s
kubefed-controller-manager-54bbc759c8-77kkx  1/1    Running  0         15s
kubefed-controller-manager-54bbc759c8-hn995  1/1    Running  0         15s

==> v1/Role
NAME                            AGE
kubefed-admission-webhook-role  15s
kubefed-config-role             15s

==> v1/RoleBinding
NAME                                           AGE
kubefed-admission-webhook-rolebinding          15s
kubefed-config-rolebinding                     15s
kubefed-admission-webhook:apiextension-viewer  15s

==> v1/Secret
NAME                                    TYPE               DATA  AGE
kubefed-admission-webhook-serving-cert  kubernetes.io/tls  2     14s

==> v1/Service
NAME                                        TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)   AGE
kubefed-admission-webhook                   ClusterIP  10.96.214.78  <none>       443/TCP   14s
kubefed-controller-manager-metrics-service  ClusterIP  10.96.52.22   <none>       9090/TCP  14s

==> v1/ServiceAccount
NAME                       SECRETS  AGE
kubefed-admission-webhook  1        14s
kubefed-controller         1        14s

==> v1beta1/FederatedTypeConfig
NAME                                    AGE
clusterroles.rbac.authorization.k8s.io  14s
configmaps                              14s
deployments.apps                        14s
ingresses.extensions                    14s
jobs.batch                              14s
namespaces                              14s
replicasets.apps                        14s
secrets                                 14s
serviceaccounts                         14s
services                                14s

==> v1beta1/KubeFedConfig
NAME     AGE
kubefed  14s

==> v1beta1/MutatingWebhookConfiguration
NAME                      CREATED AT
mutation.core.kubefed.io  2020-04-08T14:17:58Z

==> v1beta1/ValidatingWebhookConfiguration
NAME                         CREATED AT
validations.core.kubefed.io  2020-04-08T14:17:58Z

Step 3 - Add members to the federation

commands:

./kubefedctl join federation-member-1 --cluster-context federation-member-1 --host-cluster-context federation-host-cluster -v 2
./kubefedctl join federation-member-2 --cluster-context federation-member-2 --host-cluster-context federation-host-cluster -v 2

output:

Gregs-MacBook-Pro:~ greg$ ./kubefedctl join federation-member-1 --cluster-context federation-member-1 --host-cluster-context federation-host-cluster -v 2
I0408 10:26:28.116004   34190 join.go:159] Args and flags: name federation-member-1, host: federation-host-cluster, host-system-namespace: kube-federation-system, kubeconfig: , cluster-context: federation-member-1, secret-name: , dry-run: false
I0408 10:26:28.230260   34190 join.go:240] Performing preflight checks.
I0408 10:26:28.265707   34190 join.go:246] Creating kube-federation-system namespace in joining cluster
I0408 10:26:28.287494   34190 join.go:254] Created kube-federation-system namespace in joining cluster
I0408 10:26:28.287564   34190 join.go:403] Creating service account in joining cluster: federation-member-1
I0408 10:26:28.368480   34190 join.go:413] Created service account: federation-member-1-federation-host-cluster in joining cluster: federation-member-1
I0408 10:26:28.368507   34190 join.go:441] Creating cluster role and binding for service account: federation-member-1-federation-host-cluster in joining cluster: federation-member-1
I0408 10:26:28.430394   34190 join.go:450] Created cluster role and binding for service account: federation-member-1-federation-host-cluster in joining cluster: federation-member-1
I0408 10:26:28.430431   34190 join.go:809] Creating cluster credentials secret in host cluster
I0408 10:26:28.447448   34190 join.go:835] Using secret named: federation-member-1-federation-host-cluster-token-lv45w
I0408 10:26:28.469348   34190 join.go:878] Created secret in host cluster named: federation-member-1-fbsbq
I0408 10:26:28.497229   34190 join.go:282] Created federated cluster resource

Gregs-MacBook-Pro:~ greg$ ./kubefedctl join federation-member-2   --cluster-context federation-member-2   --host-cluster-context federation-host-cluster -v 2
I0408 10:28:54.147912   34400 join.go:159] Args and flags: name federation-member-2, host: federation-host-cluster, host-system-namespace: kube-federation-system, kubeconfig: , cluster-context: federation-member-2, secret-name: , dry-run: false
I0408 10:28:54.273236   34400 join.go:240] Performing preflight checks.
I0408 10:28:54.286752   34400 join.go:246] Creating kube-federation-system namespace in joining cluster
I0408 10:28:54.300663   34400 join.go:254] Created kube-federation-system namespace in joining cluster
I0408 10:28:54.300761   34400 join.go:403] Creating service account in joining cluster: federation-member-2
I0408 10:28:54.325189   34400 join.go:413] Created service account: federation-member-2-federation-host-cluster in joining cluster: federation-member-2
I0408 10:28:54.325231   34400 join.go:441] Creating cluster role and binding for service account: federation-member-2-federation-host-cluster in joining cluster: federation-member-2
I0408 10:28:54.389905   34400 join.go:450] Created cluster role and binding for service account: federation-member-2-federation-host-cluster in joining cluster: federation-member-2
I0408 10:28:54.389939   34400 join.go:809] Creating cluster credentials secret in host cluster
I0408 10:28:54.401466   34400 join.go:835] Using secret named: federation-member-2-federation-host-cluster-token-jdfjd
I0408 10:28:54.415767   34400 join.go:878] Created secret in host cluster named: federation-member-2-7plzp
I0408 10:28:54.457207   34400 join.go:282] Created federated cluster resource

We can check the status of the federated clusters with

command:

kubectl -n kube-federation-system get kubefedclusters.core.kubefed.io

output:

Gregs-MacBook-Pro:~ greg$ kubectl -n kube-federation-system get kubefedclusters.core.kubefed.io
NAME                  AGE   READY
federation-member-1   1m   True
federation-member-2   1m   True

Step 4 - Now let’s play with this federation

The federation is replicated a namespace to a selection of members. in this example we have decided to replicate the namespace “global”

Federated Namespace Spec

cat > federated-namespace.yml <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: global
---
apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  name: global
  namespace: global
spec:
  placement:
    clusters:
    - name: federation-member-1
    - name: federation-member-2
EOF

Federated Deployment Spec

cat > federated-deployment.yml <<EOF
apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: threejstext
  namespace: global
  labels:
    app: threejstext
spec:
  placement:
    clusters:
    - name: federation-member-1
    - name: federation-member-2
  template:
    metadata:
      labels:
        app: threejstext
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: threejstext
      template:
        metadata:
          labels:
            app: threejstext
        spec:
          containers:
          -  image: arctiqgreg/threejstext
             name: threejstext
             ports:
             - containerPort: 80
             env:
             - name: MESSAGE
               value: "Hello Federation"
             - name: COLOR
               value: "0x0000ff"
EOF

Federated Service spec

cat > federated-service.yml <<EOF
apiVersion: types.kubefed.io/v1beta1
kind: FederatedService
metadata:
  name: threejstext
  namespace: global
spec:
  placement:
    clusters:
    - name: federation-member-1
    - name: federation-member-2
  template:
    spec:
      ports:
      - port: 80
      selector:
        app: threejstext
      type: NodePort
EOF

commands:

kubectl apply -f federated-namespace.yml
kubectl apply -f federated-deployment.yml
kubectl apply -f federated-service.yml

output:

Gregs-MacBook-Pro:~ greg$ kubectl apply -f federated-namespace.yml
namespace/global created
federatednamespace.types.kubefed.io/global created
Gregs-MacBook-Pro:~ greg$ kubectl apply -f federated-deployment.yml
federateddeployment.types.kubefed.io/threejstext created
Gregs-MacBook-Pro:~ greg$ kubectl apply -f federated-service.yml
federatedservice.types.kubefed.io/threejstext created

We can access the services by hitting the NodePort on each node. To get IP and ports.

commands:

kubectl --context=federation-member-1 get nodes -o wide
kubectl --context=federation-member-1 get services -o wide -n global

output:

Gregs-MacBook-Pro:~ greg$ kubectl --context=federation-member-1 get nodes -o wide
NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE              KERNEL-VERSION   CONTAINER-RUNTIME
minikube   Ready    master   52m   v1.17.0   192.168.99.108   <none>        Buildroot 2019.02.7   4.19.81          docker://19.3.5
Gregs-MacBook-Pro:~ greg$ kubectl --context=federation-member-1 get services -o wide -n global
NAME          TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE   SELECTOR
threejstext   NodePort   10.96.41.237   <none>        80:30272/TCP   35m   app=threejstext

in this example, the service on the cluster will be reachable at http://192.168.99.108:30272

Step 5 - Fine-tune the deployment

If specifics parameters have to be adjusted for certain clusters we could leverage the override feature. A real application could be to adjust the number of replicas with the capacity of each cluster, or change the DNS entry of an ExternalDns annotation.

kubectl edit federateddeployments -n global threejstext

then add the “overrides” block just below “placement”

placement:
    clusters:
    - name: federation-member-1
    - name: federation-member-2
overrides:
    - clusterName: federation-member-1
      clusterOverrides:
        - path: "/spec/template/spec/containers/0/env"
          op: "replace"
          value:
             - name: MESSAGE
               value: "Random message"
             - name: COLOR
               value: "0xff0000"

Now if you refresh the browsers, you will see that the environment variable has been overridden for the first member, while the second one stays with the defaults.

screenshot

In conclusion, Kubefed is an elegant solution to the multi-cluster problem, working reasonably well, but other solutions exist

To deploy a workload to multiple clusters, GitOps tools like ArgoCD are perhaps a simpler approach. Google Anthos comes with Anthos Config Management to achieve the same goal. IBM Cloud Pak or Rancher offer another alternative.

Please keep in mind that even if Kubefed is the official way of managing multi-clusters, this project is still in alpha and not recommended for production use.

Want to learn more about multi-cluster management? We’d be happy to hear from you.

//take the first step

Tagged:



//comments


//blog search


//other topics