Author: Shea Stewart


With the recently announced GA release of Google Anthos GKE on AWS, multi-cloud GKE management just got easier. This blog is focused on showing the simplicity of deploying GKE into AWS and how multi-cluster management is achieved with Anthos across another cloud provider.

Why multi-cloud deployments?

Over the years, the desire to create “hybrid-cloud” workloads have extended into “any-cloud” or “multi-cloud” application deployments. There are many reasons that teams may wish to run their application workloads across multiple cloud providers, a few of which might be:

  • Cost control: Moving workloads around due to fluctuating infrastructure costs.
  • Data control and residency: With datasets that reside in an existing cloud provider or region, it is often easier to move the compute closer to the data.
  • Resiliency: Distributing application services across multiple cloud providers reduces the chance of downtime or outages related to cloud provider infrastructure failures.
  • Locality: Deploying applications as close to the user base as possible will increase application performance and user satisfaction. In some cases, the preferred cloud provider may not have a region close enough, while an alternate provider does.
  • Feature capabilities: Cloud providers offer differing features or services to consume; moving the compute as close as possible to the desired feature set provides more flexibility to the development and operational teams.

With the adoption of containerization and microservice architectures, applications have become more portable and deployable across many flavours of container runtimes and orchestrators. While this helps developers move faster and makes it easier to consume multiple cloud provider services (such as deploying across GKE & EKS & AKS), it makes it incredibly difficult to maintain consistency across multiple cluster types and increases the operational burden. Anthos sets out to solve these operational and consistency challenges by providing a common control plane and experience by deploying GKE across multiple cloud providers and on-prem infrastructures.

The following sections outline how to deploy our sample multi-cloud architecture with Anthos:

Deploying GKE into GCP and AWS

In order to get started, GKE environments will need to be deployed into GCP and AWS. As far as regions go, as a Canadian company, we will deploy our assets into available Canadian regions.

Deploying GKE into GCP

GKE can easily be deployed in to GCP either via the Cloud Console, through the cli with the gcloud binary, or with an IaC tool such as Terraform.

For the purposes of this demo, the gcloud binary will set up the following GCP components:

  • A new project linked to a billing account
  • Required GKE and Anthos API services
  • A GKE cluster named bob-mckenzie
export PROJECT=sheastewart-multi-cloud-poc
export GCP_COMPUTE_ZONE=northamerica-northeast1-a
export GCP_GKE_CLUSTER_NAME=bob-mckenzie
export GCP_BILLING_ACCOUNT_ID=*********

# Set up project and billing account
gcloud projects create $PROJECT
gcloud config set project $PROJECT
gcloud services enable cloudbilling.googleapis.com
gcloud alpha billing projects link $PROJECT --billing-account=$GCP_BILLING_ACCOUNT_ID

# Enable dependent services
gcloud services enable  \
    anthos.googleapis.com \
    anthosaudit.googleapis.com \
    anthosgke.googleapis.com  \
    meshca.googleapis.com \
    container.googleapis.com \
    compute.googleapis.com \
    meshconfig.googleapis.com 

# Set the desired computing zone
gcloud config set compute/zone $GCP_COMPUTE_ZONE

# Create a simple GKE cluster
gcloud container clusters create $GCP_GKE_CLUSTER_NAME \
    --enable-ip-alias \
    --num-nodes=2 \
    --machine-type=n1-standard-1

# Get GKE cluster credentials and validate connectivity
kubectl config set-context $GCP_GKE_CLUSTER_NAME
kubectl get nodes
  • Looking at the Google Cloud Console will show the new GKE cluster

Deploying GKE into AWS

In order to deploy GKE into AWS, there are only a few steps that are highlighted in this blog. Please see the documentation for further details.

  • Deploy the management layer (sometimes referred to as the base layer)

    The management layer must exist within any VPC and region that will create user clusters. This is deployed using Terraform. Complete documentation can be found here.

:heavy_check_mark: You must complete the prerequisites here

:wrench: The Terraform plan will deploy user clusters in a single AZ by default. Edit the terraform.tfvars.json to include the desired AZ’s, additional private_subnets & public_subnets, and include all subnets in allowed_https_cidr_blocks if desired.

:recycle: It’s also worth mentioning that the default deployment creates it’s own VPC, but BYO VPC’s, security groups, and IAM resources are supported.

export ARN=*****
export SSH_CIDR_BLOCK=0.0.0.0/0        # This can be restricted to your own source subnet if you wish
export GCP_SA_KEY=creds_file.json      # Point this file to your GCP service account keyfile

# Generate the Terraform plans
anthos-gke aws management init \
  --aws-kms-key-arn=$ARN \
  --allowed-ssh-cidr-blocks=$SSH_CIDR_BLOCK \
  --gcp-service-account-key-file=$GCP_SA_KEY \
  --gcp-project-id=$PROJECT \
  multi-cloud-poc-workspace

# Apply the Terraform plan (customize terraform.tfvars.json prior to applying the plan)
cd multi-cloud-poc-workspace
terraform init
terraform apply --auto-approve

# Create a tunnel to the bastion for accessing the management endpoint
terraform output bastion_tunnel > bastion-tunnel.sh
chmod 755 bastion-tunnel.sh
./bastion-tunnel.sh -N &

# Get credentials for the management cluster
anthos-gke aws management get-credentials

# Validate connectivity to the cluster
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_management.conf \
    kubectl cluster-info
  • Looking at the AWS console will show the bastion and management node
  • Deploy 1 user cluster named doug-mckenzie

    User clusters are deployed by defining a custom resource type called AWSCluster and AWSNodePool in the management cluster, which takes care of provisioning and configuring each cluster. The Terraform plan output above creates a sample user cluster configuration that can be customized as required.

# Create a user cluster manifest
cat <<EOF > user-cluster-doug-mckenzie.yaml
apiVersion: cluster.gke.io/v1alpha1
kind: AWSCluster
metadata:
  name: doug-mckenzie
spec:
  region: ca-central-1
  networking:
    vpcID: vpc-0783753f6b7605193
    serviceAddressCIDRBlocks:
    - 10.1.0.0/16
    podAddressCIDRBlocks:
    - 10.2.0.0/16
  controlPlane:
    version: 1.15.8-gke.14
    keyName: gke-18831e7a-keypair
    instanceType: t3.medium
    iamInstanceProfile: gke-18831e7a-controlplane
    securityGroupIDs:
    - sg-0232bba4a23120d1a
    subnetIDs:
      - subnet-06f2af692e8109f58
    rootVolume:
      sizeGiB: 10
    etcd:
      mainVolume:
        sizeGiB: 10

---
apiVersion: cluster.gke.io/v1alpha1
kind: AWSNodePool
metadata:
  name: doug-mckenzie-pool-0
spec:
  clusterName: doug-mckenzie
  version: 1.15.8-gke.14
  region: ca-central-1
  subnetID: subnet-06f2af692e8109f58
  minNodeCount: 3
  maxNodeCount: 3
  instanceType: t3.medium
  keyName: gke-18831e7a-keypair
  iamInstanceProfile: gke-18831e7a-nodepool
  maxPodsPerNode: 100
  securityGroupIDs:
  - sg-0232bba4a23120d1a
  rootVolume:
    sizeGiB: 10
EOF

# Apply the manifest to the management cluster
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_management.conf \
    kubectl apply -f user-cluster-doug-mckenzie.yaml

# Watch the cluster creation process (if desired)
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_management.conf \
    kubectl describe awscluster

# Get Credentials for the new user cluster
env HTTP_PROXY=http://localhost:8118 \
  anthos-gke aws clusters get-credentials doug-mckenzie

# Test Access to the doug-mckenzie user cluster
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl get nodes
  • Looking at the AWS console will show the user cluster control plane and worker nodes
  • Create an admin token for the doug-mckenzie cluster to access the cluser from the Google Cloud Console
# Create an admin user in the user cluster
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl create serviceaccount -n kube-system admin-user

# Assign the correct rolebinding to the user user
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl create clusterrolebinding admin-user-binding \
        --clusterrole cluster-admin --serviceaccount kube-system:admin-user


# Get the token name assigned to the newly created admin service account (send in a ticket for this)
SECRET_NAME=$( \
  HTTP_PROXY=http://localhost:8118  \
  KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
  kubectl get serviceaccount -n kube-system admin-user -o jsonpath='{$.secrets[0].name}')

# Get the token
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl get secret -n kube-system ${SECRET_NAME} -o jsonpath='{$.data.token}' \
      | base64 -d | sed $'s/$/\\\n/g'

  • The new cluster will register with Google Cloud Console, but will need an admin token added
  • With the token obtained in the last step, use the token to log into the cluster in the Google Cloud Console
  • Looking at the Google Cloud Console will show the new GKE on AWS cluster
  • Finally, enable the ability to create LoadBalancer type objects for ingress networking on the aws public subnet
# Identify the VPC_ID for the cluster
export VPC_ID=$(\
  env HTTP_PROXY=http://localhost:8118 \
  KUBECONFIG="$HOME"/.kube/gke_aws_management.conf \
  kubectl get awscluster doug-mckenzie -o jsonpath='{.spec.networking.vpcID}') 


# Identify the cluster id
export CLUSTER_ID=$(\
  env HTTP_PROXY=http://localhost:8118 \
  KUBECONFIG="$HOME"/.kube/gke_aws_management.conf \
  kubectl get awscluster doug-mckenzie -o jsonpath='{.status.clusterID}')

# Identify the subnet ID
export SUBNET_ID=$(aws ec2 describe-subnets \
 --filters "Name=vpc-id,Values=$VPC_ID" "Name=tag:Name,Values=*public*" \
 --query "Subnets[*].SubnetId" \
 --output text)

# Tag the subnet with the cluster ID
aws ec2 create-tags \
  --resources $SUBNET_ID \
  --tags Key=kubernetes.io/cluster/$CLUSTER_ID,Value=shared

Cluster Consistency with Anthos Configuration Management

Now that there is a GKE cluster running in both GCP and AWS, there is a common control plane with a multi-cloud runtime! Bob and Doug Mckenzie are both running in Canada, but in completely different provider datacenters.

To manage multiple GKE clusters at scale (regardless of runtime location), Anthos Config Management is used to remove configuration drift and enforce configuration consistency. Without going into too much depth (since there a are blogs here here here here and here), deploy ACM in bob-mckenzie and *doug-mckenzie). See the ACM documentation for more details.

:soon: While doug-mckenzie was the nice name given to the Anthos GKE on AWS cluster, the actual name of the cluster is gke-b4ae6202-cluster. A feature request has been raised with Google to allow for nice-names of Anthos GKE on AWS clusters.

  • Deploy Anthos Config Management in the GKE on GCP cluster
kubectl config set-context $GCP_GKE_CLUSTER_NAME

# Deploy the ConfigManagement operator
gsutil cp gs://config-management-release/released/latest/config-management-operator.yaml config-management-operator.yaml
kubectl apply -f config-management-operator.yaml

# Create the ConfigManagement manifest
cat <<EOF > acm-bob-mckenzie.yaml
apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
  namespace: config-management-system
spec:
  clusterName: bob-mckenzie
  git:
    syncRepo: https://www.github.com/ArctiqTeam/community-code.git
    syncBranch: "master"
    secretType: none
    policyDir: "blogs/multi-cloud-app-deployment-with-gke-on-aws-and-gcp/anthosconfigmanagement"
  enableAggregateNamespaceQuotas: false
EOF

# Apply the ACM manifest
kubectl apply -f acm-bob-mckenzie.yaml

# Verify ACM pods are running
kubectl get pods -n config-management-system

# Verify ACM logs show the repo being synced
kubectl logs -f $(kubectl get pods -n config-management-system | grep git-import | awk '{print $1}') -c importer -n config-management-system

# Verify that the np-test-1 and rbac-manager namespaces are created
kubectl get ns
  • Deploy Anthos Config Management in the GKE on AWS cluster
# Deploy the ConfigManagement operator
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl apply -f config-management-operator.yaml

# Create the ConfigManagement manifest
cat <<EOF > acm-doug-mckenzie.yaml
apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
  namespace: config-management-system
spec:
  clusterName: doug-mckenzie
  git:
    syncRepo: https://www.github.com/ArctiqTeam/community-code.git
    syncBranch: "master"
    secretType: none
    policyDir: "blogs/multi-cloud-app-deployment-with-gke-on-aws-and-gcp/anthosconfigmanagement"
  enableAggregateNamespaceQuotas: false
EOF

# Apply the ACM manifest
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl apply -f acm-doug-mckenzie.yaml

# Verify ACM pods are running
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl get pods -n config-management-system

# Verify ACM logs show the repo being synced
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl logs -f $(\
      env HTTP_PROXY=http://localhost:8118 \
      KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
      kubectl get pods -n config-management-system | grep git-import | awk '{print $1}') -c importer -n config-management-system

# Verify that the np-test-1 and rbac-manager namespaces are created
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl get ns

The Google Cloud Console can also be used to view and validate the cluster and ACM status:



Multi-cloud Application Deployments

With both clusters deployed, use ACM to deploy a simple nginx web app into GKE across each cloud provider (ie. bob-mckenzie and doug-mckenzie GKE clusters).

:heavy_exclamation_mark: While ACM can play a role in application deployment, complex application deployment often requires additional orchestration and CI capabilities through tooling such as GitLab CI, Argo (events/workflows), Jenkins X, etc.

  • Create a new directory under namespaces in the ACM structure named strange-brew
  • Copy the manifests located here into the directory and commit the changes
  • Validate that the new namespaces and pods are created in the bob-mckenzie and doug-mckenzie cluster
# Validate in GKE on GCP
kubectl config set-context $GCP_GKE_CLUSTER_NAME
kubectl get pods -n strange-brew


# Validate on GKE on AWS
env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl get pods -n strange-brew
  • Curl public endpoints created by the LoadBalancer service to validate the app is running
# Validate in GKE on GCP
kubectl config set-context $GCP_GKE_CLUSTER_NAME
curl $(kubectl get svc -n strange-brew -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')


# Validate on GKE on AWS
curl $( \
    env HTTP_PROXY=http://localhost:8118 \
    KUBECONFIG="$HOME"/.kube/gke_aws_default_doug-mckenzie.conf \
    kubectl get svc -n strange-brew -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')
  • View both services in the Google Cloud Console

//Let’s get started

As you can see, managing GKE clusters across multiple provider environments has just gotten easier with Anthos. Additional blogs will expand on more advanced application deployment patterns and service management with Anthos Service Mesh across multiple cloud providers.

Interested in learning more about the multi-cloud journey? //take the first step

Tagged:



//comments


//blog search


//other topics