Author: Tim Fairweather


Kubernetes-native CI/CD Pipelines with Argo and Anthos GKE

Lately, there has been a ton of chatter in the Kubernetes ecosystem about “Kubernetes-native or cloud-native” pipelines and CI/CD. In a nutshell, it really comes down to simplifying the suite of tools used, and running/deploying things in a way that’s native to k8s. With a number of options available to accomplish this today, I’m not going to compare or dive deep into capabilities - there are a number of tools available to accomplish this today - but rather, I will showcase a specific way to accomplish it with solutions from Argo. The demo below at the end of this post shows the workflow and has this running on Anthos GKE, Google’s newest multi-cloud offering.

The workflow looks like this from a high-level:

Argo Workflow


The Argo toolkit

The Argo tools used in this flow are:

  • Argo Events - An event based dependency manager for Kubernetes - Used in this demo to receive webhook payload information from Git, and trigger Argo Workflow runs
  • Argo Workflows - A Kubernetes-native workflow and pipeline tool - Event triggers kick-off these workflows that run as k8s pods to perform the pipeline steps required
  • Argo CD - A gitops declarative continuous deployment tool for Kubernetes clusters - Watches for Git updates, and synchronizes changes to deployed applications, with Kustomize

Setup the components

Here is what the configuration components look like for Argo in the demo video below. Each one of these should be customized for the particular use case, and the samples below are provided for reference purposes only.

Requirements

In order to set this up you will need the following:

  1. Source GitHub repository - This repo holds source code for the application … this test application displays a random-giphy via Nginx
  2. Deploy GitHub repository - This repo holds the Kubernetes manifests to deploy the image generated via the source repo above to a Kubernetes cluster
  3. ACM (Anthos Config Mangement) or other method to deploy the Argo components
  4. Deploy key(s) for the above repos, a personal access token for GitHub, and a Google Service Account key for GCR access (to push your container image to the project)
  5. Argo Events, Argo Workflows, and Argo CD deployed to your cluster - in the argo-events, argo, and argocd namespaces respectively
  6. GKE / Kubernetes cluster (with appropriate access), in a GCP project, along with GCR
  7. Istio installed in the GKE cluster with an ingress-gateway

Setup your secrets for the various steps of this pipeline as follows:

kubectl create secret generic ci-cd-deploy \
  --namespace=argocd \
  --from-literal=ssh-privatekey=<ssh-key>

kubectl create secret generic ci-cd-deploy \
  --namespace=argo \
  --from-literal=ssh-privatekey=<ssh-key>

kubectl create secret generic ci-cd-source \
  --namespace=argo \
  --from-literal=ssh-privatekey=<ssh-key>

kubectl create secret generic github-basic \
  --namespace=argo \
  --from-literal=username=<github_username> \
  --from-literal=personal_access_token=<github_token>

kubectl create secret generic github-basic \
  --namespace=argo-events \
  --from-literal=username=<github_username> \
  --from-literal=personal_access_token=<github_token> \
  --from-literal=webhook=<github_webhook_secret>

kubectl create secret generic argo-gcr \
  --namespace=argo \
  --from-file=google.json

You will need to replace the following placeholders in the below sections to setup the appropriate Argo components:

  1. ${project} - The Google Cloud Platform (GCP) project that relates to the GCR location where you wish to build and push the container image in the workflow
  2. ${lb_ip} - The IP address associated with the Istio Ingress Gateway - can find with kubectl get svc -n istio-system
  3. ${gh_organization} - The GitHub organziation where the repos exist, and the webhook will be created
  4. ${source_repo} - The GitHub repo where the source code for the app is located. This is used to create the webhook
  5. ${source_clone_url} - The ssh clone url for the source repo
  6. ${deploy_clone_url} - The ssh clone url for the deploy repo

Source Repo Setup

Fork/clone the following repo into your Source repo above: https://github.com/ArctiqTeam/random-giphy

Deploy Repo Setup

Add the following files to the deploy repository (click to expand):

kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- virtualservice.yaml
images:
- name: gcr.io/${project}/random-giphy
  newTag: latest
virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: random-giphy
spec:
  hosts:
  - "random-giphy.${lb_ip}.xip.io"
  gateways:
  - istio-system/frontend-gateway
  http:
  - route:
    - destination:
        host: random-giphy
        port:
          number: 8080
service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    run: random-giphy
  name: random-giphy
  namespace: random-giphy
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    run: random-giphy
  sessionAffinity: None
  type: ClusterIP
deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: random-giphy
  name: random-giphy
spec:
  progressDeadlineSeconds: 600
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      run: random-giphy
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: random-giphy
    spec:
      containers:
      - image: gcr.io/${project}/random-giphy
        imagePullPolicy: Always
        name: random-giphy
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        readinessProbe:
          httpGet:
            path: /
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 20
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
      imagePullSecrets:
        - name: regcred
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

Argo Components

Deploy the following Argo components in your GKE cluster (click to expand):

event-source.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: random-giphy-event-source
  namespace: argo-events
spec:
  type: "github"
  github:
    source_repo:
      namespace: argo-events
      owner: "${gh_organization}"
      repository: "${source_repo}"
      webhook:
        endpoint: "/push"
        port: "12000"
        url: "https://random-giphy-webhook.${lb_ip}.xip.io"
        method: "post"
      events:
        - "push"
      apiToken:
        name: github-basic
        key: personal_access_token
      webhookSecret:
        name: github-basic
        key: webhook
      insecure: true
      active: true
      contentType: "json"
event-gateway.yaml
apiVersion: argoproj.io/v1alpha1
kind: Gateway
metadata:
  name: random-giphy-gateway
  namespace: argo-events
  labels:
    gateways.argoproj.io/gateway-controller-instanceid: argo-events
spec:
  type: "github"
  eventSourceRef:
    name: "random-giphy-event-source"
  template:
    metadata:
      name: "random-giphy-gateway"
      labels:
        gateway-name: "random-giphy-gateway"
    spec:
      containers:
        - name: "gateway-client"
          image: "argoproj/gateway-client:v0.13.0"
          imagePullPolicy: "Always"
          command: ["/bin/gateway-client"]
        - name: "github-events"
          image: "argoproj/github-gateway:v0.13.0"
          imagePullPolicy: "Always"
          command: ["/bin/github-gateway"]
      serviceAccountName: "argo-events-sa"
  service:
    metadata:
      name: random-giphy-gateway-svc
    spec:
      selector:
        gateway-name: "random-giphy-gateway"
      ports:
        - name: http
          port: 12000
          targetPort: 12000
      type: ClusterIP
  subscribers:
    http:
      - "http://random-giphy-sensor.argo-events.svc:9300/"
workflow-virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: random-giphy-webhook
  namespace: argo-events
spec:
  gateways:
  - istio-system/frontend-gateway
  hosts:
  - random-giphy-webhook.${lb_ip}.xip.io
  http:
  - route:
    - destination:
        host: random-giphy-gateway-svc
        port:
          number: 12000
workflow-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: build-push-pr
  namespace: argo
spec:
  templates:
  - name: git-checkout-private
    inputs:
      parameters:
      - name: repo
      - name: revision
      - name: sshsecret
      - name: sshsecretkey
      artifacts:
      - name: source
        path: /src
        git:
          repo: ""
          revision: ""
          sshPrivateKeySecret:
            name: ""
            key: ""
    outputs:
      artifacts:
      - name: source
        path: /src
    container:
      image: gcr.io/arctiqteam-images/git:v1
      command: ["/bin/sh", "-c"]
      args: ["cd /src && git status && ls -l"]
    activeDeadlineSeconds: 60

  - name: build-and-push
    inputs:
      artifacts:
      - name: source
        path: /workspace
      parameters:
      - name: pathToDockerFile
      - name: imageUrl
      - name: imageTag
      - name: pathToContext
    container:
      image: gcr.io/kaniko-project/executor:latest
      args: ["--dockerfile","","--destination",":","--context","","--build-arg","PRODUCT_VERSION="]
      env:
        - name: "GOOGLE_APPLICATION_CREDENTIALS"
          value: "/secret/google.json"
      volumeMounts:
      - name: gcr-creds
        mountPath: "/secret"
    activeDeadlineSeconds: 60

  - name: git-new-branch
    inputs:
      parameters:
      - name: release
      - name: repo
    outputs:
      artifacts:
      - name: source
        path: /git
    container:
      image: gcr.io/arctiqteam-images/git:v2.0.4
      command: ["/bin/sh", "-c"]
      args: ["git clone  /git && git checkout -b -update"]
      volumeMounts:
        - mountPath: "/opt/ssh"
          name: ssh-deploy-creds
    activeDeadlineSeconds: 60

  - name: git-commit
    inputs:
      parameters:
      - name: release
      - name: author
      - name: email
      artifacts:
      - name: source
        path: /git
    outputs:
      artifacts:
      - name: source
        path: /git
    container:
      image: gcr.io/arctiqteam-images/git:v2.0.4
      command: ["/bin/sh", "-c"]
      args: ["git add * && git commit -m 'change to image -update' && git push --set-upstream origin -update"]
      env:
        - name: "GIT_AUTHOR_NAME"
          value: ""
        - name: "GIT_AUTHOR_EMAIL"
          value: ""
        - name: "GIT_COMMITTER_NAME"
          value: ""
        - name: "GIT_COMMITTER_EMAIL"
          value: ""
      volumeMounts:
        - mountPath: "/opt/ssh"
          name: ssh-deploy-creds
    activeDeadlineSeconds: 60
    
  - name: kustomize-image
    inputs:
      parameters:
      - name: release
      - name: imageUrl
      artifacts:
      - name: source
        path: /git
    outputs:
      artifacts:
      - name: source
        path: /git
    container:
      image: gcr.io/arctiqteam-images/kustomizer:v3.3.0
      args:
        - edit
        - set
        - image
        - ":"
      workingDir: /git
    activeDeadlineSeconds: 60

  - name: git-pr
    inputs:
      artifacts:
      - name: source
        path: /git
    container:
      image: gcr.io/arctiqteam-images/git:v2.0.4
      command: ["/bin/sh", "-c"]
      args: ["hub pull-request --no-edit"]
      env:
      - name: GITHUB_USER
        valueFrom:
          secretKeyRef:
            name: github-basic
            key: username
      - name: GITHUB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: github-basic
            key: personal_access_token
    activeDeadlineSeconds: 60

  - name: artifact-cleanup
    inputs:
      parameters:
      - name: path
    container:
      image: minio/mc
      command: ["/bin/sh","-c"]
      args: ["MC_HOST_argo=http://$MINIO_LOGIN:[email protected]$MINIO_HOST mc rm -r --force /"]
      env:
      - name: MINIO_HOST
        value: "argo-minio:9000"
      - name: MINIO_LOGIN
        valueFrom:
          secretKeyRef:
            name: argo-minio
            key: accesskey
      - name: MINIO_PWD
        valueFrom:
          secretKeyRef:
            name: argo-minio
            key: secretkey
    activeDeadlineSeconds: 60
event-sensor.yaml
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: random-giphy-sensor
  namespace: argo-events
  labels:
    sensors.argoproj.io/sensor-controller-instanceid: argo-events
spec:
  template:
    spec:
      containers:
        - name: "sensor"
          image: "argoproj/sensor:v0.13.0"
          imagePullPolicy: Always
      serviceAccountName: argo-events-sa
  dependencies:
    - name: github
      gatewayName: random-giphy-gateway
      eventName: source_repo
  subscription:
    http:
      port: 9300
  triggers:
    - template:
        name: random-giphy-workflow-trigger
        argoWorkflow:
          group: argoproj.io
          version: v1alpha1
          resource: workflows
          operation: submit
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: build-random-gipy-
                namespace: argo
              spec:
                serviceAccountName: argo-workflow
                entrypoint: build-random-giphy
                onExit: exit-handler
                volumes:
                - name: gcr-creds
                  secret:
                    secretName: argo-gcr
                - name: ssh-deploy-creds
                  secret:
                    secretName: ci-cd-deploy
                arguments:
                  parameters:
                  - name: commit_id
                    value: bogus
                  - name: author
                    value: bogus
                  - name: email
                    value: bogus
                templates:
                - name: build-random-giphy
                  steps:
                  - - name: git-checkout
                      templateRef:
                        name: build-push-pr
                        template: git-checkout-private
                      arguments:
                        parameters:
                        - name: repo
                          value: ${source_clone_url}
                        - name: revision
                          value: master
                        - name: sshsecret
                          value: ci-cd-source
                        - name: sshsecretkey
                          value: ssh-privatekey
                  - - name: build-and-push
                      templateRef:
                        name: build-push-pr
                        template: build-and-push
                      arguments:
                        artifacts:
                        - name: source
                          from: ""
                        parameters:
                        - name: pathToDockerFile
                          value: "Dockerfile"
                        - name: imageUrl
                          value: "gcr.io/${project}/random-giphy"
                        - name: imageTag
                          value: ""
                        - name: pathToContext
                          value: "."
                  - - name: git-new-branch
                      templateRef:
                        name: build-push-pr
                        template: git-new-branch
                      arguments:
                        parameters:
                        - name: repo
                          value: ${deploy_clone_url}
                        - name: release
                          value: ""
                  - - name: kustomize-image
                      templateRef:
                        name: build-push-pr
                        template: kustomize-image
                      arguments:
                        artifacts:
                        - name: source
                          from: ""
                        parameters:
                        - name: release
                          value: ""
                        - name: imageUrl
                          value: "gcr.io/${project}/random-giphy"
                  - - name: git-commit
                      templateRef:
                        name: build-push-pr
                        template: git-commit
                      arguments:
                        artifacts:
                        - name: source
                          from: ""
                        parameters:
                        - name: release
                          value: ""
                        - name: author
                          value: ""
                        - name: email
                          value: ""
                  - - name: git-pr
                      templateRef:
                        name: build-push-pr
                        template: git-pr
                      arguments:
                        artifacts:
                        - name: source
                          from: ""
                - name: exit-handler
                  steps:
                  - - name: artifact-cleanup
                      templateRef:
                        name: build-push-pr
                        template: artifact-cleanup
                      arguments:
                        parameters:
                        - name: path
                          value: "argo-artifacts"
          parameters:
            - src:
                dependencyName: github
                dataKey: body.head_commit.id
              dest: spec.arguments.parameters.0.value
            - src:
                dependencyName: github
                dataKey: body.pusher.name
              dest: spec.arguments.parameters.1.value
            - src:
                dependencyName: github
                dataKey: body.pusher.email
              dest: spec.arguments.parameters.2.value
giphy-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: random-giphy
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  project: default
  source:
    repoURL: ${deploy_clone_url}
    targetRevision: master
    path: .
  destination:
    server: https://kubernetes.default.svc
    namespace: random-giphy

Run the workflow

If all worked as planned with the deployment of the components above, you will now have a webhook configured in your source repository. You will have pods in the argo-events namespace in the cluster which have been built to receive webhook information from the GitHub source repo configured above. Additionally you will have workflow template steps deployed to the argo namespace.

The giphy application should be present in ArgoCD, however it will not work right away due to the image not being present in GCR until the workflow is run for the first time.

In this flow, you will update the source repository and modify the index.html file to change the random-giphy application to display a random giphy of your choice.

<!doctype html>
<html lang="en">

<head>
  <base href="/" />
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>Random Giphy</title>
  <link rel="stylesheet" type="text/css" href="index.css" />
  <script src="lib/require.js"></script>
  <script src="bootstrap.js"></script>
  <script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
</head>

<body>
  <div class="loading">
    <p>Here is your random fails giphy!!</p>
    <div id="_giphy_tv"></div>
    <script>
      jQuery(function ($) {
        fetch('//api.giphy.com/v1/gifs/random?tag=fails&api_key=<api_key>').then(function (response) {
          return response.json();
        }).then(function (my_result) {
          $('#_giphy_tv').html('<img src="' + my_result.data.image_url + '">');
        });
      });
    </script>
  </div>
  <div style="text-align:center;"><a href="https://giphy.com"><img src="Poweredby_100px-White_VertLogo.png"
        alt="Powered by Giphy"></a></div>
</body>

</html>

Modify the tag in the fetch('//api.giphy.com/v1/gifs/random?tag=fails&api_key=') line from fails to whatever you like. Upon committing to the source repo, a webhook event should be sent to the Argo Events gateway pod setup above. This will then trigger a workflow run using the Argo Events sensor definition, which will clone the repo, build and push the container image to GCR, then update the deploy repo with a Pull Request.

You can find a demo video here showcasing the entire workflow:

Want to learn more about Argo, Anthos, and/or CI/CD? We’d be happy to hear from you.

//take the first step

Tagged:



//comments


//blog search


//other topics