Author: Shea Stewart


If you have an OpenShift (Origin or Enterprise) environment, you have likely deployed the EFK (elasticsearch, fluentd, and kibana) stack to help operators and developers easily view log files. Referred to as the Aggregated Container Logs in OSE, this stack is very useful and should be deployed in most situations. But log data tends to grow rapidly, especially in development environments with a large amount of projects and containers being deployed in a CI/CD pipeline, and cleanup is necessary; enter curator.

Curator allows operators to define how long elasticsearch indices should be retained. On a defined daily schedule it will purge any qualified indices from elasticsearch. Unfortunately, curator has only been added into the OpenShift Origin version of software, and is not yet included in the OpenShift Enterprise deployments (as of 3.2). Fortunately, version 3.1.1 and 3.2 of the EFK images do include admin credentials that can be used to authenticate a manually deployed curator template.

A few notes about these steps:

  • Commands are run with a cluster-admin authorization
  • The project we are using for logging is called ‘logging’
  • The logging-es deployment configuration specifies version 3.1.1 or 3.2
  • We are setting the defaults with environment variables within the yaml file, which can be done outside of this file as well
  • We are using v1.2.0 of the origin-logging-curator image here https://hub.docker.com/r/openshift/origin-logging-curator
  1. Create the aggregated-logging-curator service account:
oc project logging
oc create -f - <<API
apiVersion: v1
kind: ServiceAccount
metadata:
  name: aggregated-logging-curator
secrets:
- name: aggregated-logging-curator
API
  1. Extract admin keys from elasticsearch deployment and create logging-curator secret, ensuring to replace the unique ID with your instance ID:
oc exec logging-es-<uniqueID>  cat /etc/elasticsearch/keys/admin-ca | tee es-admin-ca
oc exec logging-es-<uniqueID>  cat /etc/elasticsearch/keys/admin-cert | tee es-admin-cert
oc exec logging-es-<uniqueID>  cat /etc/elasticsearch/keys/admin-key | tee es-admin-key
oc secret new logging-curator ca=es-admin-ca cert=es-admin-cert key=es-admin-key
  1. Create curator template named curator-template.yaml with the following content:
apiVersion: v1
kind: Template
labels:
  component: curator
  logging-infra: curator
  provider: openshift
metadata:
  annotations:
    description: Template for logging curator deployment.
    openshift.io/generated-by: OpenShiftNewApp
    tags: infrastructure
  labels:
    logging-infra: curator
  name: logging-curator-template
objects:
- apiVersion: v1
  kind: DeploymentConfig
  metadata:
    labels:
      component: curator
      provider: openshift
    name: logging-curator
  spec:
    replicas: 1
    selector:
      component: curator
      provider: openshift
    strategy:
      resources: {}
      rollingParams:
        intervalSeconds: 1
        timeoutSeconds: 600
        updatePeriodSeconds: 1
      type: Recreate
    template:
      metadata:
        labels:
          component: curator
          provider: openshift
        name: curator
      spec:
        containers:
        - env:
          - name: K8S_HOST_URL
            value: https://kubernetes.default.svc.cluster.local:8443
          - name: ES_HOST
            value: logging-es
          - name: ES_PORT
            value: "9200"
          - name: ES_CLIENT_CERT
            value: /etc/curator/keys/cert
          - name: ES_CLIENT_KEY
            value: /etc/curator/keys/key
          - name: ES_CA
            value: /etc/curator/keys/ca
          - name: CURATOR_DEFAULT_DAYS
            value: "30"
          - name: CURATOR_CONF_LOCATION
            value: /etc/curator
          - name: CURATOR_RUN_HOUR
            value: "0"
          - name: CURATOR_RUN_MINUTE
            value: "0"
          image: ${IMAGE_PREFIX}logging-curator:${IMAGE_VERSION}
          imagePullPolicy: Always
          name: curator
          resources:
            limits:
              cpu: 100m
          volumeMounts:
          - mountPath: /etc/curator/keys
            name: certs
            readOnly: true
        serviceAccountName: aggregated-logging-curator
        volumes:
        - name: certs
          secret:
            secretName: logging-curator
    triggers:
    - type: ConfigChange
    - imageChangeParams:
        automatic: true
        containerNames:
        - curator
        from:
          kind: ImageStreamTag
          name: logging-curator:${IMAGE_VERSION}
      type: ImageChange
parameters:
- description: The version tag of the image to use.
  name: IMAGE_VERSION
  value: v1.2.0
- name: IMAGE_PREFIX
  value: docker.io/openshift/origin-
  1. Create and deploy the curator pod :
  oc project logging
  oc create -f curator-template.yaml
  oc new-app logging-curator-template
  oc deploy logging-curator --latest

In order to customize the retention on a per-project basis, you can create a yaml file and pass it to the curator deployment configuration. An example of this file would be:

myapp-dev:
 delete:
   days: 1

myapp-qe:
  delete:
    weeks: 1

.operations:
  delete:
    weeks: 8

.defaults:
  delete:
    days: 30
  runhour: 0
  runminute: 0

Once created, modify the deployment config to include this file:

oc secrets new index-management settings=</path/to/your/yaml/file>
oc volumes dc/logging-curator --add --type=secret --secret-name=index-management --mount-path=/etc/curator --name=index-management --overwrite
oc deploy logging-curator --latest

If all is well, the logs of the curator pod should show something similar to the following:

logging-curator running [1] jobs
No indices matched provided args: {'regex': None, 'index': (), 'suffix': None, 'newer_than': None, 'closed_only': False, 'prefix': None, 'time_unit': 'days', 'timestring': u'%Y.%m.%d', 'exclude': (u'.searchguard*', u'.kibana*', u'.apiman_*'), 'older_than': 30, 'all_indices': False}
logging-curator run finish

Check here for more detail on the Origin logging deployment: https://github.com/openshift/origin-aggregated-logging

Check here for more detail on the OpenShift Enterprise logging deployment: https://docs.openshift.com/enterprise/3.2/install_config/aggregate_logging.html

Tagged:



//comments