Author: Shea Stewart


If you have an OpenShift (Origin or Enterprise) environment, you have likely deployed the EFK (elasticsearch, fluentd, and kibana) stack to help operators and developers easily view log files. Referred to as the Aggregated Container Logs in OSE, this stack is very useful and should be deployed in most situations. But log data tends to grow rapidly, especially in development environments with a large amount of projects and containers being deployed in a CI/CD pipeline, and cleanup is necessary; enter curator.

Curator allows operators to define how long elasticsearch indices should be retained. On a defined daily schedule it will purge any qualified indices from elasticsearch. Unfortunately, curator has only been added into the OpenShift Origin version of software, and is not yet included in the OpenShift Enterprise deployments (as of 3.2). Fortunately, version 3.1.1 and 3.2 of the EFK images do include admin credentials that can be used to authenticate a manually deployed curator template.

A few notes about these steps:

  • Commands are run with a cluster-admin authorization
  • The project we are using for logging is called ‘logging’
  • The logging-es deployment configuration specifies version 3.1.1 or 3.2
  • We are setting the defaults with environment variables within the yaml file, which can be done outside of this file as well
  • We are using v1.2.0 of the origin-logging-curator image here https://hub.docker.com/r/openshift/origin-logging-curator
  1. Create the aggregated-logging-curator service account:
oc project logging
oc create -f - <<API
apiVersion: v1
kind: ServiceAccount
metadata:
name: aggregated-logging-curator
secrets:
- name: aggregated-logging-curator
API
  1. Extract admin keys from elasticsearch deployment and create logging-curator secret, ensuring to replace the unique ID with your instance ID:
oc exec logging-es-<uniqueID>  cat /etc/elasticsearch/keys/admin-ca | tee es-admin-ca
oc exec logging-es-<uniqueID> cat /etc/elasticsearch/keys/admin-cert | tee es-admin-cert
oc exec logging-es-<uniqueID> cat /etc/elasticsearch/keys/admin-key | tee es-admin-key
oc secret new logging-curator ca=es-admin-ca cert=es-admin-cert key=es-admin-key
  1. Create curator template named curator-template.yaml with the following content:
apiVersion: v1
kind: Template
labels:
component: curator
logging-infra: curator
provider: openshift
metadata:
annotations:
description: Template for logging curator deployment.
openshift.io/generated-by: OpenShiftNewApp
tags: infrastructure
labels:
logging-infra: curator
name: logging-curator-template
objects:
- apiVersion: v1
kind: DeploymentConfig
metadata:
labels:
component: curator
provider: openshift
name: logging-curator
spec:
replicas: 1
selector:
component: curator
provider: openshift
strategy:
resources: {}
rollingParams:
intervalSeconds: 1
timeoutSeconds: 600
updatePeriodSeconds: 1
type: Recreate
template:
metadata:
labels:
component: curator
provider: openshift
name: curator
spec:
containers:
- env:
- name: K8S_HOST_URL
value: https://kubernetes.default.svc.cluster.local:8443
- name: ES_HOST
value: logging-es
- name: ES_PORT
value: "9200"
- name: ES_CLIENT_CERT
value: /etc/curator/keys/cert
- name: ES_CLIENT_KEY
value: /etc/curator/keys/key
- name: ES_CA
value: /etc/curator/keys/ca
- name: CURATOR_DEFAULT_DAYS
value: "30"
- name: CURATOR_CONF_LOCATION
value: /etc/curator
- name: CURATOR_RUN_HOUR
value: "0"
- name: CURATOR_RUN_MINUTE
value: "0"
image: ${IMAGE_PREFIX}logging-curator:${IMAGE_VERSION}
imagePullPolicy: Always
name: curator
resources:
limits:
cpu: 100m
volumeMounts:
- mountPath: /etc/curator/keys
name: certs
readOnly: true
serviceAccountName: aggregated-logging-curator
volumes:
- name: certs
secret:
secretName: logging-curator
triggers:
- type: ConfigChange
- imageChangeParams:
automatic: true
containerNames:
- curator
from:
kind: ImageStreamTag
name: logging-curator:${IMAGE_VERSION}
type: ImageChange
parameters:
- description: The version tag of the image to use.
name: IMAGE_VERSION
value: v1.2.0
- name: IMAGE_PREFIX
value: docker.io/openshift/origin-
  1. Create and deploy the curator pod :
  oc project logging
oc create -f curator-template.yaml
oc new-app logging-curator-template
oc deploy logging-curator --latest

In order to customize the retention on a per-project basis, you can create a yaml file and pass it to the curator deployment configuration. An example of this file would be:

myapp-dev:
delete:
days: 1

myapp-qe:
delete:
weeks: 1

.operations:
delete:
weeks: 8

.defaults:
delete:
days: 30
runhour: 0
runminute: 0

Once created, modify the deployment config to include this file:

oc secrets new index-management settings=</path/to/your/yaml/file>
oc volumes dc/logging-curator --add --type=secret --secret-name=index-management --mount-path=/etc/curator --name=index-management --overwrite
oc deploy logging-curator --latest

If all is well, the logs of the curator pod should show something similar to the following:

logging-curator running [1] jobs
No indices matched provided args: {'regex': None, 'index': (), 'suffix': None, 'newer_than': None, 'closed_only': False, 'prefix': None, 'time_unit': 'days', 'timestring': u'%Y.%m.%d', 'exclude': (u'.searchguard*', u'.kibana*', u'.apiman_*'), 'older_than': 30, 'all_indices': False}
logging-curator run finish

Check here for more detail on the Origin logging deployment: https://github.com/openshift/origin-aggregated-logging

Check here for more detail on the OpenShift Enterprise logging deployment: https://docs.openshift.com/enterprise/3.2/install_config/aggregate_logging.html

Tagged:



//comments