Author: Shea Stewart


Looking back to the KubeCon + CloudNativeCon event in Seattle 2016, the conference this year has grown over 7x in both attendees and innovative solutions that support the Kubernetes and/or CNCF related projects. The sheer growth of development and interest in this area continues to prove that Kubernetes is the defacto enterprise workload scheduler and security must become a higher priority.

This year KubeCon + CloudNativeCon added many (many many) specialized co-located events prior to the start of the conference. Not surprisingly, there was a massive line up for the over 200 attendees that signed up for KubeSec Enterprise Summit put on by Aqua Security, AWS, and Red Hat. Each of these companies puts a lot of thought leadership and engineering efforts towards securing enterprise workloads, which is easily shown by the agenda below:

KubeSec Enterprise Summit

This day was very interesting in one way; it wasn’t exciting. There were zero shock and awe moments, minimal aha moments, and mostly yeah, that makes sense moments. This is a GREAT thing, by the way, and was actually refreshing. It’s clear that container technology is being widely adopted in the Enterprise, it’s being taken seriously, and all of the good things we know about security can continue to be applied. This, in my opinion, allows us to switch from selling ideas and focus on deploying solutions and reference architectures.

A few interesting thoughts, patterns, and recommendations did emerge through the presentations and conversations:

  • Companies are actively using the cloud, and they are getting pwned.
    • Over 50% of companies have 5 or more platforms (aka; deploying hybrid solutions across multiple clouds)
    • 40% of companies surveyed admitted to getting breached;
      • 20% of those were attacked 3-5 times in a year
  • Engage members from the Security team early and often
    • Seriously, do it.
  • One size fits many:
    • Try to deploy / automate a consistent set of security tools that meet most use-cases
    • Handle the one-off use cases just as they are; one-offs.
    • Leverage a centralized / de-centralized approach where core policies/tools/principals are enforced in a centralized way, but provide latitude to the teams to add on additional security tooling to their kits
  • The platform can (and must) be secured
    • Patch early, patch often (or frequently redeploy new clusters)
    • Leverage your operations tooling to identify anomalies in container behavior
    • Leverage host-level vulnerability scanners
    • Leverage open source tools to validate your configuration and identify potential issues:
    • Leverage the CIS guidelines for secure configuration (cgroups, selinux, etc. see kube-bench for more detail)
    • Use K8s 1.8 or above (*** doesn’t apply to OpenShift)
      • Hello RBAC, goodbye ABAC
      • Avoid wildcard usage in rolebindings
      • Follow the rule of least privilege
      • Leverage open source tools to validate your configuration and identify potential issues:
    • Deploy a platform-wide secure secrets management solution:
      • Kuberenetes 1.13 will default to encryption at rest
      • Most companies deploy a more robust solution such as HashiCorp Vault
    • Use automation (aka. IaC - Infrastructure As Code) to secure all clusters in a consistent way
    • Use automation IaC to prohibit admins from directly logging into nodes and causing configuration drift
  • How you develop your container image is as important as what it does
    • The laundry list is real (and published widely):
      • Only use trusted/approved base images
      • Sign your images
      • Scan all images through the lifecycle (development, ci & cd pipelines, production running instances)
      • Identify and block any “mystery meat” in your container images
      • Don’t run as root.
      • Don’t run as root.
      • Don’t run as root.
      • Deploy runtime protection agents lock-down container capability/operations
  • Many vendors and solutions exist to help secure your container workloads
    • Some vendors are long-lived and bring a wealth of Enterprise experience
    • Some vendors are new, excited, and get it (on the technology side) but may need coaching on the Enterprise side
    • Some claim to do it all (very few do)
    • Some identify as doing one thing really well (this is most often the case)
    • A secure platform will be composed many security focused tools and solutions
  • Zero-day exploits are impossible to predict
    • Contain the blast-radius:
      • Deploy runtime protection agents that stop unknown/unexpected processes from launching
      • Split your monoliths into microservices (simplifying detection and runtime protection)
      • Deploy immutable containers and read-only filesystems
      • Deploy forensics tools to investigate breaches
  • Your container/platform service still requires:
    • A clear Service Definition with a well documented Shared Responsibility Model
    • A robust incident response plan
    • Thorough training for all teams that use or support the platform

This was a fantastic day and it is refreshing to see so many large Enterprises leaning in to their Kubernetes deployments and taking security seriously. The tools may change, but the rules are the same and we all need to ensure that Security is everyone’s responsibility.

Have fun containerizing (securely)!

Interested in learning more or discussing secure container development? We would love to hear from you.

//take the first step

Tagged:



//comments


//blog search


//other topics