Notice:
This post is older than 5 years – the content might be outdated.
When securing K8s based environments many different issues arise—from checking the actual cluster configuration to configuring K8s features like Network Policies, Pod Security Policies, Namespaces and RBAC up to the security of your images on application level. And there are even more tools that try to tackle these issues. Don’t waste your time searching and check out these open source Kubernetes Security Tools instead.
Cluster Configuration
Chances are you’ve set up your cluster successfully and have Kubernetes running. How can you make sure that you don’t expose low hanging fruit by accidentally choosing clumsy configuration options for your kube-apiserver, etcd, kubelets, … ?
The Center for Internet Security (CIS) provides benchmarks that are “continuously refined and verified by a volunteer, global community of experienced IT professionals“ and is considered “the global standard and recognized best practices for securing IT systems and data against the most pervasive attacks“. The Kubernetes CIS Benchmark provides two levels of security. Note that you shouldn’t take on every guideline in this benchmark unaltered, especially level two may negatively inhibit the utility or performance.
Because Automate All The Things isn’t just a meme around DevOps Engineers, the K8s benchmark is implemented as an InSpec Profile by the DevSec Project. In case you’re not familiar with InSpec, it’s a framework for turning compliance and security requirements into declarative code.
Running the Kubernetes Benchmark Profile from your local machine (provided you’ve installed InSpec):
- Clone the benchmark profile on your local machine:
1$ git clone https://github.com/dev-sec/cis-kubernetes-benchmark - Edit attributes in inspec.yml e.g. CIS level
- Run master-related controls against master node:
1$ inspec exec cis-kubernetes-benchmark/ --controls '/cis-kubernetes-benchmark-1\.\d\.\d/' -t ssh://<user>@<master-node> -i path/to/ssh-key - Run worker-related controls against worker node (master node with public IP works as bastion host in this example):
1$ inspec exec cis-kubernetes-benchmark/ --controls '/cis-kubernetes-benchmark-2\.\d\.\d/' -b ssh --bastion-host=<master-node> --bastion-user=<user> --host=<worker-node> --user=<user> -i path/to/ssh-key
Image Security
Dockerfile Policy
Image Security begins with your Dockerfiles. Use hadolint to build Docker images that comply with best practices for writing Dockerfiles by piping your Dockerfile to the hadolint container:
|
1 |
$ docker run --rm -i hadolint/hadolint < Dockerfile |
Vulnerability Scanning
Scanning your images for Common Vulnerabilities and Exposures (CVEs) should be a mandatory step in secure pipelines. Vulnerability Scanners collect the operating system package information of your image and compare it to corresponding package vulnerability databases like the NVD or databases specific to a particular operating system or application package system.
Currently Clair from CoreOSs, Anchore Engine and MicroScanner from Aqua Security seem to be the most promising tools. You can find a comparison of the tools and scanning results here.
MicroScanner is probably the easiest way to integrate CVE scanning in your pipeline. We add these three lines to our Dockerfiles and images will be scanned during build:
|
1 2 3 4 5 |
ADD https://get.aquasec.com/microscanner / RUN chmod +x /microscanner RUN /microscanner |
If high severity vulnerabilities are found, this will fail the image build. Simple as that.
Image Signing
Signing container images you can be sure that images are coming from a trusted source, are not tampered with and are up to date. You’re probably familiar with Docker Content Trust which integrates The Update Framework into Docker using Notary. But there’s also an interesting K8s-specific project working on top of these technologies.
Portieris is a Kubernetes Admission Controller for enforcing Content Trust, meaning we can enforce that only trusted images will be deployed in our cluster.
Resource Validation
Prior to actually deploying secure images, we should check the manifest files of K8s resources we’re going to create. The manifest mode of kubeaudit allows auditing our manifest files before applying to the cluster by simply hitting:
|
1 |
$ kubeaudit -f /path/to/manifest.yml |
This will apply kubeaudit’s default checks like auditing Security Contexts, Container Image Tags, Service Accounts, Network Policies and Resource Limits.
kubeaudit even comes with a handy auto-fix feature. Applying all the checks might end up being too restrictive for your application, but don’t worry, configuring the audit is possible of course.
Network Policy Validation
Kubernetes Network Policies control the in- and outbound traffic for a group of pods on TCP/IP level and are pretty straightforward to define.
However, as stated in this great article by my colleague Maximilian, involving network plugins that provide Network Policy support like Calico or Weave Net may add obscure complexity. Network Policy Validation Tools can remedy this problem by testing that defined policies are really in effect and allow only traffic they are supposed to. There are two noteworthy nmap-based tools out there:
- netassert creates the test cases from a YAML file containing a list of hosts/pods to test from as well as the hosts/pods and ports that should be able to be reached.
- illuminatio takes a more automated approach by scanning your Kubernetes cluster for existing Network Policies and creating test cases accordingly on its own.
BTW: illuminatio is developed by inovex engineers ?
RBAC & Principle of Least Privilege
Role-Based Access Control (RBAC) should be the preferred authentication mechanism, but can be daunting sometimes.
audit2rbac can help you following the principle of least privilege, by making sure, users and services only have the access they should. This tool takes a Kubernetes audit log containing all the API requests you expect your user / service to perform and the username / serviceaccount as input and generates RBAC Role and RoleBinding objects. Of course you have to insistently create all API requests needed, fewer than least privileges aren’t wise either. But the tool really seems to do its job.
Live Auditing & Pentesting
Remember kubeaudit? It can not only audit resource manifest files before applying, but also audit a live K8s environment after deployment.
Yet there is more, especially for getting an attackers-eye-view of your Kubernetes setup. kube-hunter is a set of Python scripts that hunt for security weaknesses and, in Active Hunting mode, exploit detected vulnerabilities in order to find further ones. It’s quite a script kiddie-like experience simply executing these scripts against the public IP or domain name of your cluster, but a meaningful one.
Cluster Visualization
Maintaining an overview of what’s running in your cluster is essential, not only for security reasons. Deployed on each cluster node, Weave Scope can visualize the topology of our cluster on application as well as infrastructure level. It’s definitely worth having a look at the docs to discover all the features Weave Scope offers.
Intrusion Detection
There is no such thing as unhackable, even after applying all the tools mentioned. How do we know if we’ve been hacked ?
Falco is a CNCF project for intrusion detection and can alert us if suspicious activities take place. It does so by using two types of event sources, syscalls executed on the nodes of our cluster as well as requests to the kube-apiserver logged by Kubernetes Auditing. These event sources will subsequently be matched with our own rules defining how alerts should be handled. Falco provides a set of default rules, but customization with your own rules is also possible. Alerts can then be sent to Stdout, Syslog, Slack, Fluentd, … let your imaginations run wild.
Getting Falco up and running as DaemonSet on Kubernetes isn’t that complicated, but remember to enable the audit logging feature of your kube-apiserver. There’s also a Helm Chart for Deployment, if this is your preferred way.
Hopefully, this snapshot of open source tools around Kubernetes security comes in handy on your road to DevSecOps. But it’s no more than a snapshot. Stay tuned! ???
Read on
If you want to join in on the fun have a look at our job offerings: We’re looking for juniors, seniors as well as working students. Read more about our take on DevOps in our portfolio.
