Ultimate Kubernetes Security Guide — Part 3 — Attack vs Defence -Contd
In Part 3 we learn security the way everyone learns best how you can attack 1 Kubernetes Cluster and then we build defences for our cluster at scale
If you prefer listening to the audio, Click on the image below to hear the audio version of this post:
Attacking & Defending Kubernetes Cluster
My personal opinion to learn Security is to learn about the Architecture Components which we did in Part 2 and then look at Attack vectors for how the thing that I’m learning can be compromised.
Applying the same principle here. We understand the key components of Kubernetes now let’s build an Attack tree.
Most of the scenarios discussed as default configurations and not sophisticated attacks so do keep that in consideration as you go through this list.
Kubernetes Cluster
Kubernetes Cluster has two components that are it’s strength and weakness. This is the Attack surface for a Kubernetes Cluster
(a) Components that can be configured
- Master/ Worker Nodes
- Containers
- Supply Chain
- Kubernetes Plugins
(b) Components that are deployed
- Application deployed in the container — A previously vulnerable container will not become secure by deploying into a secure Kubernetes Cluster.
Let’s dig deeper into Components that can be configured
(1) Nodes
Each Kubernetes Cluster consists 2 or nodes which can have Master/Worker Nodes.
- Nodes should not be public internet facing
- Nodes should be configured to only allow traffic from Control Plane. Network Access Control list can help with this.
(2) Attack Vector for Server/Master Node
Starting with probably the one component that holds Kubernetes cluster together.
Kubernetes API Server:
All User Account and Service Account interactions with Kubernetes API use kubectl and has the following defaults
- Network access to Kubernetes API is available on the entire internal network or the Internet
- Kubernetes API is not password protected
- Kubernetes API allows for Anonymous request.
- Un-Encrypted Communication i.e not over TLS between Pods (Kubelets) and API Server
Recommendation: Kubernetes API server should not be restricted to known IPs or hosted as a private cluster with all communications with Pods using TLS.
etcd service:
The brain of the Kubernetes cluster that records the state of the cluster in a key/value store. These are the defaults for this service:
- etcd API is exposed on the network for any service account to change
- Communication to etcd is not encrypted
- etcd holds the state of the entire cluster (including secrets) as key/value store with no disk encryption.
Recommendation: The etcd API should only be accessible to Control Plane services and should have encryption at rest(disk encryption) and in transit (TLS)
Admission Controller:
The Security & Governance gatekeeper plugin to use in Kubernetes. All authenticated requests to Kubernetes API Server go through Admission Controller so attacks like the following can be defined to stop.
- No trusted Container registry defined to pull containers from
- No security definition for type of deployments e.g containers that run as root or ingress rules for pods etc
Recommendation: Only trusted Container registries should be defined and Security definitions created to stop node deployments that don’t match definition. Examples of definition can include but not limited to only non-privileged container types to be created, ingress rules for pods, compliance policies etc.
Real Attack Use Case: Tesla’s Cryptojacking Attack had Kubernetes API Server exposed on the internet was not password protected which a hacker was able to use the identify pods
(4) Attack Vector for Worker Node/Client
Now this is heart of the operation. This is where the application is hosted and all branches for the application to work start from here mostly.
Kubelets
Service that communicates with the Kubernetes API to confirm the state that the pod needs to maintain.
- Kubelet endpoint has anonymous access by default.
- Kubelet communication with Kubernetes API is unauthenticated by default
Recommendation: Endpoint should have authentication and authorisation enabled and should only talk to trusted Control Plane service.
Pods
Containers where the Application is deployed.
- Containers with privileges should not be allowed to run.
- Host volume is mounted on the container
Recommendation:Use Admission Controller to stop containers being created with root privilege.
Service Account:
- Service Account with access to Cloud Metadata and Kubernetes API
(5)Supply Chain
Cloud Credentials
Managed Kubernetes environments in a Cloud Service Provider (CSP) have potential access to Metadata services from the CSP.
- Server Side Request Forgery (SSRF) if older version of Cloud Metadata is in use.
- Machine Identity that has access to Cloud services e.g AWS IAM Role that is shared with other AWS services being used in the account
Recommendation: Ensure the updated version of CSP Metadata services is been used to reduce SSRF and use individual Identity per application where possible.
CI/CD pipeline
CI/CD Pipeline is used to deploy Kubernetes Clusters using a Source Code Repository. The pipeline will contain references and connections to which Container Register to download the Container Image and in complex environment even have a dedicated Secret Management store like a CSP Secret Manager or Hashicorp Vault.
Attack Vectors can include —
- Container Images with a Host volume mounted and configured to run as root or as a privileged container.
- Container Registry with a vulnerable image that the CI/CD Pipeline or Kubernetes Cluster uses to deploy Pods
- Operating System Kernel of the container itself may have a vulnerable package e.g vulnerability which is part of the OS libraries
- CI/CD Pipeline tool e.g Jenkins or similar can themselves be vulnerable which can lead to an attacker getting access to a Kubernetes Cluster.
- Other Cloud Service Provider services that have access to the CI/CD pipeline or vice versa or shares permissions to access same resources hence increasing the blast radius of a potential compromise.
(6) 3rd Party Plugins
There are so many to name here so I will encourage you to checkout the CNCF Landscape to see how many 3rd party plugins exist for Cloud Native deployment which can work with Kubernetes. Some of the popular ones include OPA, envoy, istio, prometeus and many more.
Talking about Attack vectors for the 3rd Party plugins is out of scope for this blog so I would encourage you to research the ones that you want to research or you are seeing in your environment.
In the next and final part of this Kubernetes Security blog series we will cover a good Kubernetes Architecture for production and include reference articles, videos & more.
