To give you a customized action plan with specific steps for you platform, first tell us a couple of key things.

Size isn’t everything, but it does generally indicate overall architecture complexity.

With rare exceptions, clusters should be deployed in dedicated subnets/networks from other non-cluster resources with strictly defined network access controls for ingress and egress traffic flows – especially for administrative interfaces.

To ensure all activity is attributable to a specific individual, all users who interact with the cloud/cluster environment should be given their own named accounts that are not shared, and all accounts should have Two-Factor Authentication or Two-step Verification enabled.

Implementing the principle of least privilege for access to the cluster environment can be challenging. To improve the granularity and minimize the overhead of managing those permissions long term, users should be combined into groups according to job function or role, and IAM permissions should be granted for those groups instead of on individual accounts.

Many tactics used by cloud-aware attackers can generate copious logs. However, due to cost, storage requirements, and management effort, many log sources can be left disabled or simply ignored. All logs that can be used to provide evidence of a compromise or support an investigation should be enabled and centrally collected.

Most cloud providers default to having publicly accessible Kubernetes administrative interfaces (API & SSH). Even with encryption and authentication/authorization enabled, a Kubernetes vulnerability or set of leaked/stolen credentials would lead to cluster compromise. Cluster administrative interfaces should be private or protected with restrictive network access controls.

Are all production or near-production clusters configured with both the control plane (CP) components and worker nodes across two or more availability zones?

In some Kubernetes installations, you must explicitly enable audit logging or configure the mechanism to send audit logs for all actions in the cluster. Without capturing these logs centrally, it’s nearly impossible to know what actions an attacker or legitimate user performed.

When building a cluster, there are opportunities to ensure that certain security features or add-ons are enabled and certain weak configurations are disabled. Hardened clusters enforce RBAC, enable network plugins that implement pod-to-pod firewall rules, enable admission controllers, and in the cases of cloud providers, enable integration with native IAM Roles directly to pods.

Are worker nodes running in a cloud provider exposing unnecessary permissions to pods via the instance metadata API? In some cloud providers, the worker node instances have IAM instance roles assigned to them that allow any process on the host to access and use those credentials, and this includes all pods.

Managed Kubernetes services have native logging and monitoring integrations that automatically ship worker node and container/application logs centrally, but if you opt-out or don’t use those services, you have to deploy and tune a custom solution. All worker node operating system and container logs and metrics should be collected centrally into the same pipelines as all other cloud logs and metrics to support accurate troubleshooting and incident response investigations.

While not a security feature itself, the Kubernetes namespace provides a mechanism for grouping workloads and permissions. It also serves as an anchor point for applying security policies for admission control and pod-to-pod network firewall rules. All non-system workloads should be run in dedicated namespaces and grouped according to workload ownership (separate teams) and by workload sensitivity (separate security tiers) to provide the granularity needed to implement least-privilege permissions and strong security policies.

A knowledgeable Kubernetes attacker will likely try to leverage valid credentials of either administrators or in-cluster service accounts. If those credentials are overgranted permissions, full cluster compromise is often possible. Though sometimes arduous, scoping down IAM and RBAC access to the cluster and its resources is one of the best ways to thwart privilege escalation and to generate anomalous audit logs that could alert the organization of malicious activity.

A Kubernetes user with permission to create a pod can configure that pod to run in a secure manner or in such a way that it allows direct access to the host (“container escape”). When a workload can escape to the underlying host, it provides several methods for compromising other cluster workloads and even the entire cluster. Admission controllers such as PodSecurityPolicy or dynamic admission controllers such as Gatekeeper or K-rail are necessary to give cluster administrators the ability to enforce policy on specific settings to workloads to make sure their configurations do not undermine the security posture of the entire cluster.

If an attacker compromises a workload inside your cluster, multiple lateral movement and privlege escalation techniques can be prevented by reducing the network access between pods in the cluster, to instance metadata, and to other internal networks. All namespaces should have network policies defined that both protect and restrict the ingress and egress traffic on a per-pod basis to just the necessary paths.

Kubernetes provides no native solution for monitoring what occurs inside each container as it’s running in the cluster. If an attacker exploits a vulnerable application, obtains a shell, uploads binaries, or uses that container to begin attacking the rest of the cluster, that activity needs to be detected early. Cluster administrators should implement a container behavior security monitoring and/or prevention solution to have confidence the activity inside each container is desired and to be alerted when that is not the case.

Container images have no built-in update strategy once they are running to conform with the best practices of immutable infrastructure. Patching for new code, fixes, or vulnerabilities requires rebuilding the image and redeploying it. Organizations typically approach this at scale by building and frequently updating a small number of internally managed base images that are routinely scanned for vulnerabilities and then requiring their application container images to be built using the latest base image. They also couple this with automated image vulnerability and application security scanning to know when to trigger this rebuild process.

Container images may contain operating system components, libraries, and binaries along with the actual application code as a bundled unit, and running that container implies that all elements of that container are trusted. By default, Kubernetes places no restrictions on where container images are allowed to be pulled from, so it is easy and covenient to reference an image built by a single individual in the community or one built by a trusted organization. Limiting where container images can be pulled from to a known list of container registries reduces the chances of untrusted container images being run in the cluster along side critical applications and data.

Application code and container images that contain application code should abstract away configuration settings and secret values that will change per environment so that they can be modified without having to recompile or rebuild the container image. This enhances portability, decreases the number of places secret material is stored on disk, and makes it possible to leverage a dynamic secrets management system to automatically handle short-lived credentials and revocation/rotation functions.

Application code running inside container images still requires all of the diligence of application-level security testing to help avoid classes of vulnerabilities that could allow for container compromise or leakage of cloud credentials. When container images are built using automation, there are opportunities for static and dynamic code analysis to be performed automatically to supplement the human review of the source code or application penetration testing efforts.