Darkbit has joined Aqua Security!

...

The Power of Kubernetes RBAC LIST

13 January 2021

One of the potential surprises for newcomers to Kubernetes RBAC is what the subtle, but extremely important differences are between the GET and LIST verbs. This even translates to Google Cloud’s IAM permission model with GKE clusters with opportunities for unintended consequences.

Kubernetes RBAC For Reading Secrets

If you have worked with Kubernetes’ implementation of role-based access control (RBAC) for controlling access to resources in the API server, the verbs named get, list, and watch are probably familiar to you. Here is an example ClusterRole from the documentation:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  # "namespace" omitted since ClusterRoles are not namespaced
  name: secret-reader
rules:
- apiGroups: [""]
  # at the HTTP level, the name of the resource for accessing Secret
  # objects is "secrets"
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

This ClusterRole defines permissions that basically allow “reading” of secrets in all namespaces. Once “bound” to subjects (users, groups, or service accounts), those subjects will have the secrets reading ability:

apiVersion: rbac.authorization.k8s.io/v1
# This cluster role binding allows the user "pat" to read secrets in any namespace.
kind: ClusterRoleBinding
metadata:
  name: read-secrets-global
subjects:
- kind: User
  name: pat # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

The Importance of Kubernetes Secrets

Why are secrets so important in Kubernetes? The key is that Kubernetes stores the long-lived JWT tokens associated with serviceaccounts as native secrets resources alongside regular secrets created by users for attaching to workloads. Because access to secrets is controlled by RBAC, granting the ability to read the contents of all secrets via RBAC is akin to granting access to see the serviceaccount tokens for all serviceaccounts in that namespace. And possession of a serviceaccount token and network access to the API server is all that’s typically required to authenticate as that serviceaccount and be granted the access it was assigned. So, the flow is as follows:

RBAC Read Secrets ==> Read ServiceAccount JWT Tokens ==> Authenticate to API server as all ServiceAccounts ==> Have the effective combination of all RBAC permissions bound to these ServiceAccounts 

In nearly all Kubernetes clusters in practice, at least one serviceaccount is granted sufficient permissions to be cluster-admin , so if anyone has the ability to “read all cluster secrets in all namespaces”, it’s practically the same as stating they have cluster-admin.

From our experience, one of the biggest RBAC mistakes that opens up a direct privilege escalation path from a non-privileged position inside the cluster to cluster-admin is accidental assignment of the ability to read secrets in a namespace where a serviceaccount is bound to cluster-admin. Extreme care needs to be taken to isolate workloads into dedicated namespaces where those privileges are needed to be assigned directly to pods via serviceaccount mounts to help avoid this situation.

Remember, that cluster-admin means full control over cluster resources, running workloads, the credentials housed in the cluster, and one or two steps to becoming root on all worker nodes.

Assign GET instead of LIST?

The wording with kubectl can be confusing. The following command which “lists” all the secrets in the default namespace, despite the command name being get secrets, requires the list permission:

$ kubectl get secrets --namespace default
NAME                  TYPE                                  DATA   AGE
db-creds              Opaque                                2      2d
default-token-94v59   kubernetes.io/service-account-token   3      3d

Whereas the following command requires the get permission and the knowledge of the secret resource name. In this case, db-creds:

$ kubectl get secret --namespace default db-creds
NAME       TYPE     DATA   AGE
db-creds   Opaque   2      2d

So, it looks like granting list just returns the names, but that’s incorrect!

Granting list permissions allows you to return the contents of all of that resource type despite the default output from kubectl get secrets only showing what appears to be a summary listing. If the --output=json flag is applied, the full contents of the secrets are displayed, and no additional permissions were needed:

$ kubectl get secret --namespace default --output=json
{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "v1",
            "data": {
                "password": "YWRtaW4="   # admin
                "username": "YWRtaW4="   # admin
            },
            "kind": "Secret",
            "metadata": {
                "name": "db-creds",
                "namespace": "default",
            },
            "type": "Opaque"
        },
        {
            "apiVersion": "v1",
            "data": {
                "ca.crt": "LS0tLS1CRUdJTiBDRVJU
                ...snip...
                UtLS0tLQo=",
                "namespace": "ZGVmYXVsdA==",
                "token": "ZXlKaGJHY2lPaUpTVXpJMU5
                ...snip...
                NnAtNlBDOGxOUlBzTFhCYjZuOTF4TDg0djZDVmFpSnRlTG9R"  # Base64 encoded JWT token used to authenticate directly to the API server as this serviceaccount
            },
            "kind": "Secret",
            "metadata": {
                "name": "default-token-94v59",
                "namespace": "default",
            },
            "type": "kubernetes.io/service-account-token"
        }
    ],
    "kind": "List",
    "metadata": {
        "resourceVersion": "",
        "selfLink": ""
    }
}

Again, you can see that the ability to list secrets in this namespace means that you can connect to a database somewhere as admin/admin and you can also try to authenticate to the API server using the default serviceaccount’s JWT token. If there were more serviceaccounts in this namespace, we’d see all of their JWT tokens, and we could loop through each one to see if they had more permissions than what we currently have. Using a token with kubectl:

$ kubectl --token myjwttokenhere get secrets --namespace default

Enumerating permissions available to this serviceaccount token shows that we are cluster-admin as there is likely a ClusterRoleBinding that grants these permissions to our serviceaccount:

$ kubectl --token myjwttokenhwere auth can-i --list
Resources                                       Non-Resource URLs   Resource Names   Verbs
*.*                                             []                  []               [*]
                                                [*]                 []               [*]
...snip...

Layering on Google Cloud IAM

Google Kubernetes Engine (GKE) Clusters have a special, automatic integration with Google Cloud IAM, and this allows Google Cloud administrators to grant permissions to GKE cluster resources (like pods and secrets) using the standard IAM permissions model in combination with or instead of native RBAC permissions defined inside each GKE cluster. IAM Roles like Kubernetes Engine Admin and Kubernetes Engine Developer are populated with individual permissions in the container.*.* format. For example:

# Kubernetes Engine Admin
...snip...
container.clusters.get
container.clusters.getCredentials
container.clusters.list
...snip...
container.secrets.create
container.secrets.delete
container.secrets.get
container.secrets.list
container.secrets.update
...snip...

You may have noticed that container.secrets.list is shown. That means, assuming the Google Cloud identity (user, group, or service account) has network access to the GKE cluster API, they can run kubectl get secrets --all-namespaces in all GKE clusters underneath where this permission is assigned. NOTE: This permission is not scoped at the namespace level. It’s at the cluster level. It makes sense, because after all, the identities you give “admin” access to would need to be able to administer all the resources inside the clusters.

But what other built-in IAM Roles have container.secrets.list? If any of them weren’t designated as roles you’d normally give to a small group of administrators, you’d have a potential avenue for privilege escalation via IAM.

Unintended Side Effects

To see which built-in roles have container.secrets.list, we can use the Google Cloud Console:

google cloud console iam roles listing

Or a repo we’ve created at https://github.com/darkbitio/gcp-iam-role-permissions that constantly keeps track of these roles as they change in easily searchable form.

$ git clone https://github.com/darkbitio/gcp-iam-role-permissions.git
$ cd gcp-iam-role-permissions
$ ./list-roles-with-permission.sh container.secrets.list
roles/composer.serviceAgent
roles/composer.worker     # Composer Worker
roles/container.admin     # Kubernetes Engine Admin
roles/container.developer # Kubernetes Engine Developer
roles/container.serviceAgent
roles/containerthreatdetection.serviceAgent
roles/editor              # Editor
roles/gameservices.serviceAgent
roles/multiclusteringress.serviceAgent
roles/owner               # Owner

Ignoring all the *.serviceAgent roles and the basic roles of owner and editor for now, we can see that roles/container.developer and roles/composer.worker are both interesting in that they have the ability to list all secrets in all GKE clusters where they are bound. This is most often done at the project level.

From this, we can draw the following conclusions:

We’ll dive into Cloud Composer in greater detail in a future posting, so stay tuned.