Darkbit has joined Aqua Security!


Google Cloud IAM Custom Role and Permissions Debugging Tricks

18 January 2021

When validating assumptions about the necessary Google Cloud IAM permissions for a given situation, using your own identity for testing can be challenging. It’s difficult to fully isolate the exact permissions without a “clean” workspace, a separate identity, and a straightforward way to see what’s going on. We’ll show you one of the ways we approach creating and verifying least-privilege Custom IAM Roles using the gcloud sdk Docker image, Data Access Logging, and the IAM Policy Troubleshooter.

IAM Challenges

Nearly every compliance and/or security framework includes a mention of “following the principle of least-privilege” when granting users access to resources. It’s one of the most critical steps necessary to limit an attacker’s reach during a breach. Yet the most frequent issue we see with Google Cloud IAM is over-granting permissions to a user or service account. Why is that?

The following list of challenges highlights why it can be difficult to achieve “least-privilege” with IAM:

An Example Scenario

A request comes in asking us, the Infrastructure Team, for [email protected] to be able to review the firewall rules in the GCP project temp-k8s-272600. We approve of this request, but our challenge is now to find a way to grant this access according to “least-privilege” and with high confidence it’s going to work.

We have a few choices:

  1. Assign [email protected] the Basic Role named Project Owner to literally take IAM out of the equation. You laugh, but it’s been done before.
  2. Assign [email protected] the Basic Role named Project Viewer, but that has over 1600 permissions because it grants read access to nearly every API possible. Knowing that there are numerous places where APIs can expose credentials or other sensitive information via GET or LIST requests, we’ll also avoid this approach.
  3. Assign [email protected] the Predefined Role named Security Reviewer, but that has over 775 permissions. We’re just talking about firewall rules, so this also suffers from lack of “least-privilege”.
  4. Instead of picking a Role at random, we can enumerate the permissions we actually need first and then see which IAM Roles, if any, meet our criteria. This may end up with us granting a narrowly scoped Predefined IAM Role or creating a Custom IAM Role, but we’ll know that it will work and that it’s least-privileged.

Let’s go with Option #4.

Clean Workspace

The first step is to create an ephemeral environment with a dedicated identity to use for testing purposes that is separate from our currently logged-in admin user identity.

  1. Create a new GCP Service Account and export a Service Account Key in JSON format. My project here is named temp-k8s-272600 and the service account is named [email protected].

  2. In a separate terminal, run a “throw-away” gcloud sdk docker image using:

    $ docker run --rm -it google/cloud-sdk:latest bash

    The --rm means it will delete itself when it exits and make cleanup much simpler.

  3. From a system terminal, copy the JSON Service Account key file into the container’s /creds.json path:

    $ docker ps
    CONTAINER ID   IMAGE                     COMMAND   CREATED        STATUS        PORTS     NAMES
    a54b4d75c39d   google/cloud-sdk:latest   "bash"    2 hours ago   Up 2 hours
    $ docker cp ~/Downloads/temp-k8s-272600-c64255ad64cf.json a54b4d75c39d:/creds.json
  4. Within the docker session, run:

    # gcloud auth activate-service-account --key-file=/creds.json
    # gcloud config set project temp-k8s-272600

Now, we have a dedicated identity with no prior IAM access in a dedicated terminal session that we can use to run any gcloud or kubectl command we need. Also, the image is debian-based, so you can install tools as needed with apt update and apt-get install <package>.

Try the Commands

Knowing that [email protected] will be running commands like gcloud compute firewall-rules list, we can now try that from within our docker container:

# gcloud compute firewall-rules list

This command will hang until it times out and provides some critical information:

ERROR: (gcloud.compute.firewall-rules.list) Some requests did not succeed:
 - Required 'compute.firewalls.list' permission for 'projects/temp-k8s-272600'

Our identity, [email protected] lacks the permisson compute.firewalls.list on the resource named projects/temp-k8s-272600. So, we need to use an existing or create an IAM Role that includes that permission and attach it to that project.

View Built-in Roles

Visit the IAM Roles Listing for the temp-k8s-272600 and type compute.firewalls.list into the “Filter Table” search box:

gcp compute firewall iam permissions

And it will return about 17 non-Service Agent Roles. However, if we examine the included permissions of each one, there aren’t any sufficiently narrow enough for our use case and grant far too much access across multiple APIs we don’t need for this use case.

Create a Custom Role

Now that we know we should create a Custom IAM Role, we can call it Firewall Rule Reviewer and include just the compute.firewalls.list permission:

gcp custom iam role firewall list

And then assign it at the project level from our non-docker terminal:

$ gcloud projects add-iam-policy-binding temp-k8s-272600 --member=serviceAccount:[email protected] --role=projects/temp-k8s-272600/roles/firewall_rule_reviewer

Invoke the Commands

Back in our docker session, the command now works as expected:

# gcloud compute firewall-rules list
NAME                    NETWORK  DIRECTION  PRIORITY  ALLOW                         DENY  DISABLED
default-allow-icmp      default  INGRESS    65534     icmp                                False
default-allow-internal  default  INGRESS    65534     tcp:0-65535,udp:0-65535,icmp        False
default-allow-rdp       default  INGRESS    65534     tcp:3389                            False
default-allow-ssh       default  INGRESS    65534     tcp:22                              False

Let’s describe one of the firewall rules:

# gcloud compute firewall-rules describe default-allow-ssh
ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
 - Required 'compute.firewalls.get' permission for 'projects/temp-k8s-272600/global/firewalls/default-allow-ssh'

Despite having the power of LIST, we lack permissions to describe that specific resourse. It’s missing the .get permission. Knowing that we want to allow describe-ing of all firewall rules in the project, we need to add compute.firewalls.get to our custom IAM Role, too.

Using the IAM Policy Troubleshooter

Now that we know the identity/principal, the resource, and the needed permission(s), we can also use the IAM Policy Troubleshooter to help, well, troubleshoot this missing permission.

First, let’s see what it looks like with something that already works. e.g. the compute.firewalls.list permission:

gcp iam policy troubleshooter compute.firewalls.list

gcp iam policy troubleshooter compute.firewalls.list result

On the left-hand side, we can see our Custom IAM Role is the only green check mark that grants this access. If we re-query but for the compute.firewalls.get permission:

gcp iam policy troubleshooter compute.firewalls.get result

So we know for certain that no other IAM Role is accidentally granting us this permission through another binding or form of inheritance.

Modify the Custom Role

Now, revisiting our Custom Role named Firewall Rule Reviewer and adding compute.firewalls.get:

gcp custom iam role compute.firewalls.get compute.firewalls.list

We can keep the IAM Binding of our test service account to this custom IAM role at the project level in temp-k8s-272600 unchanged.

Verifying with IAM Policy Troubleshooter

Now, if we add a combined query for both compute.firewalls.list and compute.firewalls.get :

gcp custom iam role compute.firewalls.get compute.firewalls.list

We can see that our Custom IAM Role again is the only one that grants these permissions:

gcp iam policy troubleshooter compute.firewalls.get result

And that our command back in the docker session now works:

# gcloud compute firewall-rules describe default-allow-ssh
- IPProtocol: tcp
  - '22'

We can now modify the IAM binding to be [email protected] instead of our test service account, confident that things should work as expected.

Clean Up

  1. We can exit out of the docker container, safe in the knowledge that its filesystem is now cleared.

  2. We can delete the service account key. In our case:

    $ rm ~/Downloads/temp-k8s-272600-c64255ad64cf.json
  3. If we didn’t modify the IAM binding to drop the user [email protected] in for our test service account, we can run the following to unbind it:

    $ gcloud projects remove-iam-policy-binding temp-k8s-272600 --member=serviceAccount:[email protected] --role=projects/temp-k8s-272600/roles/firewall_rule_reviewer

API Audit Logging

Let’s say [email protected] calls us the next day saying they aren’t able to list the firewall rules. You might want to check the audit logs to see what’s going on, right? Well, get in your time machine, go back a day, and enable Data Access Audit Logging for at least the Compute API:

gcp iam data access audit logging

The Admin Activity logs are enabled by default, so a certain set of logs that “contain log entries for API calls or other administrative actions that modify the configuration or metadata of resources. For example, these logs record when users create VM instances or change Identity and Access Management permissions.”

By contrast, Data Access audit logs “contain API calls that read the configuration or metadata of resources, as well as user-driven API calls that create, modify, or read user-provided resource data.” It’s the Data Access audit logs that most use-cases require to be able to troubleshoot permissions issues.

If you just enabled audit logging on the Project for the Compute API, wait a few minutes, and have [email protected] try again. Then, go to Logging > Logs Explorer in the temp-k8s-272600 project, filter on the last 15 minutes, and use the query:

protoPayload.authenticationInfo.principalEmail="[email protected]"

to filter on all actions taken by this identity. It should return the following logs:

gcp audit log sa compute firewall list


We hope you found this approach useful for following the principle of least privilege when making and debugging changes to IAM Roles and their permissions. If you have any GCP IAM questions for us, feel free to reach out!