When validating assumptions about the necessary Google Cloud IAM permissions for a given situation, using your own identity for testing can be challenging. It’s difficult to fully isolate the exact permissions without a “clean” workspace, a separate identity, and a straightforward way to see what’s going on. We’ll show you one of the ways we approach creating and verifying least-privilege Custom IAM Roles using the gcloud sdk Docker image, Data Access Logging, and the IAM Policy Troubleshooter.
Nearly every compliance and/or security framework includes a mention of “following the principle of least-privilege” when granting users access to resources. It’s one of the most critical steps necessary to limit an attacker’s reach during a breach. Yet the most frequent issue we see with Google Cloud IAM is over-granting permissions to a user or service account. Why is that?
The following list of challenges highlights why it can be difficult to achieve “least-privilege” with IAM:
Project Owner“Basic” roles in the interest of time. Securing and hardening is deferred to a later point in time (sometimes never) when the solution is proven and ready for actual use. Accepting this type of “IAM Debt” early on in a project has consequences in that it gains compounding “interest” the longer it is allowed to remain. As time passes, the cost and complexity of refactoring and re-testing IAM to a point of “least-privilege” grows beyond the perceived value of addressing the risk.
A request comes in asking us, the Infrastructure Team, for
[email protected] to be able to review the firewall rules in the GCP project
temp-k8s-272600. We approve of this request, but our challenge is now to find a way to grant this access according to “least-privilege” and with high confidence it’s going to work.
We have a few choices:
[email protected]the Basic Role named Project Owner to literally take IAM out of the equation. You laugh, but it’s been done before.
[email protected]the Basic Role named Project Viewer, but that has over 1600 permissions because it grants read access to nearly every API possible. Knowing that there are numerous places where APIs can expose credentials or other sensitive information via
LISTrequests, we’ll also avoid this approach.
[email protected]the Predefined Role named Security Reviewer, but that has over 775 permissions. We’re just talking about firewall rules, so this also suffers from lack of “least-privilege”.
Let’s go with Option #4.
The first step is to create an ephemeral environment with a dedicated identity to use for testing purposes that is separate from our currently logged-in admin user identity.
In a separate terminal, run a “throw-away” gcloud sdk docker image using:
$ docker run --rm -it google/cloud-sdk:latest bash
--rm means it will delete itself when it exits and make cleanup much simpler.
From a system terminal, copy the JSON Service Account key file into the container’s
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a54b4d75c39d google/cloud-sdk:latest "bash" 2 hours ago Up 2 hours $ docker cp ~/Downloads/temp-k8s-272600-c64255ad64cf.json a54b4d75c39d:/creds.json
Within the docker session, run:
# gcloud auth activate-service-account --key-file=/creds.json # gcloud config set project temp-k8s-272600
Now, we have a dedicated identity with no prior IAM access in a dedicated terminal session that we can use to run any
kubectl command we need. Also, the image is debian-based, so you can install tools as needed with
apt update and
apt-get install <package>.
[email protected] will be running commands like
gcloud compute firewall-rules list, we can now try that from within our docker container:
# gcloud compute firewall-rules list
This command will hang until it times out and provides some critical information:
ERROR: (gcloud.compute.firewall-rules.list) Some requests did not succeed: - Required 'compute.firewalls.list' permission for 'projects/temp-k8s-272600'
[email protected] lacks the permisson
compute.firewalls.list on the resource named
projects/temp-k8s-272600. So, we need to use an existing or create an IAM Role that includes that permission and attach it to that project.
Visit the IAM Roles Listing for the
temp-k8s-272600 and type
compute.firewalls.list into the “Filter Table” search box:
And it will return about 17 non-Service Agent Roles. However, if we examine the included permissions of each one, there aren’t any sufficiently narrow enough for our use case and grant far too much access across multiple APIs we don’t need for this use case.
Now that we know we should create a Custom IAM Role, we can call it
Firewall Rule Reviewer and include just the
And then assign it at the project level from our non-docker terminal:
$ gcloud projects add-iam-policy-binding temp-k8s-272600 --member=serviceAccount:[email protected] --role=projects/temp-k8s-272600/roles/firewall_rule_reviewer
Back in our docker session, the command now works as expected:
# gcloud compute firewall-rules list NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED default-allow-icmp default INGRESS 65534 icmp False default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp False default-allow-rdp default INGRESS 65534 tcp:3389 False default-allow-ssh default INGRESS 65534 tcp:22 False
describe one of the firewall rules:
# gcloud compute firewall-rules describe default-allow-ssh ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource: - Required 'compute.firewalls.get' permission for 'projects/temp-k8s-272600/global/firewalls/default-allow-ssh'
Despite having the power of LIST, we lack permissions to
describe that specific resourse. It’s missing the
.get permission. Knowing that we want to allow
describe-ing of all firewall rules in the project, we need to add
compute.firewalls.get to our custom IAM Role, too.
Now that we know the identity/principal, the resource, and the needed permission(s), we can also use the IAM Policy Troubleshooter to help, well, troubleshoot this missing permission.
First, let’s see what it looks like with something that already works. e.g. the
On the left-hand side, we can see our Custom IAM Role is the only green check mark that grants this access. If we re-query but for the
So we know for certain that no other IAM Role is accidentally granting us this permission through another binding or form of inheritance.
Now, revisiting our Custom Role named
Firewall Rule Reviewer and adding
We can keep the IAM Binding of our test service account to this custom IAM role at the project level in
Now, if we add a combined query for both
We can see that our Custom IAM Role again is the only one that grants these permissions:
And that our command back in the docker session now works:
# gcloud compute firewall-rules describe default-allow-ssh allowed: - IPProtocol: tcp ports: - '22' ...snip... sourceRanges: - 0.0.0.0/0
We can now modify the IAM binding to be
[email protected] instead of our test service account, confident that things should work as expected.
We can exit out of the docker container, safe in the knowledge that its filesystem is now cleared.
We can delete the service account key. In our case:
$ rm ~/Downloads/temp-k8s-272600-c64255ad64cf.json
If we didn’t modify the IAM binding to drop the user
[email protected] in for our test service account, we can run the following to unbind it:
$ gcloud projects remove-iam-policy-binding temp-k8s-272600 --member=serviceAccount:[email protected] --role=projects/temp-k8s-272600/roles/firewall_rule_reviewer
[email protected] calls us the next day saying they aren’t able to list the firewall rules. You might want to check the audit logs to see what’s going on, right? Well, get in your time machine, go back a day, and enable Data Access Audit Logging for at least the
The Admin Activity logs are enabled by default, so a certain set of logs that “contain log entries for API calls or other administrative actions that modify the configuration or metadata of resources. For example, these logs record when users create VM instances or change Identity and Access Management permissions.”
By contrast, Data Access audit logs “contain API calls that read the configuration or metadata of resources, as well as user-driven API calls that create, modify, or read user-provided resource data.” It’s the Data Access audit logs that most use-cases require to be able to troubleshoot permissions issues.
If you just enabled audit logging on the Project for the
Compute API, wait a few minutes, and have
[email protected] try again. Then, go to
Logging > Logs Explorer in the
temp-k8s-272600 project, filter on the last 15 minutes, and use the query:
to filter on all actions taken by this identity. It should return the following logs:
We hope you found this approach useful for following the principle of least privilege when making and debugging changes to IAM Roles and their permissions. If you have any GCP IAM questions for us, feel free to reach out!