Luke Marshall

Preventing a medical supply chain attack by revoking this Docker API key, and hash digest pinning

Preventing a medical supply chain attack by revoking this Docker API key, and hash digest pinning

Luke Marshall

TL;DR


Toradex confirmed that no unauthorised access or breach has occurred. Using TruffleHog Public Monitoring, we identified a leaked Docker token that could have poisoned the supply chain of Toradex, a Swiss manufacturer whose embedded systems run in medical, defense, and industrial devices. Found in plaintext in a public GitLab CI artifact, the token gave write access to all 384 of Toradex's Torizon container images, pulled over 4.5 million times, letting anyone push tampered code into the images those devices are built from.

Discovering the token

A Docker Hub organization access token (prefixed dckr_oat_) is tied to a single organization rather than a person. This one belonged to Torizon, Toradex's embedded-systems toolset, and it carried write access to all 384 of the org's images: roughly 4.5 million pulls in total, the largest being torizon/weston at nearly a million on its own.

The token sitting in plain text inside the uploaded piuparts-out/logs/ artifact.

TruffleHog verified the token was live, and we confirmed its reach without changing anything in the registry. We minted a JWT from Docker's auth service and decoded the scope it granted on Torizon's flagship repository, torizon/weston: both pull and push. torizon/weston is an image Toradex devices use to draw their on-screen interface, the layer that renders what appears on the display and handles touch and input. It is the base that countless Torizon devices with a screen build on.

The torizon/weston repository on Docker Hub, the Torizon org's flagship image.

We minted a JWT for torizon/weston and decoded the granted scope: both pull and push.


Poisoning the supply chain

We want to preface this section by saying it is speculation, and we are making some assumptions. But given the token's access and Toradex's own documentation, this attack avenue seems realistic. We cannot see which devices pull these images, nor did we try to push anything to them that would let us find out.

The token has push access to 384 of Torizon's images. One of these, torizon/weston, is the image Toradex devices use to draw their on-screen interface, the layer that renders the display and handles input. Toradex markets the same platform into healthcare and defense (radar and surveillance, drones and unmanned vehicles). We cannot confirm that any specific medical or defense product ships the weston container, but if one does, an attacker who overwrites that image is running their own code on the screen an operator reads and the input they give it.

Here is how this might be achieved. A Torizon application update is a docker-compose file that lists the container images to run, and it is that compose file, not the image bytes, that Torizon Cloud signs under the Uptane standard. A device only installs a compose file signed with the project's keys. But when that compose references an image by a mutable tag, the signature says nothing about what the tag contains. At install time the device runs docker-compose pull and fetches whatever currently sits behind the tag, so an attacker who has overwritten torizon/weston:<tag> with the leaked token gets their image installed. The signed compose still points at the tag, and the signature never covered the image it resolves to.

1. Reference

Compose by tag

Image referenced by mutable tag.

2. Sign

Compose signed

Signature covers the file, not the image.

3. Tamper

Push token

Attacker overwrites the tag.

4. Install

Device pull

Device pulls the tampered tag.

To be fair about the limits, Torizon ships a defense against this. The signing tool can canonicalize a compose file, which pins each image to its content hash instead of a tag, and a canonical compose is required for offline updates and recommended in general. A hash-pinned deployment is safe here: tampered content produces a different hash, so the device refuses to pull it. Toradex told us its internal data shows that more than 95% of production users canonicalize, as recommended. The exposure is the remaining non-canonical case: a compose that references torizon/weston by a mutable tag and shipped as an online update. That is a worst-case scenario rather than the default, but it is a realistic one, and where it holds the Torizon supply chain could still be poisoned with this token.


What went wrong?

GitLab artifacts are the files a CI job is configured to keep after it finishes: build outputs, test reports, logs, anything the pipeline writes to a path the job lists. They are stored separately from the live job log and stay downloadable afterwards, and on a public project anyone can download them without authenticating.

GitLab masks variables and secrets inside build logs, but not inside artifacts. On the face of it, if you look at the log from a job that just ran, you would see your secrets correctly [MASKED].

In the streamed job log, GitLab masks the value. Captured from the job trace.

In the Toradex build this is exactly what happened: GitLab's masking caught the PAID_DOCKERHUB_TOKEN and masked it in the live log.

The leak came from the artifact instead. The job runs piuparts to test a freshly built package, and piuparts logs every command it runs at full verbosity, including the environment that carries the token. Those logs are written into its piuparts-out/ directory, which the job uploads as an artifact on every run, even failed ones. Masking never touched that file, so the same line that read [MASKED] in the live log sat in full plaintext in the downloadable artifact for the seven days it stayed live.

The same line in the uploaded artifact, with the token in plaintext.


Disclosure

We reported the leaked token to Toradex. They rotated it, fixed the CI configuration that was writing the token into the artifact, and reviewed their Docker Hub audit logs for any unauthorised access.

Toradex conducted an internal security review and shared some of the results with us. The review covered 48,818 logged events across the period the token was exposed. They found no push events to production tags, and matched the digest of every other push that was recorded during the exposed period to a digest from a legitimate internal CI pipeline run. The audit logs also showed that no one other than Toradex and Truffle Security researchers used the token while it was exposed.

Toradex also told us they are adding GitLab CI artifact bundles to their existing TruffleHog scans to catch this class of issue earlier.

Truffle Security does fantastic work protecting the open-source ecosystem, and we're grateful to them for finding and responsibly disclosing this leak before a malicious attacker could exploit it. Their report allowed us to rotate the affected credential quickly and review Docker Hub audit logs for the full exposure period. That review found no registry activity inconsistent with our expected CI pipelines. We've also added more security controls to reduce the risk of this class of issue in the future, including adding GitLab CI artifact bundles to our existing TruffleHog scans.

Security Response Team, Toradex


Key Takeaways

Masking only applies to job logs. Any CI secret written to a build artifact is uploaded in plaintext, even when the same value reads [MASKED] in the live trace.

  • Treat artifacts as published output. Build outputs, test logs, and archives leave the runner and stay downloadable, and on a public project anyone can fetch them without authenticating.

  • Scan artifacts for secrets, not just logs. If a value is worth masking in the log, it is worth detecting in the files a pipeline emits. You can scan a downloaded artifact with TruffleHog, which extracts the archive and only reports live credentials:


    trufflehog filesystem artifact.zip --only-verified


  • Treat any exposed secret as compromised. We downloaded artifacts containing secrets that have since been deleted, but deletion is not remediation. Once a credential has appeared in a public location, assume it is leaked and rotate it.

  • Pin to digests, not tags. Canonicalization helps contain the blast radius downstream even when an upstream image is compromised. Because a hash-pinned deployment refuses any content that does not match the exact digest it expects, a tampered image pushed to a mutable tag simply will not be pulled. It is one of the strongest defenses against this class of supply chain attack.

TL;DR


Toradex confirmed that no unauthorised access or breach has occurred. Using TruffleHog Public Monitoring, we identified a leaked Docker token that could have poisoned the supply chain of Toradex, a Swiss manufacturer whose embedded systems run in medical, defense, and industrial devices. Found in plaintext in a public GitLab CI artifact, the token gave write access to all 384 of Toradex's Torizon container images, pulled over 4.5 million times, letting anyone push tampered code into the images those devices are built from.

Discovering the token

A Docker Hub organization access token (prefixed dckr_oat_) is tied to a single organization rather than a person. This one belonged to Torizon, Toradex's embedded-systems toolset, and it carried write access to all 384 of the org's images: roughly 4.5 million pulls in total, the largest being torizon/weston at nearly a million on its own.

The token sitting in plain text inside the uploaded piuparts-out/logs/ artifact.

TruffleHog verified the token was live, and we confirmed its reach without changing anything in the registry. We minted a JWT from Docker's auth service and decoded the scope it granted on Torizon's flagship repository, torizon/weston: both pull and push. torizon/weston is an image Toradex devices use to draw their on-screen interface, the layer that renders what appears on the display and handles touch and input. It is the base that countless Torizon devices with a screen build on.

The torizon/weston repository on Docker Hub, the Torizon org's flagship image.

We minted a JWT for torizon/weston and decoded the granted scope: both pull and push.


Poisoning the supply chain

We want to preface this section by saying it is speculation, and we are making some assumptions. But given the token's access and Toradex's own documentation, this attack avenue seems realistic. We cannot see which devices pull these images, nor did we try to push anything to them that would let us find out.

The token has push access to 384 of Torizon's images. One of these, torizon/weston, is the image Toradex devices use to draw their on-screen interface, the layer that renders the display and handles input. Toradex markets the same platform into healthcare and defense (radar and surveillance, drones and unmanned vehicles). We cannot confirm that any specific medical or defense product ships the weston container, but if one does, an attacker who overwrites that image is running their own code on the screen an operator reads and the input they give it.

Here is how this might be achieved. A Torizon application update is a docker-compose file that lists the container images to run, and it is that compose file, not the image bytes, that Torizon Cloud signs under the Uptane standard. A device only installs a compose file signed with the project's keys. But when that compose references an image by a mutable tag, the signature says nothing about what the tag contains. At install time the device runs docker-compose pull and fetches whatever currently sits behind the tag, so an attacker who has overwritten torizon/weston:<tag> with the leaked token gets their image installed. The signed compose still points at the tag, and the signature never covered the image it resolves to.

1. Reference

Compose by tag

Image referenced by mutable tag.

2. Sign

Compose signed

Signature covers the file, not the image.

3. Tamper

Push token

Attacker overwrites the tag.

4. Install

Device pull

Device pulls the tampered tag.

To be fair about the limits, Torizon ships a defense against this. The signing tool can canonicalize a compose file, which pins each image to its content hash instead of a tag, and a canonical compose is required for offline updates and recommended in general. A hash-pinned deployment is safe here: tampered content produces a different hash, so the device refuses to pull it. Toradex told us its internal data shows that more than 95% of production users canonicalize, as recommended. The exposure is the remaining non-canonical case: a compose that references torizon/weston by a mutable tag and shipped as an online update. That is a worst-case scenario rather than the default, but it is a realistic one, and where it holds the Torizon supply chain could still be poisoned with this token.


What went wrong?

GitLab artifacts are the files a CI job is configured to keep after it finishes: build outputs, test reports, logs, anything the pipeline writes to a path the job lists. They are stored separately from the live job log and stay downloadable afterwards, and on a public project anyone can download them without authenticating.

GitLab masks variables and secrets inside build logs, but not inside artifacts. On the face of it, if you look at the log from a job that just ran, you would see your secrets correctly [MASKED].

In the streamed job log, GitLab masks the value. Captured from the job trace.

In the Toradex build this is exactly what happened: GitLab's masking caught the PAID_DOCKERHUB_TOKEN and masked it in the live log.

The leak came from the artifact instead. The job runs piuparts to test a freshly built package, and piuparts logs every command it runs at full verbosity, including the environment that carries the token. Those logs are written into its piuparts-out/ directory, which the job uploads as an artifact on every run, even failed ones. Masking never touched that file, so the same line that read [MASKED] in the live log sat in full plaintext in the downloadable artifact for the seven days it stayed live.

The same line in the uploaded artifact, with the token in plaintext.


Disclosure

We reported the leaked token to Toradex. They rotated it, fixed the CI configuration that was writing the token into the artifact, and reviewed their Docker Hub audit logs for any unauthorised access.

Toradex conducted an internal security review and shared some of the results with us. The review covered 48,818 logged events across the period the token was exposed. They found no push events to production tags, and matched the digest of every other push that was recorded during the exposed period to a digest from a legitimate internal CI pipeline run. The audit logs also showed that no one other than Toradex and Truffle Security researchers used the token while it was exposed.

Toradex also told us they are adding GitLab CI artifact bundles to their existing TruffleHog scans to catch this class of issue earlier.

Truffle Security does fantastic work protecting the open-source ecosystem, and we're grateful to them for finding and responsibly disclosing this leak before a malicious attacker could exploit it. Their report allowed us to rotate the affected credential quickly and review Docker Hub audit logs for the full exposure period. That review found no registry activity inconsistent with our expected CI pipelines. We've also added more security controls to reduce the risk of this class of issue in the future, including adding GitLab CI artifact bundles to our existing TruffleHog scans.

Security Response Team, Toradex


Key Takeaways

Masking only applies to job logs. Any CI secret written to a build artifact is uploaded in plaintext, even when the same value reads [MASKED] in the live trace.

  • Treat artifacts as published output. Build outputs, test logs, and archives leave the runner and stay downloadable, and on a public project anyone can fetch them without authenticating.

  • Scan artifacts for secrets, not just logs. If a value is worth masking in the log, it is worth detecting in the files a pipeline emits. You can scan a downloaded artifact with TruffleHog, which extracts the archive and only reports live credentials:


    trufflehog filesystem artifact.zip --only-verified


  • Treat any exposed secret as compromised. We downloaded artifacts containing secrets that have since been deleted, but deletion is not remediation. Once a credential has appeared in a public location, assume it is leaked and rotate it.

  • Pin to digests, not tags. Canonicalization helps contain the blast radius downstream even when an upstream image is compromised. Because a hash-pinned deployment refuses any content that does not match the exact digest it expects, a tampered image pushed to a mutable tag simply will not be pulled. It is one of the strongest defenses against this class of supply chain attack.

infra