Q: A developer exposes an AWS key in a public git repository. How do you respond?
A. Take down the offending git push.
B. Push a second commit that removes the exposed key.
C. Overwrite your git commit history.
D. Rotate your key.
At Truffle Security, we help organizations discover leaked secrets (passwords, API keys, tokens, etc) before threat actors use them in an attack. But identifying a leaked secret is just the first step in the process. How should a security team respond when a key is exposed?
Despite the temptation to simply take down the code that contained the sensitive data, the most secure remediation strategy is key rotation. In this post, we’ll explain why.
The Problem
Secrets leak everywhere. We recently launched Forager, a public secret scanning tool, which monitors event feeds from NPM and GitHub to identify leaked keys. We found 0.1% of ALL pushes (not commits!) to GitHub contain a valid leaked secret. Our research into the Alexa Top 1 Million sites (post coming soon!) revealed a similarly unexpected discovery. Hundreds of valid API keys, including root AWS keys and admin GitHub tokens, were freely accessible on the internet’s most visited websites.
Depending on the permissions and third-party services involved, a leaked key could provide attackers with the means to orchestrate sophisticated social engineering campaigns or gain control over your entire online infrastructure.
Remediating leaked API keys and tokens requires a systematic and efficient approach. The most secure way to remediate a leaked secret is key rotation.
What is Key Rotation?
Here’s the “Official” definition from NIST:
Official NIST Definition for Key Rotation
In other words, key rotation refers to the process of (1) generating a new API key, (2) rendering the compromised key obsolete, and (3) updating the associated systems with the new key (like your CI/CD pipeline). This practice helps minimize the impact of an exposed API key or password.
Here’s how you rotate Twilio API keys:
Twilio’s Key Rotation
While the GIF above is unique to Twilio, most SaaS companies provide users with a simple way to rotate their secrets.
REGULAR KEY ROTATION
In addition to rotating secrets after inadvertently exposing a key, cryptography experts suggest organizations regularly change their keys. Why?
Attackers might gain access to your keys without your knowledge. Regularly changing the secret shortens the attack window.
An attacker who compromises a cryptographic key can decrypt all of the information encrypted with that key. By reducing the key’s validity time frame, we minimize the data exposure.*
Security teams can thwart attacker attempts to use cracked cryptographic keys by rotating the keys before adversaries could computationally complete their cryptanalysis.*
* These ideas came from this post on Stack Overflow by Mike Ounsworth, a cryptography expert.
Why can’t I just remove the code exposing the secret?
Persistence in Forks and Clones: You cannot force forked or cloned versions of a repository to update. Code exposing an API key in your repository will remain unchanged in others’ versions. Version Control Challenges: Version control systems retain a history of changes, including the commit that introduced the leaked API key. This history can be challenging to manage. Removing the key from the current commit does not erase its existence from historical commits.
Potential Unauthorized Usage: In the time that the key was exposed, unauthorized individuals might have accessed and copied it. (We recently tested this theory by publishing an AWS key to a brand new repository on GitHub; within 10 minutes, someone had used our key). This is particularly concerning when numerous employees or contributors had access to the key, such as in the case of a public repository hosted on GitHub. Key rotation helps address this risk by rendering the leaked key useless, while allowing monitoring systems to track any unauthorized usage attempts.
Mirrored Sources: Sites that mirror repositories on NPM (and similar registries) will often retain a copy of any exposed secrets. By design, mirrors retain multiple versions of software packages to help users avoid breaking changes in new updates. Unfortunately, when developers remove the offending code, the mirrored sources will rarely delete the version containing the leaked secret.
SHOULD I ROTATE THE KEY AND TAKE DOWN THE CODE EXPOSING THE SECRET?
Some organizations will choose to both rotate their keys and remove the impacted code, while others will just perform a key rotation. From a security perspective, as long as you’ve invalidated all hard-coded secrets, attackers can no longer use the exposed secret for malicious purposes. It can be extremely painful for teams to handle force pushes and force pulls, as merging divergent git histories is non-trivial (especially for larger teams)
Many times it may not be worth the trouble moving the old key from history, but consider a few edge cases such as passwords. As an example, if you hard-code a password such as CompanyNameSpring2023 and you “rotate” it by updating Spring2023 to Summer2023, then a moderately savvy attacker would likely guess your new password. In that instance, you should both rotate the exposed secret and remove it from git history (or use random passwords!).
Another challenging example are TLS certificate private keys. Although there is a process to revoke TLS certificates, many devices cache revocation lists, and some ignore them all together. This makes this key dangerous for a period of time after revocation.
Generally we lean towards just rotating the secret and not bothering to purge the history due to the disruptive nature of git history overriding, but the specific context is key.
Conclusion
Key rotation is the best response for remediating an inadvertently exposed secret. There are only three steps: (1) generate a new key, (2) invalidate the leaked key, and (3) update systems to use the new key. When combined with continuous secret scanning for CI/CD systems, communication tools (e.g., Slack, Teams), and file shares, key rotation empowers organizations to fortify their defenses against credential-based breaches.
For more information on key rotation, here are a few solid resources:
