Every SaaS provider deals with customers exposing their API keys. Sometimes they’re accidentally committed to GitHub, other times they’re leaked in Slack, sometimes s3 buckets.
Is it the provider’s fault a customer exposed a credential? Not entirely, but if there’s not a good runbook, and support for how to identify and remediate leaked credentials, it can make a bad situation worse.
As security teams shift left, the best-in-class organizations are proactively identifying ways to help their customers secure their credentials outside of their own software. While most of their customer’s security hygiene is beyond their control, SaaS providers can take a few steps.
In this post, we highlight Tailscale’s contribution to TruffleHog as a blueprint for how security-first organizations can contribute to open-source security tools while simultaneously strengthening their customer’s security posture.
PR #1719
Screenshot of Tailscale’s PR #1719
The idea for this blog post originated with a new Pull Request submitted to TruffleHog in September. Pull Request #1719 added a new Tailscale API key detector to TruffleHog’s 820+ existing secret detectors.
As our engineering team reviewed the PR, a few things immediately stood out.
First, an engineer from Tailscale, Marwan Sulaiman, submitted the PR. We typically don’t see SaaS providers put this level of care when they contribute their own secret detector.
Second, Tailscale’s approach to API key formatting makes it easy to identify their keys in massive blocks of code. All Tailscale API keys use a consistent and unique format:
Unlike many other SaaS providers that generate a random alphanumeric string, Tailscale’s tskey prefix enables secret scanners to more efficiently identify leaked secrets in the wild.
Third, Tailscale authored an authentication verification endpoint so that secret scanners can effectively validate the authenticity of potentially leaked keys. Seeing Tailscale dedicate engineering time to releasing a new “secret scanning” authentication endpoint delighted our team for a few reasons. This is above and beyond what we see most companies do when they contribute their detectors to TruffleHog.
Tailscale has 5 different credential types. Writing logic to validate leaked credentials against 5 different API endpoints requires more engineering and QA time. Consolidating verification to one endpoint makes Tailscale key verification more performant and easier to maintain.
Unverified secrets often create a large volume of false positives for already-overwhelmed security analysts. By explicitly granting all secret scanners permission to use their API for credential verification, Tailscale is helping to reduce unnecessary alerts.
While recognizing the need for live key validation, Tailscale also endeavored to minimize their client’s data exposure during key verification. In many cases, in order to verify credentials, TruffleHog must interact with an API endpoint that reveals client profile information. As a policy, we do not inspect that information. However, by creating a simple API endpoint that returns a boolean response regarding key validation, Tailscale minimized the data returned about their customer during key verification. See Marwan’s explanation below:
Tailscale Engineer’s Comment on their PR
If you work for a SaaS provider…
We suggest following Tailscale’s lead and proactively taking steps to help strengthen your customer’s credential security posture.
Work with all major secret scanning providers to create a detector for your organization’s secret keys and passwords. We’re happy to help you get started.
Provide a simple endpoint to detect live API keys and passwords and make it explicitly available to all secret scanners.
Consider creating a unique format for your organization’s credentials.
Educate your customers about credential security – even just a callout in the API docs is better than nothing.
There is so much out of your control, but contributing key detection logic to secret scanning vendors and making it easy to tell which keys are live are great first steps.
What’s next?
Looking forward, there are two trends that we’d like to see more providers follow.
First, some organizations are creating API endpoints that revoke API keys. Historically, users have to log into an administrative console, navigate to the API keys/credentials page and then manually revoke a key. This is a friction-filled process. Creating an API endpoint for users that need to immediately revoke an API key enables a more efficient process. Additionally, as secret management solutions progress, this type of endpoint will help automate the secret lifecycle.
Finally, we’re seeing SaaS providers auto-revoke keys discovered in certain contexts, such as a public GitHub repository (for example: see this forum discussion about OpenAI keys auto-revoking). This is an excellent step toward shortening the timeframe available to attackers to discover and then use exposed credentials.
