Credentials Leaking with Subdomain Takeover

tl;dr We’re disclosing new techniques to steal sensitive data in localStorage (like API keys and passwords) via subdomain takeover.

A few months ago, I pasted an API key into a SaaS product’s documentation portal to test out a new API in the browser. 

When I was done, I closed the tab and went on to other work. 

A week later, I needed to test another API endpoint, so I opened up the same vendor’s documentation portal and to my surprise, my API key was sitting in the “Authorization” field.

I was shocked! How? I wasn’t logged in. How would they have my API key? 

Turns out, the documentation platform (ReadMe) stores API keys in localStorage. What’s worse, ReadMe is vulnerable to subdomain takeovers. This kicked off months-long research into the second-order effects of subdomain takeovers - namely, sensitive data exfiltration from localStorage.

Typically, threat actors will use a subdomain takeover to phish for credentials, and sometimes they’re chained with other bugs to bypass a Content Security Policy (CSP), or get past a CORS restriction etc.. But exfiltrating live API keys from localStorage via a subdomain takeover was a new one to us. As an organization focused on preventing secrets leakage, we thought we’d share our findings with the community.

The rest of this post is a case study on how we found the subdomain vulnerability + leveraged it to exfiltrate localStorage information. While we focus on the organization ReadMe, the purpose of this post is to raise awareness among the security community that storing sensitive data in localStorage can lead to unforeseen consequences.

Note: Before publishing this post, we (a) reported our findings to ReadMe, (b) provided ReadMe with 90 days to address the vulnerabilities, (c) notified all of ReadMe’s impacted customers, and (d) pushed a stop-gap fix to prevent attackers from taking over the impacted customers’ subdomains.

Part 1: A Subdomain Takeover on ReadMe

A occurs when a threat actor controls the content on a victim’s subdomain. This usually occurs when a victim leaves a dangling CNAME pointed to a previously-used SaaS product, and that SaaS product allows an attacker to associate a new account that they control with that same subdomain. The impact of these events is particularly severe when an attacker can supply arbitrary JavaScript that is rendered on pages located on the subdomain (which is the case here).

ReadMe is a very popular SaaS product that helps organizations, like Asana, Rackspace, Notion and Crunchbase, turn API documentation into an interactive developer experience. Readme users create projects to house their API documentation and then use all types of different features to customize the developer experience. It’s pretty cool. 

By default, ReadMe projects are hosted on <subdomain>.readme.io. Like many SaaS products, ReadMe allows users to provide their own custom domain.

We identified hundreds of instances of customers (and churned customers) that removed ReadMe content from their supplied subdomain, but left the CNAME record pointing to ReadMe’s servers (ssl.readmessl.com). We call these stale DNS entries “dangling CNAME records”.

Crucially, ReadMe does not prevent users from inputting domains into the custom domain field that were previously associated with another account.

As a result, a threat actor could input any subdomain with a dangling CNAME record pointed to ssl.readmessl.com into their own custom domain field and take over that organization’s subdomain.

To verify the subdomain takeover, we created two test accounts. On the first account, we added a custom domain pointing to the subdomain docs.webapptests.com and then added a CNAME record pointing to ssl.readmessl.com. 

On the second account, we attempted to add docs.webapptests.com as the custom domain; however, since the first account still had that subdomain in its custom domain field, the request was denied.

Then, we deleted the first account, but left the CNAME record dangling (to simulate what many ReadMe customers do). 

On the second account, we again tried to add docs.webapptests.com to the custom domain field and this time it worked! The second account could control the contents of the first account’s subdomain.

So far, this is a pretty standard subdomain takeover. 

Part 2: Credential Theft

We discovered multiple locations within ReadMe’s service that allows users to supply arbitrary code, such as JavaScript, CSS, SVG files, etc. 

Each input provides a threat actor with the opportunity to force anyone navigating to the taken-over subdomain to execute arbitrary code in their browser. This represents a significant vulnerability to each impacted organization, but still doesn’t stretch beyond the average subdomain takeover.

Here’s where it gets interesting.

One of ReadMe’s core features (see screenshot above) is providing developers with the ability to input an API key directly into the documentation portal and send a test request. 

This feature helps organizations increase developer adoption, since a new developer can start testing the API immediately. In fact, I unknowingly used this ReadMe feature a few months back. 

I was testing a service provider’s API to ensure TruffleHog’s secret verification worked as expected. I added an API key into the service provider’s documentation portal, ran a few test API calls and then moved on to other work. 

A couple weeks later, I needed to run a similar test, so I re-visited that same API provider’s documentation and to my shock, my API key was sitting in clear text in the “Authorization” field.

It turns out that when users input a secret credential, ReadMe saves the key in localStorage under a variable named @readme/reference:auth.

Although the value is encrypted, the decryption key is located in a cookie named ekfls.

After parsing ReadMe’s JavaScript source code, I discovered the ekfls value is used as a symmetric key to encrypt/decrypt the secret credential located in @readme/reference:auth using AES. 

So now, consider the following attack chain: 

1. Threat actor creates a free account on ReadMe.com.

2. Threat actor conducts a subdomain takeover of a current (or churned) ReadMe customer. 

3. Threat actor adds custom JavaScript that gets the decryption key from a user’s cookies, gets the encrypted credential from a user’s localStorage, decrypts it using AES and then exfiltrates the secret (API key) to a server the attacker controls.

4. Threat actor induces a victim to visit the taken-over subdomain (watering hole attack, email, etc) and then steals their API key. 

While we’re not including the full JavaScript exploit code, it’s trivial to write (less than 20 lines). 

To verify this attack path, we wrote the JavaScript exploit, added it to the Custom HTML field and then visited the site as a simulated victim. It worked! Below is a screenshot of our team successfully exfiltrating a sample API secret to a remote server we controlled.

ReadMe’s Response

If you’re a ReadMe customer, you might be curious how the Readme team addressed our vulnerability disclosure. After attempting to contact them several times over a two month period, we finally received this message from their team:

Despite explaining the elevated risk to their customers of this vulnerability, ReadMe decided to accept the risk on behalf of their users. While we disagreed with their analysis, we understand that building an enterprise product isn’t easy and they chose to prioritize other work. The challenge is this security vulnerability doesn’t really impact Readme, it impacts their customers (and churned customers).

Since we weren’t comfortable leaving ReadMe’s customers vulnerable, we created a test ReadMe account and assigned nearly 600 vulnerable subdomains to our account. This means an attacker cannot take over those subdomains, unless ReadMe removes them from our test account (please don’t!). 

Additionally, we sent an email to every single impacted organization explaining the vulnerability and provided remediation instructions (ie: remove the CNAME record).

A couple weeks before publishing this post, we reached out directly to the CEO and he replied that they’d prioritize pushing a fix. While a subdomain takeover is still possible at the time of publication, we’re happy to share that they’re working on a fix.

Our Advice to ReadMe

In our disclosure to ReadMe, we provided remediation guidance. We’re including a copy of our advice below, in case your organization implements a custom domain feature too.

1. Consider requiring users to add an additional DNS record to prove ownership over a subdomain. When an account attempts to add a subdomain, generate a unique value and require the user to create a TXT record with that value. Verify both the CNAME record and TXT record prior to associating an account with a custom domain.

   1. Google follows a similar process for verifying domain ownership.

2. If the first option is not possible, consider locking subdomains to a particular account the first time they are registered. If another account wants to use that subdomain, they’ll have to contact customer support to verify their ownership of the domain. I imagine this is a rare occurrence, so it shouldn’t impede user experience or cost customer support too much.

3. Review the information your application stores in localStorage and sanitize it, since threat actors can often exfiltrate that data.

Conclusion

Subdomain takeovers are pretty standard, but when targeted toward an application that stores sensitive data in user’s localStorage, their impact becomes much more serious. Unfortunately, it’s often up to the SaaS provider to remediate this vulnerability on behalf of their clients, despite the fact that the vulnerability doesn’t directly impact the provider’s security. If you are (or were) a Readme customer, check your DNS records for dangling CNAME entries pointing to ssl.readmessl.com.