For the past 8 years, security researchers and threat actors have patrolled Travis CI’s public log files for leaked API keys and passwords, often with amazing success. Just last year, security researchers discovered 73,000 credentials in Travis CI logs!
TruffleHog integrates with Travis CI, so we’ve been following their security team’s efforts to tamp down on secrets leakage. After several recent Travis CI security policy changes, we decided to revisit the public log files to determine how often secrets still leak.
Analyzing 4.7 million build logs
Our research process consisted of three steps: (1) Identify the public log file API links and the range of valid build ID integers, (2) Download all 4.7 million build logs, while respecting Travis CI’s API, (3) Run TruffleHog’s `filesystem` command against each downloaded build log. Below, we break down each step in greater detail.
1. Both the Aqua and EdOverflow articles queried api.travis-ci.org; however, Travis CI deprecated travis-ci.org in favor of travis-ci.com. The .org API no longer returned valid build logs; instead, we accessed build logs through api.travis-ci.com, like this:
https://api.travis-ci.com/v3/job/<LOG NUMBER>/log.txt
At the time of research, the latest log ID was 606844645 and the earliest unexpired was 578292151. (Travis CI’s 2022 policy of expiring logs after 365 days limited our research to build logs released in the past year) We discovered the latest log ID by creating a new repository, executing a Travis CI build job, and then examining the output link for the job ID. We identified the earliest available log ID by decrementing our build ID until we received an expiration error.
We downloaded all available 4.7 million public logs.
To find leaks, we passed each log file into TruffleHog and used the `filesystem` subcommand:
Out of the 4.7 million log files, TruffleHog identified only 80 unique, live secrets. For perspective, Aqua’s 2022 research found 73,000 unique secrets (although it’s unclear how many were live and unique). This indicates an almost complete elimination of leaks. Travis CI’s log expiration and scanning are working.
Interestingly, the 80 secrets we discovered appeared across 6,690 build logs, which means the same repositories kept leaking the same credentials, over and over again. However, once an impacted key is rotated, it doesn’t matter how many files leak that key, an attacker would no longer be able to use it.
Further recommendations
While Travis CI has made significant security improvements in the last year, we recommend their team investigate a few additional ways to limit secrets leakage.
ENVIRONMENT VARIABLES
Most of the secrets discovered in our research leaked in the “Setting environment variables” step (as shown below).
This step enables users to inject secrets into their build process, which is why secrets often leaked here. Travis CI provides users with a toggle to display or mask environment variables in build logs.
While the default option is to mask secret data, most of the leaks we discovered are likely due to the display setting being toggled. Perhaps users intentionally leaked their secrets; however, it is more than likely a lack of user education resulted in a misconfiguration (and secret leak).
Assuming this feature is critical for Travis CI’s users, we suggest adding in-line education + a confirmation interstitial to ensure users understand the magnitude of their decision.
FORMAT ISSUES
The remaining leaks were mostly due to output format issues—such as secrets entangled in JSON structures. It’s understandable that Travis CI’s scanners may not pick that up. We recommend users supplement Travis CI’s built-in secret scanning with TruffleHog (or another secret scanner that can parse JSON structures for secrets) to monitor build log output.
RATE LIMITING
Travis CI should consider rate limiting their API. We experienced no rate limits when downloading the 4.7 million log files. Malicious actors could abuse this feature to discover secrets en-masse (similar to the one from the 2015 API incident).
OTHER IDEAS
In a previous blog post, we outlined more general advice for preventing secrets leakage in CI/CD pipelines and believe some of those ideas would benefit Travis CI users as well.
Timeline of events
2015
An internal Travis CI API incident report admitted “We are currently undergoing a distributed attack on our public API”. Press coverage later revealed threat actors were scraping Travis CI’s build logs for GitHub tokens.
2017
A bug bounty report appeared on HackerOne titled A HackerOne employee’s GitHub personal access token exposed in Travis CI build logs. This sparked lots of interest in the bug bounty community, since it provided ethical hackers with a new source of bug bounty findings.
2019
Ed Overflow and other security engineers (including myself), released research documenting the extent of leakage from Travis CI build logs at that time. I even sent a bug bounty report to Grammarly, which documented how a threat actor could have used a GitHub token in Travis CI to access Grammarly’s private repositories. There was no shortage of secrets, and certainly no shortage of security impact.
2022
Three years after our research, the security team at Aqua scanned Travis CI build logs and reported that Public Travis CI Logs (Still) Expose Users to Cyber Attacks. Aqua’s research revealed 73,000 “tokens, secrets, and various credentials” in 8 million sampled logs. The leaked credentials included GitHub access tokens, AWS access keys, database credentials, and Docker Hub passwords, among other types.
A couple months later, Travis CI’s security team announced the first of several major policy changes. In a September 2022 blog post, the team announced that they will expire logs older than 365 days. Log expiration would shorten the potential attack window for any secrets that escape into the wild.
2023
In January 2023, Travis CI began using Trivy and Detect-Secrets to automatically detect and censor accidental secrets exposure. Before that, it was unknown what tools they were using—but some type of leak censoring was in place.
During the summer of 2023, the Truffle Security research team started wondering how effective those policy changes had been. Many of our TruffleHog Enterprise clients ask for secure CI/CD tooling recommendations, so we attempt to stay up-to-date on the security policies of tools like Travis CI. In the spirit of our past research, we attempted to scan (again) all public build logs to quantify the secrets exposed on Travis CI in 2023.
Conclusion
Travis CI has had a battled history with security. For nearly a decade, security researchers and threat actors pilfered secrets from Travis CI build logs. Today, however, the story is changing. Our research demonstrated that leaks have become uncommon: only 80 live secrets among 4.7 million logs. Travis CI’s commitment to addressing secrets leakage, specifically through the use of log expiration and additional secret scanning, has made their platform more secure.
