Running TruffleHog in Travis CI

This post is part of a series documenting how to incorporate TruffleHog into CI/CD platforms. If you use GitHub Actions, Azure, or Circle CI, please see those posts. If you use another tool, please let us know and we’ll do our best to write a post for your use case.

TruffleHog supports scanning for exposed credentials in Travis CI pipelines and job log files. The steps required to scan in a pipeline differ from a log file, so we broke them down into different sections below.

Adding TruffleHog to a New Travis CI Pipeline

Let’s start by understanding the code required to prepare a Travis CI/CD Pipeline to run Trufflehog. (Want to skip the tutorial? View the code here.)

THE PIPELINE FILE

To run a Travis CI workflow, developers must place a file named .travis.yml in the main directory of their repository. 

PIPELINE TRIGGERS

The first lines in our Travis CI YAML file define when we will run TruffleHog. 

branches: 
  only: 
  -

At a minimum, we recommend running TruffleHog against all PRs as well as all pushes directly to the production branch (main in this example). Read more about Travis CI’s branches directive here.

START DOCKER

In CI/CD, we recommend running TruffleHog using Docker. It’s easier than installing Go and compiling from scratch. Travis CI has a convenient services descriptor that allows developers to install and start specific services, like Docker.

services: 
  -

GIT CLONING

By default, Travis CI clones the 50 most recent commits from the target source code repository. Since we don’t need that much git history (we only want to scan code changes), we prevent the default git clone process.

git: 
  clone: false

Instead of using Travis CI’s built-in git module, we’ll manually calculate our clone depth and execute a shallow clone of our repository.

What’s git clone depth?

Here is the official Git documentation about the --depth flag:

Screenshot of Git’s Documentation on Clone Depth

Essentially, the depth argument limits how far back in git history Travis CI grabs commits. This enables developers to only clone down the commits between the base branch and the current Push or PR commit. This process is called shallow cloning.

Why does git clone depth matter to us?

Restricting git cloning to the new commits enables TruffleHog to run significantly faster. 

Imagine cloning a 2GB git repository every time you want to scan the latest PR or push for secrets. Not only does it take time to clone that much data, but then TruffleHog has to scan the entire repository (all 2GB), every single time. This is repetitive, wasteful and time-consuming work. 

Manual Git Shallow Cloning

This is a four step process.

1. Count the Commits (then add 1)

script:
  - | 
    COMMIT_COUNT=$(curl -L \ 
    -H "Accept: application/vnd.github+json" \ 
    -H "X-GitHub-Api-Version: 2022-11-28" \ 
    -H "Authorization: Bearer $GH_TOKEN" \ 
    
https://api.github.com/repos/$TRAVIS_REPO_SLUG/compare/$TRAVIS_COMMIT_RANGE | jq ".total_commits")

First, we execute a bash script that counts the number of commits present in the PR or Push. Unfortunately, Travis CI does not provide a built-in variable to reference this value in a pipeline. Developers must query the remote repository service’s API to get the commit count. The code above corresponds to GitHub, but other VCS services like GitLab should have similar API endpoints to query.

Our code block calls the cURL command with 3 headers. The first two provide instructions to the API about how we want to interact with and receive data from the API. The third header is an “Authorization” header and requires a Bearer token (GitHub Personal Access Token). 

To securely set your GitHub token in Travis CI, navigate to the repository’s settings and fill out a new Environment Variable named GH_TOKEN. It is incredibly important to leave the “Display Value in Build Log” toggled off; otherwise, your secret will leak in your build logs.

Adding a GitHub PAT as a Travis CI Environment Variable

The URL in the cURL command references two built-in Travis CI values: $TRAVIS_REPO_SLUG and $TRAVIS_COMMIT_RANGE.

The $TRAVIS_REPO_SLUG variable is a string containing the user_name/repo_name of the target repository.

The $TRAVIS_COMMIT_RANGE variable is a string containing the Short SHA-1 hash corresponding to the first and last commits referenced in this pipeline. For example: 1c002dd…​fe3842.

Next, the command uses the jq tool (pre-installed on Ubuntu) to parse GitHub’s JSON response and grab the total commit count. We store this value in the COMMIT_COUNT environment variable. 

Finally, we add one to the COMMIT_COUNT variable.

- COMMIT_COUNT=$((COMMIT_COUNT+1))

Why do we add 1? When running TruffleHog in CI/CD, we recommend using the --since-commit flag, which instructs the secrets scanner to only look for secrets in code changes since a particular commit (not inclusive of that commit). If we’re scanning 3 commits worth of code changes, TruffleHog needs a reference point to one commit prior to our changes (4 commits back). As a result, we’ll take the total commit value determined above and then add one.

2. Identify the Target Branch

- BRANCH="${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}"

Travis CI provides two built-in git branch values: $TRAVIS_PULL_REQUEST_BRANCH and $TRAVIS_BRANCH. 

If the event triggering Travis CI is a Pull Request, then $TRAVIS_PULL_REQUEST_BRANCH will contain the target branch name. Otherwise, we want the $TRAVIS_BRANCH value.

3. Clone the Repository

- git clone --depth=$COMMIT_COUNT --branch=$BRANCH https://github.com/$TRAVIS_REPO_SLUG

Using the $COMMIT_COUNT variable from step 1, the $BRANCH variable from step 2, and the built-in $TRAVIS_REPO_SLUG, we clone our git repository into the pipeline’s VM.

4. Change Directories

- cd $(echo $TRAVIS_REPO_SLUG | cut -d'/' -f2-)

As mentioned previously, the $TRAVIS_REPO_SLUG variable is a string containing the user_name/repo_name of the target repository. However, a remote git repository clones into a local folder with only the repo_name. As a result, we need to separate the user_name from the $TRAVIS_REPO_SLUG variable and then cd into that directory.

Run TruffleHog

With all of the setup completed, we can now run TruffleHog against the cloned repository.

- docker run --rm -v "$(pwd)":/tmp 
  ghcr.io/trufflesecurity/trufflehog:latest --only-verified 
  --fail --no-update git file:///tmp/ --since-commit $(git rev-list --max-parents=0 HEAD)

Above we’re running a fancy docker command that mounts our git repository inside the docker container and checks it for secrets. Here’s a breakdown of that command:

docker run : run a command in a new container

--rm : automatically remove the container when it exits

-v “$(pwd)”:/tmp : mount our Ubuntu machine’s current working directory (where our git code is) into the /tmp folder in our docker container

ghcr.io/trufflesecurity/trufflehog:latest : use the latest version of TruffleHog’s docker image

--only-verified : only report secrets that are verified to be current/valid

--fail : if a secret is discovered, exit the program (which will fail the pipeline)

--no-update : don’t reach out to our update server (you’re already using the latest version + this would only slow things down)

git file:///tmp/ : look through our git repository located at /tmp/.git in the Docker container

--since-commit : only scan for secrets since the commit referenced (not inclusive of that commit)

$(git rev-list --max-parents=0 HEAD) : get the commit hash for the oldest commit in our shallow clone git history

That’s it! If TruffleHog discovers a secret it will fail the build process.

Adding TruffleHog to Existing Travis CI Pipelines

If you already have an existing pipeline and can’t implement git shallow cloning as described in the section above, here’s the code required to add TruffleHog.

services: 
  - docker 

script: 
  - | 
    if [ "${TRAVIS_PULL_REQUEST}" = "true" ] ; then SINCE_COMMIT="main" 
    else SINCE_COMMIT=$(echo ${TRAVIS_COMMIT_RANGE} | cut -f 1 -d '.') 
    fi 
  - docker run --rm -v "$(pwd)":/tmp 
  ghcr.io/trufflesecurity/trufflehog:latest --only-verified 
  --fail --no-update git file:///tmp/ --since-commit ${SINCE_COMMIT} --branch HEAD

Similar to the process above, we first install and run Docker using the services descriptor. 

Then, we check if the build is a PR or Push. 

If it’s a PR, we set an environment variable named SINCE_COMMIT to main, which is our default branch. 

If it’s a Push, we parse the first commit referenced in the TRAVIS_COMMIT_RANGE variable (which contains the range of commits in the push) and assign it to SINCE_COMMIT. This value is our base commit; we don’t scan our base commit, but instead use it as a lower bound for the commits we want to scan. 

Last, we run a Docker command that is nearly identical to the command in the section above. The only changes are we provide the SINCE_COMMIT environment variable to the --since-commit flag and HEAD to the --branch flag. HEAD references the most recent commit in either the PR or push.

Note: HEAD only works if the target branch (main for Pushes and feature-branch-name for PRs) is checked out in git (default Travis CI behavior). If it’s not checked out, you’ll need to provide main to –branch for a Push and TRAVIS_PULL_REQUEST_BRANCH for a PR.

That’s it! If TruffleHog discovers a secret it will fail the build process.

Scanning Travis Job Logs

In Travis CI, every “build” generates a job log file. These files contain detailed information about the CI/CD pipeline execution process during that build.

Travis CI Build Log Output Example

Over the past decade, Travis CI users have leaked thousands of secrets in public log files. Just last year, researchers discovered 73,000 exposed credentials in Travis CI logs! 

Fortunately, Travis CI has since taken steps to reduce the number of publicly exposed secrets; however, this threat vector still remains actively exploited.

To help our users prevent secrets leakage in Travis CI, we extended TruffleHog’s secret scanning engine to Travis CI logs.

SCAN TRAVIS CI JOB LOGS WITH TRUFFLEHOG

If you already have a Travis CI account, click on your avatar, navigate to settings and then copy your API key.

Copying a Travis API Key

In a terminal, run the following TruffleHog command to scan all of your Travis CI build job logs.

trufflehog travisci --token

If a valid secret exists in a job log, you’ll see an output like the screenshot below.

Sample Output from TruffleHog Scanning Travis CI Build Logs

If you discover a secret, we recommend immediately rotating that key.