Preventing git leaks

I track most of my code and prose via git and I store it on GitHub in public repos. I do it because I get reliable storage for free that I can access from any computer. And some of the repos might be helpful to others. Also it engages my hubris :-). It’s working nicely but sometimes I get a bad feeling when I push stuff. I’m worried that I might leak some sensitive information like passwords, API keys or tokens.

NOTE: don’t assume that committing secrets is ok when the repo is private. The files or their contents get copied and the platforms holding the repos get compromised.

Spotting secrets in new commits

The obvious solution is to think twice before committing and pushing data. But there are also some helpful tools, like gitleaks. It basically finds and reports secrets in the files you are about to commit. I want to run it whenever I commit something in any of my repos. These are the steps to make that happen:

1. Install gitleaks.
2. Add this to your ~/.gitconfig:

[core]
    hooksPath = ~/.git-global-hooks

3. Create ~/.git-global-hooks/pre-commit:

#!/bin/bash
# Detect secrets in a git repo using https://github.com/zricethezav/gitleaks

if [[ $SKIP == "gitleaks" ]]; then
    echo "skipping gitleaks checks ..."
    exit 0
fi

set -xe

# Check uncommitted changes (parsing output of 'git diff') that had been 'git add'ed.
gitleaks protect --no-banner --staged

If, for some good reason, you want to skip gitleaks when committing: SKIP=gitleaks git commit -m "commit message"

You can also make gitleaks ignore a secret either by #gitleaks:allow inline comment next to the secret or by adding finding’s fingerprint to the .gitleaksignore file at the root of your repo. See docs for details.

Existing commits

The steps above will prevent you from committing secrets from now on. But you should also check existing commits because you might have committed a secret in the past. You can either do it on each commit by adding these lines to ~/.git-global-hooks/pre-commit:

# Check existing commits (parsing output of 'git log -p').
gitleaks detect --no-banner

But on bigger repos this might take several seconds every time you commit. To avoid this you can check all your historical commits in all your repos once. I used gh and runp to do it:

export GHORG=jreisinger # CHANGE ME
mkdir /tmp/$GHORG && cd /tmp/$GHORG
# clone all my (1000) repos in parallel
gh repo list $GHORG --source --limit 1000 | cut -f 1 | runp -p 'gh repo clone'
# check existing commits in all repos in parallel
ls | runp -p 'gitleaks detect --no-banner -s'

If runp exits with 0, all is good. Otherwise scroll up to review the output. To check a repo for committed leaks:

cd <repo>
gitleaks detect --no-banner -v

Gitleaks cheatsheet

# show secrets I'm about to commit (use --staged in pre-commit hook)
gitleaks protect -v

# show secrets that have been committed in the past
gitleaks detect -v

# show secrets no matter whether they are tracked by git
gitleaks detect --no-git -v

Simple Source Code Management with Git

(Up-to-date source of this post.)

Although I'm more of a sysadmin than a developer I often write scripts (in Perl or Bash). And I tend to use Git for tracking my programs. Every Git repository contains complete history of revisions and is not dependent on a central server or network access. I don't work within a big group of developers, so I try to keep things simple.

First time setup

The following steps are usually needed to be done only once on a machine and are global for all Git repositories:

git config --global user.name "Jeffrey Lebowski"
git config --global user.email "jeffrey.lebowski@dude.com"
git config --global color.ui true
git config alias.lol 'log --pretty=oneline --abbrev-commit --graph --decorate'
git config --global core.editor vim
git config --global merge.tool vimdiff

The I can check the configuration like this:

git config --list

or

cat ~/.gitconfig

Starting a new Git repository

One way to start working with Git is to initialize a directory as a new Git repository:

mkdir project
cd project
git init

Notice a new .git directory. Then I take the snapshot of the contents of all files within current working directory (.):

git add .    # temporary storage - index

This command adds the files to a temporary staging area called index. To permanently store the index:

git commit   # permanent storage

I enter a commit message and I'm done.

Cloning a Git repository

I often need to get an already existing repository (some code me or someone else has already written and stored for example on GitHub):

## Any server with ssh and git
git clone ssh://[user@]server.xy/path/to/repo.git/
## GitHub
git clone git://github.com/ingydotnet/....git

Working with Git

When dealing with Git, it's best to work in small bits. Rule of thumb: if you can't summarize it in a sentence, you've gone too long without committing.

My typical working cycle is:

1. Work on my project.

2. Check whether something has changed:

git status

3. Check what has changed:

git diff

4. Add and commit changes (combines two steps in one git add + git commit):

git commit -am "commit message"

If I not only changed files but also added some new ones I have to add them explicitly:

git add newfile1 newfile2 newfolder3

and then commit as in step 4.

Excluding some files

To set certain files or patterns to be ignored by Git, I create a file called .gitignore in my project’s root directory:

# Don't track dot-files
.*
!/.gitignore

.gitignore is usually checked into Git and distributed along with everything else.