in Databases, Testing

Jenkins to test only files changed in Git commit

In our development stack, we use Git and Jenkins to automate testing of our pull requests and commits. From a development perspective we are typically checking for things like “did the build pass” that encompass the entire working tree and it works great for that purpose.

The database team has recently been working to integrate automated testing into our workflow as well. One of the things that drove this is that some of the editors our developers work with insert a Byte Order Mark (BOM) character at the beginning of the file and it prevents our tools from deploying the migration file correctly. We worked with one of our testers to get Jenkins setup to reference a bash script that could facilitate our automated tests.

There is no need to waste CPU cycles processing every file for this BOM character, so we only wanted to check for new files. The command to check for new files that worked locally for me was git diff --name-only origin/master while I was on my feature branch. When I tried to run this in the bash script through Jenkins I got the error fatal: ambiguous argument 'refs/heads/master': unknown revision or path not in the working tree.

I struggled for a while trying a number of things to get past this issue before a team member was able to explain to me what was happening. When Jenkins does its checkout it is explicitly getting the pull request and not the refs for master so there was nothing to compare to. By performing a git fetch prior to running the above command that compares the current commit chain to master, it was able to calculate the differences.

In developing the script to check for BOM characters I spent quite a bit of time searching the internet for a bash script that would allow me to iterate over just the files in my PR or commit that are different than master. I was surprised that I had difficulty finding something, so I wanted to share the script that is now working for us. This iterates over the files and passes them to the bom_sniffer function. Please note, the below script will not work with spaces in filenames or folders. Please adjust accordingly to fit your needs.


# disabling set -e and relying on explicitly defined exit codes due to expected
# failures as negative tests
#set -e

bom_sniffer() { 
  # exit code is 1 if character not found
  head -c3 "$1" | LC_ALL=C grep -qP '\xef\xbb\xbf'; 
  if [ $? -eq 0 ] 
    exit 1
check_rc() {
  # exit if passed in value is not = 0
  # $1 = return code
  # $2 = command / label
  if [ $1 -ne 0 ]
    echo "$2 command failed"
    exit 1

# finding files that differ from this commit and master
echo 'git fetch'
check_rc $? 'echo git fetch'
git fetch
check_rc $? 'git fetch'
echo 'git diff --name-only origin/master'
check_rc $? 'echo git diff'
diff_files=`git diff --name-only origin/master | xargs`
check_rc $? 'git diff'
for x in ${diff_files}
  #echo "${x}"
  bom_sniffer "${x}"
  check_rc $? "BOM character detected in ${x},"

Write a Comment