processing the github json payload in jenkins

In an earlier post, I talked about configuring the github webhook to fire on a pull request, rather than just a commit.  As mentioned, there are many events that happen on a github repo, and as per the github documentation, a lot of these can be used to trigger the webhook.

Regardless of what event you decide to trigger on, when the webhook fires from github, it essentially makes a POST to the URL configured in the webhook, including a json payload in the body.  The json payload contains various details about the event that caused the webhook to fire. An example payload that fired on a simple commit can be seen here (check this link for an easy way to examine the content of a webhook payload):

payload
{    "after": "c04a2b2af96a5331bbee0f11fe12965902f5f571",     "before": "78d414a69db29cdd790659924eb9b27baac67f60",     "commits": [        {
            "added": [
                "afile"
            ], 
            "author": {
                "email": "myemailaddress@mydomain.com", 
                "name": "Darren Birkett", 
                "username": "mancdaz"
            }, 
            "committer": {
                "email": "myemailaddress@mydomain.com", 
                "name": "Darren Birkett", 
                "username": "mancdaz"
            }, 
            "distinct": true, 
            "id": "c04a2b2af96a5331bbee0f11fe12965902f5f571", 
            "message": "adding afile", 
            "modified": [], 
            "removed": [], 
            "timestamp": "2012-09-03T02:35:59-07:00", 
            "url": "https://github.com/mancdaz/mygithubrepo/commit/c04a2b2af96a5331bbee0f11fe12965902f5f571"
        }
    ], 
    "compare": "https://github.com/mancdaz/mygithubrepo/compare/78d414a69db2...c04a2b2af96a", 
    "created": false, 
    "deleted": false, 
    "forced": false, 
    "head_commit": {
        "added": [
            "afile"
        ], 
        "author": {
            "email": "myemailaddress@mydomain.com", 
            "name": "Darren Birkett", 
            "username": "mancdaz"
        }, 
        "committer": {
            "email": "myemailaddress@mydomain.com", 
            "name": "Darren Birkett", 
            "username": "mancdaz"
        }, 
        "distinct": true, 
        "id": "c04a2b2af96a5331bbee0f11fe12965902f5f571", 
        "message": "adding afile", 
        "modified": [], 
        "removed": [], 
        "timestamp": "2012-09-03T02:35:59-07:00", 
        "url": "https://github.com/mancdaz/mygithubrepo/commit/c04a2b2af96a5331bbee0f11fe12965902f5f571"
    }, 
    "pusher": {
        "email": "myemailaddress@mydomain.com", 
        "name": "mancdaz"
    }, 
    "ref": "refs/heads/master", 
    "repository": {
        "created_at": "2012-07-12T04:17:51-07:00", 
        "description": "", 
        "fork": false, 
        "forks": 1, 
        "has_downloads": true, 
        "has_issues": true, 
        "has_wiki": true, 
        "name": "mygithubrepo", 
        "open_issues": 0, 
        "owner": {
            "email": "myemailaddress@mydomain.com", 
            "name": "mancdaz"
        }, 
        "private": false, 
        "pushed_at": "2012-09-03T02:36:06-07:00", 
        "size": 124, 
        "stargazers": 1, 
        "url": "https://github.com/mancdaz/mygithubrepo", 
        "watchers": 1
    }
}

This entire payload gets passed in the POST requests as a single parameter, with the imaginitive title ‘payload’.  It contains a ton of information about the event that just happened, all or any of which can be used by jenkins when we build jobs after the trigger.  In order to use this payload in Jenkins, we have a couple of options.  I discuss one below.

Getting the $payload

In jenkins, when creating a new build job, we have the option of specifying the names of parameters that we expect to pass to the job in the POST that triggers the build.  In this case, we would pass a single parameter ‘payload’, as seen here:

passing parameters to a jenkins build job

Further down in the job configuration, we can specify that we would like to be able to trigger the build remotely (ie. that we want to allow github to trigger the build by posting to our URL with the payload):

Then, when we set up the webhook in our github repo (as described in the first post), we give it the URL that jenkins tells us to:

You can’t see it all in the screencap, but the URL I specified for the webhook was the one that jenkins told me to:

http://jenkins-server.chloky.com:8080/job/mytestbuild//buildWithParameters?token=asecuretoken

Now, when I built my new job in jenkins, for the purposes of this test I simply told it to echo out the contents of the ‘payload’ parameter (which is available in paramterized builds as a shell variable of the same name), using a simple script:

#!/bin/bash

echo "the build worked! The payload is $payload"

Now to test the whole thing we simply have to make a commit to our repo, and then pop over to jenkins to look at the job that was triggered:

mancdaz@chloky$ (git::master)$ touch myfile

mancdaz@chloky$ (git::master) git add myfile

mancdaz@chloky$ (git::master) git commit -m 'added my file'
[master 4810490] added my file
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 myfile

mancdaz@chloky$ (git::master) git push
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 232 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
To git@github.com:mancdaz/mygithubrepo.git
 c7ecafa..4810490 master -> master

And over in our jenkins server, we can look at the console output of the job that was triggered, and lo and behold there is our ‘payload’ contained in the $payload variable and available for us to consume:

So great, all the info about our github event is here. and fully available in our jenkins job!  True enough, it’s in a big json blob, but with a bit of crafty bash you should be good to go.

Of course, this example used a simple commit to demonstrate the principles of getting at the payload inside jenkins.  As we discussed in the earlier post, a commit is one of many events on a repo that can trigger a webhook.  What you do inside jenkins once you’ve triggered is up to you, but the real fun comes when you start interacting with github to take further actions on the repo (post comments, merge pull requests, reject commits etc) based on the results of your build jobs that got triggered by the initial event.

Look out for a subsequent post where I tie it all together and show you how to process, run tests for, and finally merge a pull request if successful – all automatically inside jenkins.  Automation is fun!

5 thoughts on “processing the github json payload in jenkins

  1. Hey there! Awesome Post. Can you post the remainder of the left over part. I am also trying to implement something similar.

    Great Work Again!.

    Thanks.

  2. Pingback: Dynamically Building Package.xml - Kevin Bromer

  3. This is the only post I found that details how to build a job that displays the json payload. However, my job doesn’t display the payload. It does get triggered however, upon a push. The only thing different between your and my setup, is that, I notice that you have specified the git repo path under “Github project” on your Jenkins Job. My Jenkins job page doesn’t even have a “Github project” field. Just wondering, if that missing link is responsible for my job not displaying the payload. Can you please specify the plug-in for the “Github project”?

    Thanks for taking the time to put up this post. It did provide me some good information.

  4. I’ve tried this scenario but my build does not include a $payload variable. I am suing the same plugin and my Github hook have the same format. Does this still work?

  5. Hi,

    Thanks for taking the time to write this. I’m actually trying to setup the same thing myself but found that GitHub sends the JSON payload as a top level object in the POST body, whereas in your example GitHub rather conveniently attached it to a top-level ‘payload’ variable which you were then able to reference in Jenkins. Did you have to do anything to configure GitHub to do this? I’m at a bit of a loss regarding how I can get hold of the payload in Jenkins without being able to do this.

    Many thanks,

    Edd

Leave a Reply to Manpreet Singh Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>