Apr 04

Add a Discussion to a Rally Task/Defect using the python api bindings

OK so we’ve started using rally as our scrum tool in work. Since we use github a lot for tracking issues reported by external users, we wanted to enforce some github > rally integration. As part of my investigations into this, I found myself wanting to add a discussion item to a defect, using the pyral python bindings. This was not at all obvious (to me at least), and the documentation knocking around for the rally api, and the python bindings, also didn’t seem to contain the information needed.

Initially, I thought I could just update the defect with some text passed in to the ‘Discussion’ field, in the same way you would when updating the ‘Notes’ field. For example, to update the Notes for a defect, you would do this:

rally = Rally(server, user, password, project=project)

defectID = 'DE100'
notes = 'some text'

# build the data dict
defect_data = {"FormattedID": defectID, "Notes": notes}

# update the defect
defect = rally.update('Defect', defect_data)

# did it work?
print defect.Notes
some text

 

However, trying to do the same for the discussion does not:

rally = Rally(server, user, password, project=project)

defectID = 'DE100'
discussion = 'some discussion text'

# build the data dict
defect_data = {"FormattedID": defectID, "Discussion": discussion}

# update the defect
defect = rally.update('Defect', defect_data)

# did it work? (no)
print defect.Discussion
[]

 

OK, this *kind* of makes sense. ‘Notes’ is a single field associated with the defect. When we update it, we literally wipe out what was there with our new text.  Fields like ‘Title’ and ‘Description’ behave in the same way.  However, discussions in rally are different – each discussion post has at least a user and some text associated with it, and each one is in effect it’s own item. There’s a hint to this in the output at the end of the last script:

print defect.Discussion
[]

 

We see here that if there were any discussion entries, they would be returned in a list.  And in fact, not just a list of text entries, but a list of objects.  Here’s what the output from a defect object with some discussion entries looks like:

print defect.Discussion
[<pyral.entity.ConversationPost at 0x10968e350>,
<pyral.entity.ConversationPost at 0x10969c050>,
<pyral.entity.ConversationPost at 0x10969c250>,
<pyral.entity.ConversationPost at 0x10969c1d0>,
<pyral.entity.ConversationPost at 0x10969c390>]

 

Interesting…what’s a pyral.entity.ConversationPost object when it’s at home? Well, to cut a long story short, it turns out that a Discussion entry (or ConversationPost as we’ll call it from now on), is actually an artifact in rally, just the same as a defect, or a story, or a task.  In this is the crux of the matter…

To add a discussion to a defect, what we actually need to do is create a new ConversationPost, and pass it some data so it knows which defect to associate itself with.

Turns out this is pretty easy:

rally = Rally(server, user, password, project=project)

defectID = 'DE100'
discussion_text = 'this is a discussion entry'

# get the defect object so we can extract it's oid (unique object number)
defect = rally.get('Defect', query='FormattedID = %s' % defectID, instance=True)

# build the data dict
discussion_data = {"Artifact": defect.oid, "Text": discussion_text}

# create the discussion
discussion = rally.create('ConversationPost', discussion_data)

# did it work?
print discussion.Text
this is a discussion entry
Sep 03

processing the github json payload in jenkins

In an earlier post, I talked about configuring the github webhook to fire on a pull request, rather than just a commit.  As mentioned, there are many events that happen on a github repo, and as per the github documentation, a lot of these can be used to trigger the webhook.

Regardless of what event you decide to trigger on, when the webhook fires from github, it essentially makes a POST to the URL configured in the webhook, including a json payload in the body.  The json payload contains various details about the event that caused the webhook to fire. An example payload that fired on a simple commit can be seen here (check this link for an easy way to examine the content of a webhook payload):

payload
{    "after": "c04a2b2af96a5331bbee0f11fe12965902f5f571",     "before": "78d414a69db29cdd790659924eb9b27baac67f60",     "commits": [        {
            "added": [
                "afile"
            ], 
            "author": {
                "email": "myemailaddress@mydomain.com", 
                "name": "Darren Birkett", 
                "username": "mancdaz"
            }, 
            "committer": {
                "email": "myemailaddress@mydomain.com", 
                "name": "Darren Birkett", 
                "username": "mancdaz"
            }, 
            "distinct": true, 
            "id": "c04a2b2af96a5331bbee0f11fe12965902f5f571", 
            "message": "adding afile", 
            "modified": [], 
            "removed": [], 
            "timestamp": "2012-09-03T02:35:59-07:00", 
            "url": "https://github.com/mancdaz/mygithubrepo/commit/c04a2b2af96a5331bbee0f11fe12965902f5f571"
        }
    ], 
    "compare": "https://github.com/mancdaz/mygithubrepo/compare/78d414a69db2...c04a2b2af96a", 
    "created": false, 
    "deleted": false, 
    "forced": false, 
    "head_commit": {
        "added": [
            "afile"
        ], 
        "author": {
            "email": "myemailaddress@mydomain.com", 
            "name": "Darren Birkett", 
            "username": "mancdaz"
        }, 
        "committer": {
            "email": "myemailaddress@mydomain.com", 
            "name": "Darren Birkett", 
            "username": "mancdaz"
        }, 
        "distinct": true, 
        "id": "c04a2b2af96a5331bbee0f11fe12965902f5f571", 
        "message": "adding afile", 
        "modified": [], 
        "removed": [], 
        "timestamp": "2012-09-03T02:35:59-07:00", 
        "url": "https://github.com/mancdaz/mygithubrepo/commit/c04a2b2af96a5331bbee0f11fe12965902f5f571"
    }, 
    "pusher": {
        "email": "myemailaddress@mydomain.com", 
        "name": "mancdaz"
    }, 
    "ref": "refs/heads/master", 
    "repository": {
        "created_at": "2012-07-12T04:17:51-07:00", 
        "description": "", 
        "fork": false, 
        "forks": 1, 
        "has_downloads": true, 
        "has_issues": true, 
        "has_wiki": true, 
        "name": "mygithubrepo", 
        "open_issues": 0, 
        "owner": {
            "email": "myemailaddress@mydomain.com", 
            "name": "mancdaz"
        }, 
        "private": false, 
        "pushed_at": "2012-09-03T02:36:06-07:00", 
        "size": 124, 
        "stargazers": 1, 
        "url": "https://github.com/mancdaz/mygithubrepo", 
        "watchers": 1
    }
}

This entire payload gets passed in the POST requests as a single parameter, with the imaginitive title ‘payload’.  It contains a ton of information about the event that just happened, all or any of which can be used by jenkins when we build jobs after the trigger.  In order to use this payload in Jenkins, we have a couple of options.  I discuss one below.

Getting the $payload

In jenkins, when creating a new build job, we have the option of specifying the names of parameters that we expect to pass to the job in the POST that triggers the build.  In this case, we would pass a single parameter ‘payload’, as seen here:

passing parameters to a jenkins build job

Further down in the job configuration, we can specify that we would like to be able to trigger the build remotely (ie. that we want to allow github to trigger the build by posting to our URL with the payload):

Then, when we set up the webhook in our github repo (as described in the first post), we give it the URL that jenkins tells us to:

You can’t see it all in the screencap, but the URL I specified for the webhook was the one that jenkins told me to:

http://jenkins-server.chloky.com:8080/job/mytestbuild//buildWithParameters?token=asecuretoken

Now, when I built my new job in jenkins, for the purposes of this test I simply told it to echo out the contents of the ‘payload’ parameter (which is available in paramterized builds as a shell variable of the same name), using a simple script:

#!/bin/bash

echo "the build worked! The payload is $payload"

Now to test the whole thing we simply have to make a commit to our repo, and then pop over to jenkins to look at the job that was triggered:

mancdaz@chloky$ (git::master)$ touch myfile

mancdaz@chloky$ (git::master) git add myfile

mancdaz@chloky$ (git::master) git commit -m 'added my file'
[master 4810490] added my file
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 myfile

mancdaz@chloky$ (git::master) git push
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 232 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
To git@github.com:mancdaz/mygithubrepo.git
 c7ecafa..4810490 master -> master

And over in our jenkins server, we can look at the console output of the job that was triggered, and lo and behold there is our ‘payload’ contained in the $payload variable and available for us to consume:

So great, all the info about our github event is here. and fully available in our jenkins job!  True enough, it’s in a big json blob, but with a bit of crafty bash you should be good to go.

Of course, this example used a simple commit to demonstrate the principles of getting at the payload inside jenkins.  As we discussed in the earlier post, a commit is one of many events on a repo that can trigger a webhook.  What you do inside jenkins once you’ve triggered is up to you, but the real fun comes when you start interacting with github to take further actions on the repo (post comments, merge pull requests, reject commits etc) based on the results of your build jobs that got triggered by the initial event.

Look out for a subsequent post where I tie it all together and show you how to process, run tests for, and finally merge a pull request if successful – all automatically inside jenkins.  Automation is fun!

Aug 30

reinstalling an OpenStack nova compute node with Alamo

Recently, Rackspace[1] released an OpenStack compute installer called ‘Alamo‘.  This is effectively a downloadable ISO that will, after asking the user a few questions, install a fully functional OpenStack compute cluster, with integrated nova, glance, keystone and horizon components.

The installer is really easy to use.  When you pop the Alamo ISO into your machine and boot up, one of the main questions you are asked is what type of node you want to install – controller (everything but nova-compute), or compute (only nova-compute).  You need to install a controller node first, before you go ahead and install compute nodes to connect to the cluster.

Depending on what you’re doing, it’s quite likely that you’ll want to reinstall one of your compute nodes at some point, and without going through some additional steps you’re going to have issues.  Issues like this:

openstack installation errors with Alamo

 

So what’s happening here?  Well, it’s important to understand what the installer actually does when we’re installing a compute node before we can understand what we need to do to fix it.  So, I hear you cry, ‘WHAT DOES THE INSTALLER DO??’.  Well, this:

  • boots using the ubuntu/debian installer
  • asks you a few questions (most importantly – what is your ip address, hostname, and the ip of the previously installed controller node)
  • installs ubuntu precise (12.04)
  • registers the machine with the chef server (that is running inside a vm on the controller node)
  • assigns itself the nova-compute chef role
  • runs chef-client, which then runs all the chef recipes associated with the nova compute role and thereby configures the node as a compute node in this cluster

See how one of those steps is highlighted?  Yep, that’s our problem right there.  When we re-install a compute node with this method (giving it exactly the same hostname as before), it tries to re-register itself as a client with the chef server.  But the chef server won’t have any of it; as far as it’s concerned, it already has a client with that name and you’re just an imposter.  You can see this by logging on to the controller node and running the following:

root@mycontroller:~# knife node list
  compute1.example.com
  compute2.example.com
  mycontroller.example.com

See!  I’m trying to reinstall compute1.example.com, but it’s already there!

Luckily enough, the fix is easy. Shut down the compute node that you are trying to install (you need to do this as it will continually attempt to re-register itself every 30 seconds otherwise). And then, on the controller node:

root@mycontroller:~# knife node delete -y compute1.example.com && knife client delete -y compute1.example.com
Deleted node[compute1.example.com]
Deleted client[compute1.example.com]

That’s it! Now restart the compute node and re-run the installer, and you will enjoy many minutes of success!

[1] full disclosure – I work on the team at Rackspace that produced the Alamo installer

Jul 12

using github webhooks to trigger jenkins jobs on a pull request

github provides a nice way to fire off notifications to a CI system like jenkins whenever a commit is made against a repository.  This is really useful for kicking off build jobs in jenkins to test the commits that were just made on the repo.  You simply need to go to the administration section of the repository, click on service hooks on the left, click ‘webhook URLs’ at the top of the list, and then enter the URL of the webhook that jenkins is expecting (look at this jenkins plugin for setting up jenkins to recieve these hooks from github).

Recently though, I was looking for a way to make a webhook fire when a pull request is made against a repo, rather than when a commit is made to the repo.  This is so that we could have jenkins run a bunch of tests on the pull request, before deciding whether to merge the pull request in – useful for when you have a lot of developers working on their own forks and regularly submitting pull requests to the main repo.

It turns out that this is not as obvious as one would hope, and requires a bit of messing about with the github API.

By default, when you configure a github webhook, it is configured to only fire when a commit is made against a repo.  There is no easy way to see, or change, this in the github web interface when you set up the webhook.  In order to manipulate the webhook in any way, you need to use the API.

To make changes on a repo via the github API, we need to authorize ourselves.  We’re going to use curl, so if we wanted to we could pass our username and password each time, like this:

# curl https://api.github.com/users/mancdaz --user 'mancdaz'
Enter host password for user 'mancdaz':

Or, and this is a much better option if you want to script any of this stuff, we can grab an oauth token and use it in subsequent requests to save having to keep entering our password.  This is what we’re going to do in our example.  First we need to create an oauth authorization and grab the token:

curl https://api.github.com/authorizations --user "mancdaz" \
--data '{"scopes":["repo"]}' -X POST

You will be returned something like the following:

{
"app": {
"name": "GitHub API",
"url": "http://developer.github.com/v3/oauth/#oauth-authorizations-api"
},
"token": "b2067d190ab94698a592878075d59bb13e4f5e96",
"scopes": [
"repo"
],
"created_at": "2012-07-12T12:55:26Z",
"updated_at": "2012-07-12T12:55:26Z",
"note_url": null,
"note": null,
"id": 498182,
"url": "https://api.github.com/authorizations/498182"
}

Now we can use this token in subsequent requests for manipulating our github account via the API.  So  let’s query our repo and find the webhook we set up in the web interface earlier:

# curl  https://api.github.com/repos/mancdaz/mygithubrepo/hooks?access_token=b2067d190ab94698592878075d59bb13e4f5e96
[
  {
    "created_at": "2012-07-12T11:18:16Z",
    "updated_at": "2012-07-12T11:18:16Z",
    "events": [
      "push"
    ],
    "last_response": {
      "status": "unused",
      "message": null,
      "code": null
    },
    "name": "web",
    "config": {
      "insecure_ssl": "1",
      "content_type": "form",
      "url": "http://jenkins-server.chloky.com/post-hook"
    },
    "id": 341673,
    "active": true,
    "url": "https://api.github.com/repos/mancdaz/mygithubrepo/hooks/341673"
  }
]

Note the important bit from that json output:

"events": [
      "push"
    ]

This basically says that this webhook will only trigger when a commit (push) is made to the repo.  The github API documentation describes numerous different event types that can be added to this list – for our purposes we want to add pull_request, and this is how we do it (note that we get the id of the webhook from the json output above. If you have multiple hooks defined, your output will contain all these hooks so be sure to get the right ID):

# curl  https://api.github.com/repos/mancdaz/mygithubrepo/hooks/341673?access_token=b2067d190ab94698592878075d59bb13e4f5e96 -X PATCH --data '{"events": ["push", "pull_request"]}'
{
  "created_at": "2012-07-12T11:18:16Z",
  "updated_at": "2012-07-12T16:03:21Z",
  "last_response": {
    "status": "unused",
    "message": null,
    "code": null
  },
  "events": [
    "push",
    "pull_request"
  ],
  "name": "web",
  "config": {
    "insecure_ssl": "1",
    "content_type": "form",
    "url": "http://jenkins-server.chloky.com/post-hook"
  },
  "id": 341673,
  "active": true,
  "url": "https://api.github.com/repos/mancdaz/mygithubrepo/hooks/341673"
}

See!

"events": [
    "push",
    "pull_request"
  ],

This webhook will now trigger whenever either a commit OR a pull request is made against our repo. Exactly what you do in your jenkins/with this webhook is up to you. We use it to kick off a bunch of integration tests in jenkins to test the proposed patch, and then actually merge and close (again using the API) the pull request automatically. Pretty sweet.