Circle CI + Github API to auto-update gems

Hi! This is Rob! In this blog post, I’ll show you in detail how to create daily/nightly builds in CircleCI with AWS Lambda functioning as a cron server, and take the results of those builds and create a Github pull request with the changes. Everything in this tutorial is 100% free, and MIT licensed.

For this project, I am assuming that you already have your project on CircleCI and Github. If you don’t use Github, the Lambda and CircleCI parts will still be useful to you, just ignore the part at the very end where we create pull requests. If you don’t have CircleCI, this project unfortunately has almost nothing for you unless you want to see an example of what a Github API request looks like.

CAVEAT: In the bash script, we use the line git add Gemfile.lock to add our newest files. I would recommend writing git add <whatever your files are expected to be> instead of some catch all expression like git add .. For example, if you added the file “apples.txt” I would recommend adding that file explicitly as git add apples.txt. This is because CircleCI can mess with your repo in crazy ways when it sets up for things. Example: Before it gets to your code, it may run something like bundle install to get all of your project dependencies. This means that you will suddenly have tons of files related to gems in vendor/bundle, and if you didn’t gitignore that file, you are looking at a huge diff. It can also edit database.yml, so there’s that.

Along the way, I’ll be explaining gotchas or odd lines of code. Let’s begin.

The way that nightly builds work is that first, we’ll make a bash script to run all the code we want to execute on the build machine. This could be anything from rails t to bundle update to a bash script that checks the weather around your Singapore server (if you write that, please show me, I would be super interested). The end of that script will have us push any changes we made to Github as a pull request.

Then, we’ll put that script in our CircleCI File to make it activate when we trigger our builds via API. This is relatively simple, a few lines.

Finally, we’ll build a lambda and slap a cron job on there to make it proc our build at whatever time and frequency you please. Note: Lambda cron is limited to a rate of no greater than 1/minute (source).

First, the bash script. Create a file with a name that describes what the code will do. I wrote a script called bundle_update.sh, because we’re gonna be updating bundles. Don’t forget to chmod +x ! If you need to test this locally.

bundle_update.sh

“`#!/bin/bash

assuming master is our base branch for simplicity

git checkout ‘master’

NOTE: Force our branch to be in alignment with remote, just in case

git reset –hard origin/master

See if our branch exists, if it does, switch to it. If not, create it.

We do this because if we don’t merge the pull request before this file runs a second time, then it will try to push the same changes to github and throw an error. This was, we’re sure to have the latest changes

Check branch exists. This will throw a 0 when checked with the $? command if the branch doesn’t exist. Explained below

git rev-parse –verify circle_ci/update_gemfile_lock
if ! [ 0 == $? ]; then
git checkout -b ‘circle_ci/update_gemfile_lock’
else

checkout the branch if it exists, update it to its remote status

git checkout ‘circle_ci/update_gemfile_lock’
git reset –hard origin/circle_ci/update_gemfile_lock
fi

Update those gems

bundle update

If git status –porcelain returns anything at all, that means updates happened

if [[ git status --porcelain ]]; then
git config –global user.email “circleciemail@email.com”
git config –global user.name “circle-ci”
# Double check that our gem branch we switched to exists!
if [ git rev-parse --abbrev-ref HEAD == ‘circle_ci/update_gemfile_lock’ ]; then
git add ‘Gemfile.lock’
git commit -m ‘Update Gemfile.lock’
git push –set-upstream origin circle_ci/update_gemfile_lock
# See if our pull request already exists. If so, no need to update our pull request
RES=$(curl -X GET -G -d ‘head=:circle_ci/update_gemfile_lock’ https://api.github.com/repos///pulls?access_token=$GITHUB_ACCESS_TOKEN)

# If the response contains the string "html_url", it already exists
if [[ $RES == *"html_url"* ]]; then
  echo 'Branch already exists and was updated via push. No problem'
# The branch-not-exist response is something like "[\n]"
else
  curl -H "Content-Type: application/json" -X POST -d '{"title":"Updated Gemfile Lock","body":"Gem versions have changed", "head": "circle_ci/update_gemfile_lock", "base":"master"}' https://api.github.com/repos/<organization i.e. Rob117>/<project i.e. NightlyBuilds>/pulls?access_token=$GITHUB_ACCESS_TOKEN
fi

fi
fi


Before I explain the gotchas of the above code, go onto Github and create a [token](https://github.com/settings/tokens). That token should have the repo checkbox ticked so it has `Full control of private repositories`. **Save that token**. Switch over to CircleCI, and on your project settings page under Environment Variables, create a variable with the name GITHUB_ACCESS_TOKEN and a value of whatever your token string was. Also ensure that your project has ssh permissions for Github set as well, or we can't push code from within our scripts. You can check by going to Checkout SSH Keys in the project settings and ensuring that you have a key that is permitted to ssh to your git repo there. If not (i.e., you don't have a user key there), click add user key and authorize the application to use your key. After authorizing, click on the 'create and add user key' button to get your ssh key on the machines so they can push code. If you want to test the code on your local machine, go ahead and `export GITHUB_ACCESS_TOKEN=<string here>`. Okay, now that we can actually run this code, let's break it down. Most of the comments explain the code clearly. Remember to replace your email in the git config section, and your Github API URL organziation name and project name in the last line. The gotchas are: We have to reset our base branch to whatever it is on remote with `git reset --hard origin/master`. The reason for this is because as a byproduct of using CircleCI, a lot of things happen to our build machine repository before we execute a single line of code, and they could taint the final results. Syncing master with its remote counterpart ensures that we have a pristine copy, so it's just good practice anyway. These two lines:

git rev-parse –verify circle_ci/update_gemfile_lock
if ! [ 0 == $? ]; then


The first line checks to see if the git branch exists. If it does, it will show 0 when you run the command $?, else it will show something else. This tells us if the branch exists so we know whether to make one or not.

if [[ git status --porcelain ]];

Simply runs git status and outputs nothing if there is no change. If this outputs text, we want to make a pull request.

if [ git rev-parse --abbrev-ref HEAD == ‘circle_ci/update_gemfile_lock’ ]; then

Checks if our branch was actually created correctly. We do this so that we are sure we aren't going to force push to the wrong branch.

curl -H “Content-Type: application/json” -X POST -d ‘{“title”:”Title Of Pull Request Here”,”body”:”Body of pull request here”, “head”: “circle_ci/make_apples”, “base”:”master”}’
https://api.github.com/repos///pulls?access_token=$GITHUB_ACCESS_TOKEN
#Example
#https://api.github.com/repos/Rob117/circletest/pulls?access_token=$GITHUB_ACCESS_TOKEN


This is the magic. Base is the branch we started with in the beginning, the one that we want to reflect our code changes post-merge. Head is the branch we created in this script to do all of our work. We take the changes and make a pull request using our github access token that is stored in our environment variable. Okay! We've broken down the bash script. Remember- just change the name of the base branch, created branch, and the file creation part to use the script in your own project. Running this script on your local environment - assuming you set your environment GITHUB_ACCESS_TOKEN var - should work smoothly and create a pull request on github. Next, let's edit our circle CI. It's just a few lines of code added to your normal circle.yml:

Other Circle.yml code

test:
post:
– >
if [ -n “${RUN_NIGHTLY_BUILD}” ]; then
# If you put the script in your scripts folder, the path is ./scripts/
./bundle_update.sh
# As many scripts as you want here
fi

More circle CI code


This code says 'if we get a post request that has variable called RUN_NIGHTLY_BUILD, execute the following scripts'. That's it! Push this code to the branch that you want to call it from. For example, maybe you want to keep all of your scripts in your builds branch. Simply merge the above script and yml file into a branch called builds, and push that branch. You're all set to trigger your builds! You could also use a staging branch, develop branch, etc. SUPER IMPORTANT NOTE: If you have more than one script set to run and you only store the scripts in a single branch - say, `builds` - and you change branches within a script, you MUST run `git checkout builds` between script runs / at the end of each script. If you don't, Circle CI may fail to run all your scripts because you'll have changed to a branch where they don't exist! Okay, next let's set up our lambda and be done with this. First, hop on over to Circle CI, go to project settings, then go to API Permissions, then click Create Token. The scope of the token should be `All`, and the label can be whatever you want. I chose `Lambda Execution`. Save that token, and open up your AWS Control Panel. Switch over to Lambda. Click Create Lambda Function, then click Blank Function. Next, click the empty dotted box to the left of the word Lambda and click Cloudwatch Events as the trigger. For the rule, create new rule. Rule name is `Nightly-Build` and the schedule expression is `cron(0 0 * * ? *)` -run at 12 UTC every day. [(Source)](http://docs.aws.amazon.com/lambda/latest/dg/tutorial-scheduled-events-schedule-expressions.html) Check Enable Trigger, then move to the next screen. Name the function 'NightlyBuild'. Skip the code for now. Set the environment variable CircleCIToken to <token you copied from CircleCI site>. Set the role to a basic lambda execution role. Under advanced settings, set the timeout to 15 seconds. Finally, the lambda code:

// Based on: https://github.com/godfreyhobbs/circleci-cron-serverless/blob/master/handler.js
‘use strict’;
var http = require(“https”);

exports.handler = (event, context, callback) => {
var options = {
“method”: “POST”,
“hostname”: “circleci.com”,
“port”: null,
// example: “/api/v1.1/project/github/RobUserName117/secret-project/tree/builds”
“path”: “/api/v1.1/project////tree/” +
“?circle-token=” + process.env.CircleCIToken,
“headers”: {
“content-type”: “application/json”,
“cache-control”: “no-cache”
}
};

// Make request, write out all of response
var req = http.request(options, function (res) {
    var chunks = [];

    res.on("data", function (chunk) {
        chunks.push(chunk);
    });

    res.on("end", function () {
        var body = Buffer.concat(chunks);
        const response = body.toString();
        console.log(response);
        callback(null, response);
    });
});

req.write(JSON.stringify({
  "build_parameters": {
    "RUN_NIGHTLY_BUILD": "true"
  }
}));
req.end();

};

“`

Most of this code is self explanatory. The line that I would like to direct your attention to is the “path” part of your options object. According to the Circle CI docs, the URL that you post to is /api/v1.1/project/vcs system/organization or user name/project name/tree/branch

So if you wanted to push to the staging branch of the blogly project of the Google organization, stored in github, you would use:
/api/v1.1/project/github/google/blogly/tree/staging

The second thing is that we are calling our builds by sending a JSON with the RUN_NIGHTLY_BUILD parameter set to true. You could set or send as many variables as you please – see the docs for more details.

When this code runs successfully (click ‘TEST’), you should get a response that says something like “Last build was blah blah blah”. If you see a response that looks like CircleCI Webpage HTML, you made a configuration error somewhere.

Switch over to CircleCI and watch your build happen, then switch to Github and witness your glorious pull request!

Finished! Thanks for reading to the end.

Serverless Backends with AWS Cloud: Sending tweets and collecting emails.

The final disclaimer

I know I’ve said it about a dozen times before, but here it is for the last time: This tutorial was built to teach you cool AWS services and give you some practical experience with them. There are many, many different ways that you could have done this, and not all of them are equally suited for all purposes. Still, this very closely resembles actual production code that I have used in a company (with their explicit permission to write this guide, of course). It works, and it’s battle-tested.

Continue reading

Serverless Backends with AWS Cloud: Cloudfront and WAF

CloudFronts and Firewalls

Why Cloudfront for an API?

I’m gonna be honest: When my boss told me to put the API on CloudFront, I thought he might have been joking. Why would you need to put an API on a distribution network? Setting it up was the hardest part of this project (no, seriously), and working with the quirks of CloudFront was a pain.

Then it became super obvious why we did it once it started working.

Continue reading

Serverless Backends With AWS Cloud: Twitter Sessions

How to make a session in DynamoDB

In this section, we’re just going to save some browsing information in a session (held in DynamoDB) and redirect the user to Twitter. Once we have a callback in place, we’ll use that session data to have our user follow us. We can also use that data to message that user later (we actually have the user DM themselves, explained in the next section) with some promotional news.

Continue reading

Serverless Backends With AWS Cloud: Email Lambda and DynamoDB

Email, Lambda, Mailgun, Dynamo, S3

There is going to be a lot coming at you all at once in this post. This is a large project with a lot of moving parts; while I actually built this incrementally, we’ll just be building pretty much everything final-stage here (with a few exceptions). Take your time and read everything. We’ll be touching S3, Lambda, IAM, KMS, and DynamoDB services in this post.

Continue reading