Category Archives: Uncategorized

Starting with Git

Making and Continuing Projects

Let’s learn about git. It’s a very useful tool that is really in-depth and has a bunch of awesome features, like how to bug hunt with git. If you understand what is going on in that post, this post may be too basic for you.

We assume:

  • You can use very basic terminal commands (mkdir, cd)
  • You have at least a basic idea that git is some kind of work-saving system, and you want to use it in projects but don’t even know where to start

We are going to do three things:

  1. Start an empty project on Github and clone the repo to our computer
  2. Take an existing project and push it to github
  3. Clone an existing repository on Github and put it on our computer.

We are not going to:

  • Have an in-depth explanation of what git is
  • Talk about how it works on a complex level.

For all projects not yet created / on Github

So, you always want to create the project on Github first (if you already have files on your computer, don’t worry, we deal with files next). Make an account on Github if you don’t have one, then click the green button “New Repository.”

On the next page, give your project some kind of cool name, like “gitexample”. Give it whatever description you want, and leave the README box unchecked. Reference shot:

Click the “Create Repository” button at the bottom of the screen.

Normally, we would use ssh because in the real world it is much more convenient, but ssh is a whole different thing so we’ll be using https and typing our password each time we want to move files back and forth between Github and our computer.

On the next screen, click the HTTPS button under “Quick setup — if you’ve done this kind of thing before”. You should see this screen (Note the https: at the beginning of the github link string):

How to create the project itself / move out project to github

So, now we have a place to actually put the files on Github (in the screenshot, that would be the string

If you have no files on your computer (Fresh start):

First go to the folder where you want to actually have the project reside on your computer. For example, if you want the project folder to be Documents/gitexample, make sure you cd Documents/gitexample before starting the following.

The format is:

Comment about following commands
lines that look like this are terminal commands

This simply creates a file in the folder we are currently in. The file name is, and the contents just read # gitexample
echo "# gitexample" >>

Tell the git system we’ll be tracking changes here
git init

Tell git that we want to add the file that we just created so we can save it to our system
git add

Save all the files we just added (currently just the one) – note, the “first commit” can be anything you like
git commit -m "first commit"

Tell git “The link after origin is where we want to push and pull files from.”
PLEASE REPLACE with your own link from earlier
git remote add origin <>

push the file we just made to the remote repository we created!
git push -u origin master

It will prompt you for your password. That’s it! When you go check the repository page on Github, you should see your file!

If you have files on your computer

Go to the folder with the project files in your terminal cd Documents/MyFolderName
lines that look like this are terminal commands

Tell the git system we’ll be tracking changes here
git init

Tell git that we want to add all the files in this folder to our git repository for tracking.
git add .

Save all the files we just added – note, the “first commit” can be anything you like
git commit -m "first commit"

Tell git “The link after origin is where we want to push and pull files from.”
PLEASE REPLACE with your own link from earlier
git remote add origin <>

push the file we just made to the remote repository we created!
git push -u origin master

It will prompt you for your password. That’s it! When you go check the repository page on Github, you should see your files from your project.

Pulling from an existing project

This one is actually super easy.
Go to where you want the folder to reside on your computer. For example, if you want the file path to be Documents/Project, cd Documents

Then open the project GitHub Page (example) and notice the green “Clone or download” button. Click that, then change it to https (or ssh, depending on what your boss tells you). It will give you a string like

Then just simply type in terminal git clone <>, using your own string.

The project folder will be automatically created, and all the files transferred into it!

And that’s the basics of git transferring files. If anything I’ve written is unclear, please let me know so I can refine the process.

Circle CI + Github API to auto-update gems

Hi! This is Rob! In this blog post, I’ll show you in detail how to create daily/nightly builds in CircleCI with AWS Lambda functioning as a cron server, and take the results of those builds and create a Github pull request with the changes. Everything in this tutorial is 100% free, and MIT licensed.

For this project, I am assuming that you already have your project on CircleCI and Github. If you don’t use Github, the Lambda and CircleCI parts will still be useful to you, just ignore the part at the very end where we create pull requests. If you don’t have CircleCI, this project unfortunately has almost nothing for you unless you want to see an example of what a Github API request looks like.

CAVEAT: In the bash script, we use the line git add Gemfile.lock to add our newest files. I would recommend writing git add <whatever your files are expected to be> instead of some catch all expression like git add .. For example, if you added the file “apples.txt” I would recommend adding that file explicitly as git add apples.txt. This is because CircleCI can mess with your repo in crazy ways when it sets up for things. Example: Before it gets to your code, it may run something like bundle install to get all of your project dependencies. This means that you will suddenly have tons of files related to gems in vendor/bundle, and if you didn’t gitignore that file, you are looking at a huge diff. It can also edit database.yml, so there’s that.

Along the way, I’ll be explaining gotchas or odd lines of code. Let’s begin.

The way that nightly builds work is that first, we’ll make a bash script to run all the code we want to execute on the build machine. This could be anything from rails t to bundle update to a bash script that checks the weather around your Singapore server (if you write that, please show me, I would be super interested). The end of that script will have us push any changes we made to Github as a pull request.

Then, we’ll put that script in our CircleCI File to make it activate when we trigger our builds via API. This is relatively simple, a few lines.

Finally, we’ll build a lambda and slap a cron job on there to make it proc our build at whatever time and frequency you please. Note: Lambda cron is limited to a rate of no greater than 1/minute (source).

First, the bash script. Create a file with a name that describes what the code will do. I wrote a script called, because we’re gonna be updating bundles. Don’t forget to chmod +x ! If you need to test this locally.


assuming master is our base branch for simplicity
git checkout 'master'
# NOTE: Force our branch to be in alignment with remote, just in case
git reset --hard origin/master
# See if our branch exists, if it does, switch to it. If not, create it.
# We do this because if we don't merge the pull request before this file runs a second time, then it will try to push the same changes to github and throw an error. This was, we're sure to have the latest changes

# Check branch exists. This will throw a 0 when checked with the $? command if the branch doesn't exist. Explained below
git rev-parse --verify circle_ci/update_gemfile_lock
if ! [ 0 == $? ]; then
  git checkout -b 'circle_ci/update_gemfile_lock'
# checkout the branch if it exists, update it to its remote status
  git checkout 'circle_ci/update_gemfile_lock'
  git reset --hard origin/circle_ci/update_gemfile_lock
# Update those gems
bundle update

# If git status --porcelain returns anything at all, that means updates happened
if [[ `git status --porcelain` ]]; then
  git config --global ""
  git config --global "circle-ci"
  # Double check that our gem branch we switched to exists!
  if [ `git rev-parse --abbrev-ref HEAD` == 'circle_ci/update_gemfile_lock' ]; then
    git add 'Gemfile.lock'
    git commit -m 'Update Gemfile.lock'
    git push --set-upstream origin circle_ci/update_gemfile_lock
    # See if our pull request already exists. If so, no need to update our pull request
    RES=$(curl -X GET -G -d 'head=<organization (i.e. Rob117)>:circle_ci/update_gemfile_lock'<organization i.e. Rob117>/<project i.e. NightlyBuilds>/pulls?access_token=$GITHUB_ACCESS_TOKEN)

    # If the response contains the string "html_url", it already exists
    if [[ $RES == *"html_url"* ]]; then
      echo 'Branch already exists and was updated via push. No problem'
    # The branch-not-exist response is something like "[\n]"
      curl -H "Content-Type: application/json" -X POST -d '{"title":"Updated Gemfile Lock","body":"Gem versions have changed", "head": "circle_ci/update_gemfile_lock", "base":"master"}'<organization i.e. Rob117>/<project i.e. NightlyBuilds>/pulls?access_token=$GITHUB_ACCESS_TOKEN

Before I explain the gotchas of the above code, go onto Github and create a token. That token should have the repo checkbox ticked so it has Full control of private repositories. Save that token. Switch over to CircleCI, and on your project settings page under Environment Variables, create a variable with the name GITHUB_ACCESS_TOKEN and a value of whatever your token string was.

Also ensure that your project has ssh permissions for Github set as well, or we can’t push code from within our scripts. You can check by going to Checkout SSH Keys in the project settings and ensuring that you have a key that is permitted to ssh to your git repo there. If not (i.e., you don’t have a user key there), click add user key and authorize the application to use your key. After authorizing, click on the ‘create and add user key’ button to get your ssh key on the machines so they can push code.

If you want to test the code on your local machine, go ahead and export GITHUB_ACCESS_TOKEN=<string here>.

Okay, now that we can actually run this code, let’s break it down.

Most of the comments explain the code clearly. Remember to replace your email in the git config section, and your Github API URL organziation name and project name in the last line. The gotchas are:

We have to reset our base branch to whatever it is on remote with git reset --hard origin/master. The reason for this is because as a byproduct of using CircleCI, a lot of things happen to our build machine repository before we execute a single line of code, and they could taint the final results. Syncing master with its remote counterpart ensures that we have a pristine copy, so it’s just good practice anyway.

These two lines:

git rev-parse --verify circle_ci/update_gemfile_lock
if ! [ 0 == $? ]; then

The first line checks to see if the git branch exists. If it does, it will show 0 when you run the command $?, else it will show something else. This tells us if the branch exists so we know whether to make one or not.

if [[ `git status --porcelain` ]];

Simply runs git status and outputs nothing if there is no change. If this outputs text, we want to make a pull request.

if [ `git rev-parse --abbrev-ref HEAD` == 'circle_ci/update_gemfile_lock' ]; then

Checks if our branch was actually created correctly. We do this so that we are sure we aren’t going to force push to the wrong branch.

curl -H "Content-Type: application/json" -X POST -d '{"title":"Title Of Pull Request Here","body":"Body of pull request here", "head": "circle_ci/make_apples", "base":"master"}'<organization i.e. Rob117>/<project i.e. NightlyBuilds>/pulls?access_token=$GITHUB_ACCESS_TOKEN

This is the magic. Base is the branch we started with in the beginning, the one that we want to reflect our code changes post-merge. Head is the branch we created in this script to do all of our work. We take the changes and make a pull request using our github access token that is stored in our environment variable.

Okay! We’ve broken down the bash script. Remember- just change the name of the base branch, created branch, and the file creation part to use the script in your own project. Running this script on your local environment – assuming you set your environment GITHUB_ACCESS_TOKEN var – should work smoothly and create a pull request on github.

Next, let’s edit our circle CI. It’s just a few lines of code added to your normal circle.yml:

# Other Circle.yml code

    - >
      if [ -n "${RUN_NIGHTLY_BUILD}" ]; then
        # If you put the script in your scripts folder, the path is ./scripts/<name>
        # As many scripts as you want here

# More circle CI code

This code says ‘if we get a post request that has variable called RUN_NIGHTLY_BUILD, execute the following scripts’.

That’s it! Push this code to the branch that you want to call it from. For example, maybe you want to keep all of your scripts in your builds branch. Simply merge the above script and yml file into a branch called builds, and push that branch. You’re all set to trigger your builds! You could also use a staging branch, develop branch, etc.

SUPER IMPORTANT NOTE: If you have more than one script set to run and you only store the scripts in a single branch – say, builds – and you change branches within a script, you MUST run git checkout builds between script runs / at the end of each script. If you don’t, Circle CI may fail to run all your scripts because you’ll have changed to a branch where they don’t exist!

Okay, next let’s set up our lambda and be done with this.

First, hop on over to Circle CI, go to project settings, then go to API Permissions, then click Create Token. The scope of the token should be All, and the label can be whatever you want. I chose Lambda Execution.

Save that token, and open up your AWS Control Panel. Switch over to Lambda.

Click Create Lambda Function, then click Blank Function. Next, click the empty dotted box to the left of the word Lambda and click Cloudwatch Events as the trigger. For the rule, create new rule. Rule name is Nightly-Build and the schedule expression is cron(0 0 * * ? *) -run at 12 UTC every day. (Source)

Check Enable Trigger, then move to the next screen.

Name the function ‘NightlyBuild’. Skip the code for now.

Set the environment variable CircleCIToken to .

Set the role to a basic lambda execution role.

Under advanced settings, set the timeout to 15 seconds.

Finally, the lambda code:

// Based on:
'use strict';
var http = require("https");

exports.handler = (event, context, callback) => {
    var options = {
        "method": "POST",
        "hostname": "",
        "port": null,
        //example: "/api/v1.1/project/github/RobUserName117/secret-project/tree/builds"
        "path": "/api/v1.1/project/<github or bitbucket>/<organization or user name>/<project name>/tree/<branch name with scripts in it>" +
        "?circle-token=" + process.env.CircleCIToken,
        "headers": {
            "content-type": "application/json",
            "cache-control": "no-cache"

    // Make request, write out all of response
    var req = http.request(options, function (res) {
        var chunks = [];

        res.on("data", function (chunk) {

        res.on("end", function () {
            var body = Buffer.concat(chunks);
            const response = body.toString();
            callback(null, response);

      "build_parameters": {
        "RUN_NIGHTLY_BUILD": "true"

Most of this code is self explanatory. The line that I would like to direct your attention to is the “path” part of your options object. According to the Circle CI docs, the URL that you post to is /api/v1.1/project/vcs system/organization or user name/project name/tree/branch

So if you wanted to push to the staging branch of the blogly project of the Google organization, stored in github, you would use:

The second thing is that we are calling our builds by sending a JSON with the RUN_NIGHTLY_BUILD parameter set to true. You could set or send as many variables as you please – see the docs for more details.

When this code runs successfully (click ‘TEST’), you should get a response that says something like “Last build was blah blah blah”. If you see a response that looks like CircleCI Webpage HTML, you made a configuration error somewhere.

Switch over to CircleCI and watch your build happen, then switch to Github and witness your glorious pull request!

Finished! Thanks for reading to the end.