Using CircleCI as Continuous Deployment platform

Deploying new application versions manually should be a thing of the past. Especially in a microservice setup building and releasing new artifacts without a good tooling is a nightmare and having a good toolsuite for getting your code to run on a target environment should be one of the first things to setup.

After evaluating several different solutions we have settled for CircleCI as our Continuous Deployment platform. It’s fast, easy to configure, has a reasonable pricing model and a pretty good documentation.

In the following sections I want to describe our requirements for a Continuous Deployment platform and how we have achieved this with CircleCI.

The basic idea of our release flow (how we move a feature from code to actually be running and visible for our users) has been described by Sebastian in this blog post so here I will focus more on the how and not so much on the why.

The overall picture

A typical service within our microservice setup consists of a typical web application written in Ruby. For us moving a change on such an application to the target system involves the following series of steps:

1. Checkout the code from the version control system.
2. Run static code analysis and linting (we’re using Rubocop).
3. Run unit tests (we’re using RSpec).
4. Package the application into a Docker container.
5. Deploy the Docker container to the target runtime.

The deployment is splitted into two parts, depending on whether or not the change should be deployed to production:

1. Every commit that is pushed to our Git repository is built and analyzed.
2. Every commit on the master branch is also deployed to the test environment.
3. Every commit that is tagged with a tag confirming to the pattern release-X.Y.Z is also deployed to the production environment.

For now we use Heroku for all our deployments, so each application directly corresponds to one app in Heroku that is designated as test and one app in Heroku that is designated as production (following our naming schema).

A build script for a Ruby application

Let’s take a look at a simplified version of our build script that is part of the applications code base and stored in the file .circleci/configuration.yml within the Git repository.

It consists of two sections, with three jobs:

• The build job performs the code analysis and testing.
• The deploy jobs perform the deployment to either the test or production system depending on the type of commit found in the repository.

I have removed some steps (like caching dependencies) to focus on the overall picture.

version: 2.1
jobs:
  build:
    docker:
      - image: circleci/ruby:2.5-node-browsers

    steps:
      - checkout

      - run:
          name: Install dependencies
          command: |
            gem install bundler
            bundle install --jobs=8 --retry=3 --path vendor/bundle

      - run:
          name: Run Rubocop verifications
          command: |
            mkdir -p /tmp/test-results
            bundle exec rubocop --require rubocop/formatter/junit_formatter \
                                --format RuboCop::Formatter::JUnitFormatter \
                                --out /tmp/test-results/rubocop.xml

      - run:
          name: Run unit tests
          command: |
            mkdir -p /tmp/test-results
            TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)"
            bundle exec rspec --format progress \
                              --format RspecJunitFormatter \
                              --out /tmp/test-results/rspec.xml \
                              --format progress \
                              $TEST_FILES

      - store_test_results:
          path: /tmp/test-results

  deploy_to_test:
    docker:
      - image: circleci/ruby:2.5-node-browsers
    steps:
      - checkout

      - setup_remote_docker

      - run:
          name: Install Heroku CLI
          command: |
            wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh

      - run:
          name: Login into Heroku Docker Repository
          command: |
            docker login --username=$HEROKU_LOGIN --password=$HEROKU_API_KEY registry.heroku.com

      - run:
          name: Deploy Heroku Docker Container
          command: |
            heroku container:push worker -a t-example-ws-indexer
            heroku container:release worker -a t-example-ws-indexer

  deploy_to_production:
    docker:
      - image: circleci/ruby:2.5-node-browsers
    steps:
      - checkout

      - setup_remote_docker

      - run:
          name: Install Heroku CLI
          command: |
            wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh

      - run:
          name: Login into Heroku Docker Repository
          command: |
            docker login --username=$HEROKU_LOGIN --password=$HEROKU_API_KEY registry.heroku.com

      - run:
          name: Deploy Heroku Docker Container
          command: |
            heroku container:push worker -a p-example-ws-indexer
            heroku container:release worker -a p-example-ws-indexer

workflows:
  version: 2
  build_and_deploy:
    jobs:
      - build:
          filters:
            tags:
              only: /.*/
      - deploy_to_test:
          requires: [ build ]
          filters:
            branches:
              only: master
            tags:
              ignore: /release-.*/
          context: heroku-deploy
      - deploy_to_production:
          requires: [ build ]
          filters:
            branches:
              ignore: /.*/
            tags:
              only: /release-.*/
          context: heroku-deploy

CircleCI can directly integrate with GitHub (which we use to host our repositories) and will automatically be notified by GitHub whenever a new commit has been pushed to a repository.

Optimizing the configuration for a microservice setup

While the configuration listed above works perfectly and does exactly what we want it to do it’s very much focused on one service.

However our application landscape is made up of multiple services.

Copying and pasting the same build configuration into all services may be a good start but won’t allow easy changes in the future. What if we decide to add an additional step or replace Rubocop with some other static code analyzer? We would have to make the same change in all our repositories.

That’s not what we want.

Luckily CircleCI provides a way for us to centralize our configuration but using what CircleCI calls Orbs. An Orb is basically a set of externalized pieces of build configuration.

Nearly all of the build configuration listed above is independent of the actual project that is being built, so we can extract all of this into an Orb.

The only configuration that remains in the actual project is a much smaller build configuration referencing our external orbs:

version: 2.1

orbs:
  recipe: "betterdoc/recipes-ruby@1.1.0"
  deploy: "betterdoc/deploy-to-heroku@1"

workflows:
  version: 2
  build_and_deploy:
    jobs:
      - recipe/build-regular:
          name: "build"
          filters:
            tags:
              only: /.*/
      - deploy/docker-build-release:
          name: "deploy-to-test"
          requires: [ "build" ]
          context: heroku-deploy
          app-name: t-example-ws-indexer
          filters:
            branches:
              only: /master/
            tags:
              ignore: /.*/
      - deploy/docker-build-release:
          name: "deploy-to-production"
          requires: [ "build" ]
          context: heroku-deploy
          app-name: p-example-ws-indexer
          filters:
            branches:
              ignore: /.*/
            tags:
              only: /.*/

All the nitty gritty details of how to perform the build and how to perform the deployment have been moved into the betterdoc/recipes-ruby and betterdoc/deploy-to-heroku Orbs.

Let’s take a simplified version of what an Orb actually looks like (in this case we use the betterdoc/recipes-ruby Orb).

version: 2.1
description: "Building Ruby applications"

executors:
  ruby-25:
    docker:
      - image: betterdoc/circleci-ruby-25
  ruby-25-postgres:
    docker:
      - image: betterdoc/circleci-ruby-25
        environment:
          DATABASE_URL: postgresql://root@localhost/circle_test
      - image: circleci/postgres:9.6.5-alpine-ram

commands:
  run-tests:
    steps:
      - run:
          name: "Run Rubocop verifications"
          command: |
            mkdir -p /tmp/test-results
            bundle exec rubocop \
              --require rubocop/formatter/junit_formatter \
              --format RuboCop::Formatter::JUnitFormatter \
              --out /tmp/test-results/rubocop.xml
      - run:
          name: "Run unit tests"
          command: |
            mkdir -p /tmp/test-results
            TEST_FILES="$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)"
            bundle exec rspec \
              --format progress \
              --format RspecJunitFormatter \
              --out /tmp/test-results/rspec.xml \
              --format progress \
              $TEST_FILES
      - store_test_results:
          path: /tmp/test-results
  resolve-dependencies:
    steps:
      - run:
          name: "Install dependencies"
          command: |
            gem install bundler
            bundle install --jobs=8 --retry=3 --path vendor/bundle
  migrate-database:
    steps:
      - run:
          name: Migrate test database
          command: |
            bin/rake db:migrate RAILS_ENV=test
            bin/rake db:seed RAILS_ENV=test

jobs:
  build-regular:
    executor: ruby-25
    steps:
      - checkout
      - resolve-dependencies
      - run-tests
  build-rails:
    executor: ruby-25-postgres
    steps:
      - checkout
      - resolve-dependencies
      - migrate-database
      - run-tests

Make future configuration updates easy

CircleCI enforces a semantic versioning scheme for the Orbs.

In our example we’re using two kinds of references:

• The betterdoc/recipes-ruby Orb is referenced by its exact version number. No matter what we change within the Orb itself, the build configuration will always use the Orb in version 1.0.0.
• The betterdoc/deploy-to-heroku Orb is referenced more general by using only the major version 1. This means that whenever we publish a new Orb version that uses the same major version number all future builds will automatically use the new Orb version without any changes to the build configuration in the actual application.

The Orbs that we’re using for our builds are publicly available:

• The deployed versions are located within the CircleCI Orb repository at: https://circleci.com/orbs/registry/?query=betterdoc&filterBy=all
• The source code at GitHub: https://github.com/betterdoc-org/circleci-orbs/

Conclusion

For us CircleCI is a perfect match for what we look for in a Continuous Deployment platform. It’s easy to integration, easy to configure to our specific needs and integrates seamlessly into our development process.