Advanced Serverless CICD - Part 1 :  AWS CodeBuild

In this guest post, the first of a two-part series, Erez Rokah, developer of the open source aws-testing-library, outlines an advanced serverless CI/CD process for AWS Lambda.

---

Introduction

Deploying a basic Serverless application has been made easy with the abundance of frameworks out there.

If you’re part of a small team or working on a relatively simple project, setting up a basic serverless CICD process is also pretty straightforward, since there is plenty of information on the subject.

But when a Serverless application grows it can get very complex very fast.
Requirements such as different deployment stages, deploying only modified services and acceptance testing intensify when dealing with the many small services forming such an application.

---

---

Let’s get started

In this article I’m going to walk you step by step through the process of building an advanced CICD process that can be used as a basis for complex applications.

I’ll be using AWS CodeBuild for setting up the process, due to easier integration with AWS services and CloudFormation support.

In the next article we’ll focus on CircleCI, Travis Or Jenkins (let us know in the comments which one you’d prefer to see).

---

CICD Process Requirements

1. Have different stages of deployment (e.g staging, prod)
2. Ability to deploy all/specific services with their dependencies
3. Reach high confidence when deploying services
4. Ability to easily add new services to the deployment process
5. Ability to recreate the CICD process in an automated way

Tool Stack

1. Serverless framework due to its inherent stages support and extensive plugin system
2. Lerna to manage dependencies, since we structure our code in a monorepo
3. Use aws-testing-library, a library I developed to run acceptance tests
4. Lerna also makes it is easier to generalize deployment scripts to avoid extra work when adding a new service
5. Using CodeBuild lets us describe our CICD using a CloudFormation stack so we can easily recreate the process

You should check my tweet here if you want to know why I’m not using CodePipeline with GitHub and only using CodeBuild.

I’m also ignoring the monorepo/multirepo discussion for the sake of simplicity. More about it here, here and here.

---

Prerequisites

• Nodejs (at least version 8)
• Yarn (we’ll be using the yarn workspaces feature with Lerna)
• Amazon AWS account and awscli installed and configured
• Serverless framework
• git and a GitHub account

---

---

Initial Setup

We’ll be using the following repository as a baseline for setting up the deployment process. I recommend forking and cloning it so you can get your hands dirty 🙂

Make sure to run yarn install after you fork and clone the repository in order to install required dependencies.

Repository Structure

• A react frontend app under frontend
• “Back-end” services under services
• CICD setup under cicd and buildspec.yml in the root directory

---

---

CICD Setup

Let’s look at the serverless.yml file under the cicd directory:

Note the serverless-plugin-cid. This is a useful plugin that let’s us describe a CodePipeline and CodeBuild CloudFormation stack directly from our serverless.yml. You’ll notice I’m using a patched local version of the plugin as it seems the plugin is no longer maintained:

The serverless.yml allows us to create two CICD processes, one for staging stage and one for prod stage.

Regions per stage are defined in the package.json file:

More settings to take note of:

• Github owner and repository values are generated automatically using the githubConfig.js script
• Push commits or pull requests to master branch will invoke a build for staging, and a tag push matching “v.*” (e.g. “v0.0.1”) will invoke a build for prod

---

---

Creating GitHub Tokens

This is a one-time manual step (although it can be further automated using GitHub REST API).
Visit the following link to create two GitHub Tokens, one for staging and one for prod. Copy the values somewhere as we’ll need to add them to Parameter Store. Set the permissions as follows:

Next run the following commands from the cicd directory and replace with your token values:

yarn connectToGitHubStaging --token ****************************
yarn connectToGitHubProd --token ****************************

You can see the full command in package.json:

While you’re there, run the following commands and replace with your email:

yarn storeAdminEmailStaging --value admin@email.com
yarn storeAdminEmailProd --value admin@email.com

The commands will set the required environment variables for one of the application’s services.

Finally run the following commands to set up CodeBuild:

yarn setup --stage staging
yarn setup --stage prod

You can verify the build project was created using the AWS Console (change to the appropriate region if necessary).

---

Build Setup

The buildspec.yml file describes our build and deploy process.

Build Phases

• install : Installs serverless framework and yarn
• pre_build : Installs project dependencies and setup environment for current stage
• build : runs linting, unit tests, deployment and e2e tests

The cache directive is used to cache project dependencies to save time on future builds

Environment variables

Set by CodeBuild:

• AWS_REGION is the region where the build is running
• CODEBUILD_WEBHOOK_TRIGGER is the event that triggered the build (we use it to skip deployment for pull requests)
• CODEBUILD_RESOLVED_SOURCE_VERSION is the commit id for the build (we use it to figure out which services have changed since the previous build and require deployment)

Set by us:

• STAGE was set by the serverless-plugin-cicd plugin
• admin_email is taken from Parameter Store
• FORCE_DEPLOY_ALL is an optional variable (can be set manually using the console when running a build) to force deployment of all services regardless of changes

---

Build Scripts

Each service has is own build scripts, which are orchestrated using lerna from the pacakge.json at the root of the project:

For example yarn print:name will run the print:name script in every service that defines it in its package.json, based on the repository dependency tree. You’ll notice that the script will run last for the frontend service since it depends on monitoring-tester-service and monitoring-appsync-service:

While one might argue that this is abusing package dependency for services dependency, it does serve our purpose very well

Since we need more granular control over how lerna runs our scripts, I wrote a deploy utility script under cicd/scripts/deploy.js.
The utility uses some git diff-tree magic, lerna (internal) apis and the aws sdk to figure out which services to deploy (only non-existing or changed services will be deployed).
Services deployment is batched according to the repository dependency tree.
I recommenced going over the code here for a better understanding.

---

End to end tests

An example for an e2e test can be found under services/monitoring-tester-service/e2e/checkEndpointStepFunction.chai.test.ts:

The test starts a step function and asserts it executed properly using aws-testing-library.

---

---

Summary

We’ve seen that setting up an advanced serverless CICD process using CodeBuild is possible when choosing the right tools, and has several advantages:

• It can be configured using CloudFormation
• It’s easier to integrate with other AWS services (e.g. CloudWatch events, Lambda)
• It makes permissions management easier (no need to give external services access to your AWS account)

On the other hand it’s missing some features that make such a service more useful like:

• Smart Email notifications. The process here will get you basic notifications on build progress, but is missing customization like only sending emails to PR initiators/reviewers, or only reporting state changes (e.g. build changing from failure to success or from success to failure)
• Slack integration can be added, but it would make sense to have native support for it
• CodeBuild knowledge base is not on par with more popular services (e.g. the unclear differences from CodePipeline described here)

Any missing feature in CodeBuild can probably be implemented using a Lambda function, but it doesn’t make sense to do so when other more popular services have native support for those features

In future articles we’ll discuss other more popular services and their advantages.

Until then, if you have questions about replicating this setup or simply want to talk serverless CICD, get in touch on Twitter.