In this guest post, the first of a two-part series, Erez Rokah, developer of the open source aws-testing-library, outlines an advanced serverless CI/CD process for AWS Lambda.
Deploying a basic Serverless application has been made easy with the abundance of frameworks out there.
If you’re part of a small team or working on a relatively simple project, setting up a basic serverless CICD process is also pretty straightforward, since there is plenty of information on the subject.
But when a Serverless application grows it can get very complex very fast.
Requirements such as different deployment stages, deploying only modified services and acceptance testing intensify when dealing with the many small services forming such an application.
In this article I’m going to walk you step by step through the process of building an advanced CICD process that can be used as a basis for complex applications.
I’ll be using AWS CodeBuild for setting up the process, due to easier integration with AWS services and CloudFormation support.
You should check my tweet here if you want to know why I’m not using CodePipeline with GitHub and only using CodeBuild.
awscliinstalled and configured
We’ll be using the following repository as a baseline for setting up the deployment process. I recommend forking and cloning it so you can get your hands dirty 🙂
Make sure to run
yarn install after you fork and clone the repository in order to install required dependencies.
buildspec.ymlin the root directory
Let’s look at the
serverless.yml file under the
Note the serverless-plugin-cid. This is a useful plugin that let’s us describe a CodePipeline and CodeBuild CloudFormation stack directly from our
serverless.yml. You’ll notice I’m using a patched local version of the plugin as it seems the plugin is no longer maintained:
serverless.yml allows us to create two CICD processes, one for
staging stage and one for
Regions per stage are defined in the
More settings to take note of:
repositoryvalues are generated automatically using the
masterbranch will invoke a build for
staging, and a tag push matching “v.*” (e.g. “v0.0.1”) will invoke a build for
This is a one-time manual step (although it can be further automated using GitHub REST API).
Visit the following link to create two GitHub Tokens, one for
staging and one for
prod. Copy the values somewhere as we’ll need to add them to Parameter Store. Set the permissions as follows:
Next run the following commands from the
cicd directory and replace with your token values:
yarn connectToGitHubStaging --token ****************************
yarn connectToGitHubProd --token ****************************
You can see the full command in
While you’re there, run the following commands and replace with your email:
yarn storeAdminEmailStaging --value email@example.com
yarn storeAdminEmailProd --value firstname.lastname@example.org
The commands will set the required environment variables for one of the application’s services.
Finally run the following commands to set up CodeBuild:
yarn setup --stage staging
yarn setup --stage prod
You can verify the build project was created using the AWS Console (change to the appropriate region if necessary).
buildspec.yml file describes our build and deploy process.
pre_build: Installs project dependencies and setup environment for current stage
build: runs linting, unit tests, deployment and e2e tests
cachedirective is used to cache project dependencies to save time on future builds
Set by CodeBuild:
AWS_REGIONis the region where the build is running
CODEBUILD_WEBHOOK_TRIGGERis the event that triggered the build (we use it to skip deployment for pull requests)
CODEBUILD_RESOLVED_SOURCE_VERSIONis the commit id for the build (we use it to figure out which services have changed since the previous build and require deployment)
Set by us:
STAGEwas set by the
admin_emailis taken from Parameter Store
FORCE_DEPLOY_ALLis an optional variable (can be set manually using the console when running a build) to force deployment of all services regardless of changes
Each service has is own build scripts, which are orchestrated using
lerna from the
pacakge.json at the root of the project:
yarn print:name will run the
print:name script in every service that defines it in its
package.json, based on the repository dependency tree. You’ll notice that the script will run last for the
frontend service since it depends on
While one might argue that this is abusing package dependency for services dependency, it does serve our purpose very well
Since we need more granular control over how
lerna runs our scripts, I wrote a deploy utility script under
The utility uses some
git diff-tree magic,
lerna (internal) apis and the
aws sdk to figure out which services to deploy (only non-existing or changed services will be deployed).
Services deployment is batched according to the repository dependency tree.
I recommenced going over the code here for a better understanding.
An example for an e2e test can be found under
The test starts a
step function and asserts it executed properly using aws-testing-library.
We’ve seen that setting up an advanced serverless CICD process using CodeBuild is possible when choosing the right tools, and has several advantages:
On the other hand it’s missing some features that make such a service more useful like:
Any missing feature in CodeBuild can probably be implemented using a Lambda function, but it doesn’t make sense to do so when other more popular services have native support for those features
In future articles we’ll discuss other more popular services and their advantages.
Until then, if you have questions about replicating this setup or simply want to talk serverless CICD, get in touch on Twitter.