In this guest post, the first of a two-part series, Erez Rokah, developer of the open source aws-testing-library, outlines an advanced serverless CI/CD process for AWS Lambda.
Deploying a basic Serverless application has been made easy with the abundance of frameworks out there.
If you’re part of a small team or working on a relatively simple project, setting up a basic serverless CICD process is also pretty straightforward, since there is plenty of information on the subject.
But when a Serverless application grows it can get very complex very fast.
Requirements such as different deployment stages, deploying only modified services and acceptance testing intensify when dealing with the many small services forming such an application.
In this article I’m going to walk you step by step through the process of building an advanced CICD process that can be used as a basis for complex applications.
I’ll be using AWS CodeBuild for setting up the process, due to easier integration with AWS services and CloudFormation support.
In the next article we’ll focus on CircleCI, Travis Or Jenkins (let us know in the comments which one you’d prefer to see).
You should check my tweet here if you want to know why I’m not using CodePipeline with GitHub and only using CodeBuild.
I’m also ignoring the monorepo/multirepo discussion for the sake of simplicity. More about it here, here and here.
awscli
installed and configuredWe’ll be using the following repository as a baseline for setting up the deployment process. I recommend forking and cloning it so you can get your hands dirty 🙂
Make sure to run yarn install
after you fork and clone the repository in order to install required dependencies.
frontend
services
cicd
and buildspec.yml
in the root directoryLet’s look at the serverless.yml
file under the cicd
directory:
Note the serverless-plugin-cid. This is a useful plugin that let’s us describe a CodePipeline and CodeBuild CloudFormation stack directly from our serverless.yml
. You’ll notice I’m using a patched local version of the plugin as it seems the plugin is no longer maintained:
The serverless.yml
allows us to create two CICD processes, one for staging
stage and one for prod
stage.
Regions per stage are defined in the package.json
file:
More settings to take note of:
owner
and repository
values are generated automatically using the githubConfig.js
scriptmaster
branch will invoke a build for staging
, and a tag push matching “v.*” (e.g. “v0.0.1”) will invoke a build for prod
This is a one-time manual step (although it can be further automated using GitHub REST API).
Visit the following link to create two GitHub Tokens, one for staging
and one for prod
. Copy the values somewhere as we’ll need to add them to Parameter Store. Set the permissions as follows:
Next run the following commands from the cicd
directory and replace with your token values:
yarn connectToGitHubStaging --token ****************************
yarn connectToGitHubProd --token ****************************
You can see the full command in package.json
:
While you’re there, run the following commands and replace with your email:
yarn storeAdminEmailStaging --value admin@email.com
yarn storeAdminEmailProd --value admin@email.com
The commands will set the required environment variables for one of the application’s services.
Finally run the following commands to set up CodeBuild:
yarn setup --stage staging
yarn setup --stage prod
You can verify the build project was created using the AWS Console (change to the appropriate region if necessary).
The buildspec.yml
file describes our build and deploy process.
install
: Installs serverless framework
and yarn
pre_build
: Installs project dependencies and setup environment for current stagebuild
: runs linting, unit tests, deployment and e2e testsThe
cache
directive is used to cache project dependencies to save time on future builds
Set by CodeBuild:
AWS_REGION
is the region where the build is runningCODEBUILD_WEBHOOK_TRIGGER
is the event that triggered the build (we use it to skip deployment for pull requests)CODEBUILD_RESOLVED_SOURCE_VERSION
is the commit id for the build (we use it to figure out which services have changed since the previous build and require deployment)Set by us:
STAGE
was set by the serverless-plugin-cicd
pluginadmin_email
is taken from Parameter StoreFORCE_DEPLOY_ALL
is an optional variable (can be set manually using the console when running a build) to force deployment of all services regardless of changesEach service has is own build scripts, which are orchestrated using lerna
from the pacakge.json
at the root of the project:
For example yarn print:name
will run the print:name
script in every service that defines it in its package.json
, based on the repository dependency tree. You’ll notice that the script will run last for the frontend
service since it depends on monitoring-tester-service
and monitoring-appsync-service
:
While one might argue that this is abusing package dependency for services dependency, it does serve our purpose very well
Since we need more granular control over how lerna
runs our scripts, I wrote a deploy utility script under cicd/scripts/deploy.js
.
The utility uses some git diff-tree
magic, lerna
(internal) apis and the aws sdk
to figure out which services to deploy (only non-existing or changed services will be deployed).
Services deployment is batched according to the repository dependency tree.
I recommenced going over the code here for a better understanding.
An example for an e2e test can be found under services/monitoring-tester-service/e2e/checkEndpointStepFunction.chai.test.ts
:
checkEndpointStepFunction.chai.test.ts
The test starts a step function
and asserts it executed properly using aws-testing-library.
We’ve seen that setting up an advanced serverless CICD process using CodeBuild is possible when choosing the right tools, and has several advantages:
On the other hand it’s missing some features that make such a service more useful like:
Any missing feature in CodeBuild can probably be implemented using a Lambda function, but it doesn’t make sense to do so when other more popular services have native support for those features
In future articles we’ll discuss other more popular services and their advantages.
Until then, if you have questions about replicating this setup or simply want to talk serverless CICD, get in touch on Twitter.