Serverless from the trenches 1: Protect your stack from deletion

It’s time to talk about the everyday challenges of serverless.

Whenever I scroll through the latest blog posts on serverless it feels like there are only two types of problems in the field: how do I get started and how do I architect my solution.

But what about all the day-to-day problems that developers and DevOps encounter when dealing with serverless? From simple deployment issues like protecting your stack from deletion to stress-testing your solution using serverless techniques.

That’s where Serverless from the Trenches comes in. Every other week these bite-sized blog posts will highlight a technique or tool to solve a real world problem that those of us on the frontline of serverless face on a daily basis.

Oops I deleted it again

Protecting your CloudFormation stacks from deletion is critical for production. So, in this post we’ll share an automated way to protect your CloudFormation stack using a simple bash script.

Each of our developers has their own AWS account (under one organization), because we believe that the best way to test your serverless code is in the cloud, not locally. Once in a while, the developer’s environment requires grooming, so we created a script that goes over all the stacks and deletes them, keeping the environment fresh, and avoiding the annoying sls remove.

In addition to our personal AWS environments, we have a shared environment that is used as part of our CI/CD. Before being merged into master, each branch is deployed to a shared AWS environment on which integration tests are running. We run our grooming script on that environment as well, usually once per week, as we like to keep old environments in case debugging is required. On one of those sleepless nights while working on our deployment we accidentally ran the grooming script on our production.

Fortunately, our IAM role forbids stack deletion, but it made us think that we’d better add another layer of protection.

One of the ways that are promoted by AWS is to use termination protection, just click on your stack and choose Edit termination protection:

But how do you automate it?

Automate the unautomatable

We wanted to automate it during the deployment process, i.e. on sls deploy, but since we couldn’t find a Serverless plugin to do the trick for us, we improvised using bash and AWS CLI.

AWS CLI supports the update-termination-protection command which enables us to quickly turn on the protection for any installed stack. All we need is to grab the stack name.

The following bash script does the trick:

1. sls deploy –region ${1} –stage ${2}
2. stack_name=$(sls info –stage ${2} –region ${1}|grep stack:|awk ‘{print $2}’)
3. echo ${stack_name}
4. aws cloudformation update-termination-protection –region ${1} –enable-termination-protection –stack-name ${stack_name}

Add this script to your deployment script and worry no more about accidental deletions.

So, how do you make sure nothing is being deleted by accident? Do you use bash scripts or other tools? Share your techniques for making your environment safer.