Web frameworks implication for serverless cold starts

TL;DR

• Web application frameworks with all of their benefits add their own set of problems to FaaS environments.
• They introduce performance issues during cold start.
• We demonstrate that the web framework’s code initialization has more influence than the package size.

Introduction

In the world of web applications, Web Frameworks (WF) has existed for a very long time. Traditional WF existed in a world without the notion of microservices, let alone the notion of serverless. Many first-time serverless users use WF to ease the transition into the ecosystem, sometimes unaware of its effect on the performance. The following post focuses on potential cold start issues when using WF in the NodeJS ecosystem.

In the first part, we define serverless cold start, and how WF can affect it.
We then present our research hypothesis and describe the construction of the test.
Finally, we present our findings and recommendations for choosing WF.
We refer to other research work on the topic as well.

What is a cold start

…AWS Lambda launches an execution context…The execution context is a temporary runtime environment…It takes time to set up an execution context and do the necessary bootstrapping…You typically see this latency when a Lambda function is invoked for the first time… AWS Documentation

Each time your lambda runs for the first time, three things happen:

1. A container that will run your code is initialized.
2. The Lambda’s code is pulled.
3. The code is loaded into memory and initialized.

The Lambda’s execution context then passes arguments to your code for actual execution. The above steps introduce latency to your running function, a latency that in some cases, specifically in front-facing functionality, can harm the user experience.

WF increases the package size and the initialization time; therefore, it might increase the overall cold start time.

The paper’s goals

We believe, at Lumigo, that web frameworks are a great addition to the toolbox of a developer, and actually, quite a lot of Lumigo’s customers are using these types of frameworks. Our goal is to measure how real-world NodeJS WF affect cold start in AWS Lambda, considering:

• Package size.
• Code initialization time of the framework.

Testing methodology

• We chose 5 NodeJS frameworks with varying package sizes:
• Happi (337K)
• Koa (415K)
• Express (677K)
• Lambda API (45K)
• Restify (6700K)

We have two extremes here in terms of package size, Restify, which is ~ 6.7MB, and Lambda API, which is ~ 45K. In addition, we chose well known and widely used frameworks, like Express and Koa. 
All results were compared against a plain NodeJS package, i.e. a package that does not use any frameworks; instead, it uses just plain old Lambda (POLA).

• The results are based on the memory configuration of 512MB.
• The invocation was done through sls invoke on an EC2 (C5) machine located in the same region as the Lambda. It should be noted that no direct API GW call was made in order to reduce any latency caused by it.
• We wanted to test the initialization time, which affects the cold start; therefore, all test codes had a similar structure.

// Initialization code

const api = require('lambda-api')();

api.get('/', async (req,res) => {});

// Always same response
module.exports.handler = (event, context, callback) => {
 const response = {
   statusCode: 200
   body: "Hello from Lumigo!"
 };
 callback(null, response);
}

Depending on the framework, different initialization was done. All initializations were done according to the official framework documentation, and they included a single routing that does nothing. The actual response was consistent for all frameworks and returned the same json value.

• No executions were in parallel, i.e. only a single Lambda was executed concurrently to prevent a situation mentioned in [4], in which multiple containers ran on the same EC2, reducing performance for each Lambda.
• To allow reproducibility, the source code that ran the testing was opened sourced. More details are included at end of the blog post.

Package size vs. initialization

One of the hypotheses we wanted to test is whether code initialization incurs a bigger penalty than does package size. To test this hypothesis, we created an artificial package, which we call express-bigger in our chart, that contains Express with an additional asset (an image). The asset weighed around 5800K, with a total package size of 6210K. The asset was not loaded during the initialization.

Results

The following outcomes were measured:

A couple of words on the chart called candlestick.

Each X-axis item is built from 5 values,

• The center of the box is the median of all results.
• Open is median plus single standard deviation.
• Close is the median minus single standard deviation.
• The rectangular area between Open and Close values encompasses about 68% of all results.
• Around 95% of all results fall between the High and Low values.

Insights

• Adding WF adds latency to your cold start.
• For some frameworks, the cold start overhead is negligible.
• Initialization time of a WF package affects cold start time much more than the package size. Our tests suggest that code initialization time rather than package size should be the main criterion for choosing WF.
• Restify, which is almost as large as express-bigger, is much slower due to initialization time.
• Koa has the least effect on cold start time.

Our recommendation is not to rule out WF based only on its size, as code initialization time is a major factor in cold start performance. Use tools to assess the cold start time.

Source code

To allow reproducibility and/or add more frameworks, you can clone the source code that runs the test from https://github.com/lumigo-io/cold-start-frameworks. The testing framework was built with the ability to add more NodeJS frameworks, so if you use other frameworks, you would need to test your own configuration.

Conclusion

Web Frameworks is an important tool in the hands of developers, but using them will incur a cold start penalty, which needs to be considered, especially if it is a front-facing Lambda. Choosing the right Web Frameworks that minimizes cold start while giving you the right “bang for your buck” is not easy, but we have shown that package size should be the last criteria in your list. Instead, you should concentrate on code initialization minimization.

I would love to get your feedback on this research and hear your thoughts specifically regarding your experience with web frameworks with serverless applications and their effect on cold starts.

Related posts

Much work has been done in the area of cold start, usually trying to compare different package sizes, memory configurations, and languages. Unlike previous work, this post focuses on specific NodeJS packages and the effects of Web Frameworks on cold start.

Mikhail Shilkov did a thorough review of package size and its effect on the cold start and found that a 5MB Lambda package is twice as slow as a no dependencies package. On the other end, Yan Cui did not find any correlation between package size and cold start. Another interesting piece of research was done by Liang Wang et al. which thoroughly tested how Lambda environments behave both in cold start and in other scenarios. They found that running multiple Lambdas concurrently might affect performance due to the bin-packing algorithm of AWS. This insight informed our testing methodology. In addition, they found two types of cold starts, an existing VM cold start, and a new VM cold start. In this post, we used an existing VM cold start.

Bibliography

1. https://docs.aws.amazon.com/lambda/latest/dg/running-lambda-code.html
2. https://medium.com/@MikhailShilkov/serverless-cold-start-war-3978187677ce
3. https://read.acloud.guru/does-coding-language-memory-or-package-size-affect-cold-starts-of-aws-lambda-a15e26d12c76
4. https://www.usenix.org/conference/atc18/presentation/wang-liang
5. https://www.npmjs.com/package/lambda-api
6. https://en.wikipedia.org/wiki/Candlestick_chart