In the world of web applications, Web Frameworks (WF) has existed for a very long time. Traditional WF existed in a world without the notion of microservices, let alone the notion of serverless. Many first-time serverless users use WF to ease the transition into the ecosystem, sometimes unaware of its effect on the performance. The following post focuses on potential cold start issues when using WF in the NodeJS ecosystem.
In the first part, we define serverless cold start, and how WF can affect it.
We then present our research hypothesis and describe the construction of the test.
Finally, we present our findings and recommendations for choosing WF.
We refer to other research work on the topic as well.
…AWS Lambda launches an execution context…The execution context is a temporary runtime environment…It takes time to set up an execution context and do the necessary bootstrapping…You typically see this latency when a Lambda function is invoked for the first time… AWS Documentation
Each time your lambda runs for the first time, three things happen:
The Lambda’s execution context then passes arguments to your code for actual execution. The above steps introduce latency to your running function, a latency that in some cases, specifically in front-facing functionality, can harm the user experience.
WF increases the package size and the initialization time; therefore, it might increase the overall cold start time.
We believe, at Lumigo, that web frameworks are a great addition to the toolbox of a developer, and actually, quite a lot of Lumigo’s customers are using these types of frameworks. Our goal is to measure how real-world NodeJS WF affect cold start in AWS Lambda, considering:
We have two extremes here in terms of package size, Restify, which is ~ 6.7MB, and Lambda API, which is ~ 45K. In addition, we chose well known and widely used frameworks, like Express and Koa.
All results were compared against a plain NodeJS package, i.e. a package that does not use any frameworks; instead, it uses just plain old Lambda (POLA).
sls invoke
on an EC2 (C5) machine located in the same region as the Lambda. It should be noted that no direct API GW call was made in order to reduce any latency caused by it.// Initialization code
const api = require('lambda-api')();
api.get('/', async (req,res) => {});
// Always same response
module.exports.handler = (event, context, callback) => {
const response = {
statusCode: 200
body: "Hello from Lumigo!"
};
callback(null, response);
}
Depending on the framework, different initialization was done. All initializations were done according to the official framework documentation, and they included a single routing that does nothing. The actual response was consistent for all frameworks and returned the same json value.
One of the hypotheses we wanted to test is whether code initialization incurs a bigger penalty than does package size. To test this hypothesis, we created an artificial package, which we call express-bigger in our chart, that contains Express with an additional asset (an image). The asset weighed around 5800K, with a total package size of 6210K. The asset was not loaded during the initialization.
The following outcomes were measured:
A couple of words on the chart called candlestick.
Each X-axis item is built from 5 values,
Our recommendation is not to rule out WF based only on its size, as code initialization time is a major factor in cold start performance. Use tools to assess the cold start time.
To allow reproducibility and/or add more frameworks, you can clone the source code that runs the test from https://github.com/lumigo-io/cold-start-frameworks. The testing framework was built with the ability to add more NodeJS frameworks, so if you use other frameworks, you would need to test your own configuration.
Web Frameworks is an important tool in the hands of developers, but using them will incur a cold start penalty, which needs to be considered, especially if it is a front-facing Lambda. Choosing the right Web Frameworks that minimizes cold start while giving you the right “bang for your buck” is not easy, but we have shown that package size should be the last criteria in your list. Instead, you should concentrate on code initialization minimization.
I would love to get your feedback on this research and hear your thoughts specifically regarding your experience with web frameworks with serverless applications and their effect on cold starts.
Much work has been done in the area of cold start, usually trying to compare different package sizes, memory configurations, and languages. Unlike previous work, this post focuses on specific NodeJS packages and the effects of Web Frameworks on cold start.
Mikhail Shilkov did a thorough review of package size and its effect on the cold start and found that a 5MB Lambda package is twice as slow as a no dependencies package. On the other end, Yan Cui did not find any correlation between package size and cold start. Another interesting piece of research was done by Liang Wang et al. which thoroughly tested how Lambda environments behave both in cold start and in other scenarios. They found that running multiple Lambdas concurrently might affect performance due to the bin-packing algorithm of AWS. This insight informed our testing methodology. In addition, they found two types of cold starts, an existing VM cold start, and a new VM cold start. In this post, we used an existing VM cold start.