Lambda Execution Leaks: A Practical Guide

At Lumigo, we recently ran into some issues with a service we built on top of our Nodejs AWS Lambda handler. These issues were the result of lambda execution leaks from within our serverless code. In this article, I’ll explain about node.js lambda execution leaks and how to avoid them.

Lambda execution leaks in a nutshell

A Lambda execution leak takes place when executing code runs in a different invocation than the original execution context. This additional execution, which often occurs in the next invocation of the Lambda function, creates complexity in call patterns. This additional complexity can cause bugs, primarily due to developers not fully understanding why the errors are occurring in the first place, or when a deviation from expected behavior occurs.

Let’s explore this issue with some simple example code. The snippet below is a basic Lambda function that executes three times (function configured with a 60-second timeout and 128 MB Memory):

const randomBetween1To10 = () =>
    Math.floor(Math.random() * 10)
exports.handler = async (event, context) => {
    const randomNumber = randomBetween1To10()
    console.log("My random number - ", randomNumber);
    setTimeout(() => {
        console.log("(After 20ms) My random number - ", randomNumber);
    }, 20);
    return "OK";
};

After the three executions have finished, open the CloudWatch logs for your function, and observe the side effects:

These logs reveal two clear issues. The first is that our callbacks aren’t executing in the originating Lambda call – they’re returning for the next call! Additionally, in this call sequence the third invocation of setTimeout never even occurs – a big hole in our execution!

Let’s explore this further by printing the AWS RequestId (a unique identifier provided to each AWS request):

exports.handler = async (event, context) => {
    const requestId = context.awsRequestId;
    setTimeout(() => {
        console.log("My request-id: ", requested);
    }, 10);
    return "OK";
};

Again, let’s review the CloudWatch logs and see what happened:

Here, we can clearly see that the time-delayed requests are appearing in the context of a different AWS RequestID, specifically the previous request ID!

Leaks like this occur in many common development use-cases, such as when calling different services and third-party APIs. For example, the Lambda function below uses an API call to axios to expose the leak:

exports.handler = async (event, context) => {
    axios.post('/someApi', {
      firstName: 'Dori',
      lastName: 'Aviram'
});
.then(function (response) {
    //this function of code can be run in other invocation
     console.log(response);
});

return "OK";
};

So, what’s happening?

To understand what’s happened, we need to remember that there are several ways to create a Lambda function handler in Nodejs. Let’s walk through two of these – async handlers, and callback-based handlers

Async handlers:

The first method creates a handler using the async syntax:

exports.handler = async (event, context) => {
    // Our handler code here
    return "OK";
};

This handler returns to the AWS Runtime asynchronously via a JS Promise. This is the same as any other async function in Nodejs – the promise is handled by the AWS runtime, which waits for the promise to be either resolved or rejected. In the async version of this handler, this is accomplished by either returning a value or throwing an exception.

Callback based handlers:

The second option avoids using the async syntax with callbacks:

exports.handler = (event, context, callback) => {
    // Our handler code
    callback(null, "OK");
};

In callback-based handlers, the handler never returns a value to the AWS Runtime.

The runtime provides a callback function, which means that the invocation continues running until both the callback is invoked and our event loop becomes empty. In this case, you won’t see any active timers or pending requests, but AWS will continue to keep the request and runtime alive until the callback is resolved.

*By the way: with callback-based handlers, we can configure the Lambda runtime to respond immediately when the callback is invoked. Do this by disabling the callbackWaitsForEmptyEventLoop flag in the function configuration. If you do not disable this flag, invoking callback with the success/handler result will only set the result of the invocation, it won’t stop the invocation as expected!.

How to avoid Lambda execution leaks

During our work, we discovered two best practices that help avoid execution leaks and promise deterministically complete executions: using ESLint require-await rule, and simply avoiding the callback pattern altogether.

Using ESLint with the require-await rule:

At Lumigo, we use ESLint to help improve coding standards, enhance the readability of our source code for developers, and integrate tightly with our CI/CD pipeline. The time we save with ESLint — and the peace of mind we gain — lets us maintain uniformity in standards and prevent problematic leaks across the entire codebase.

In this case, using ESLint’s require-await flag is critical. The require-await flag causes linting to fail any time you call a Promise without a matching resolution. These are the exact types of small things that are hard to notice in code review – the linter lets us guarantee that this type of mistake never makes its way to production.

Avoiding the callback pattern:

Checking any asynchronous function in our system for correct handling is a hard task to begin with, but this process is paltry compared to the complexity of handling a callback function correctly in all cases. To track callback leaks down, you need to deep-dive into the function’s implementation to understand the leak, which can be a daunting task in code bases of any significant size. In general, we strongly recommend you don’t use callbacks. Promises provide standards for these types of asynchronous calls that make the events both easier to track and easier to maintain.

At Lumigo, we use util.promisify to help us catch errant callback-based functions and convert them to promise-based async functions. This example shows how a couple of lines of code can help you shortcut any number of event leaks in your system, regardless of the module’s author:

exports.handler = async (event, context) => {
    const readFileAsync = util.promisify(readFile);
    const result = await readFileAsync('myFile.json')
    return result;
};

Conclusion

When working in the AWS Lambda environment, Nodejs presents you with the potential to introduce event leaks into your code. These event leaks can occur when Lambda functions make any use of asynchronous callbacks in Node, whether through using the Promise pattern or the Callback pattern. At Lumigo, we recommend two best practices for avoiding these event leaks: making use of the require-await rule in ESLint, and simply avoiding callbacks in favor of async Promise constructs. With these two basic rules, you’ll find that your asynchronous communication problems simply evaporate into the cloud.