AWS Lambda Async Invocations: A Practical Guide

In this article, which is part of our AWS Lambda Performance Optimization guide, we discuss using Lambda asynchronous invocations, including features and use cases.

AWS Lambda is a service used to build serverless applications on the AWS cloud. In AWS Lambda, you create Lambda functions to write business logic. A Lambda function can be invoked with the AWS console, Lambda API, AWS SDK, AWS CLI, and AWS toolkits. You can also configure other AWS services such as CloudWatch Events to invoke your function, or you can configure Lambda to read from a Kinesis stream or SQS queue and invoke your function.

In this article

AWS Lambda Invocations

A Lambda function is invoked in one of two ways:

Synchronous – With synchronous invocation, your function processes the event and returns a response. During this invocation, the client request is locked and awaits the completion of the function. When you invoke a function synchronously, if you receive errors in the response, the client needs to attempt retries.
Asynchronous – With asynchronous invocation, the Lambda function puts the event in a queue for processing and returns a response. It also retries and can send invocation records to a destination. When you invoke asynchronously, use an event source mapping, or configure another service to invoke your function. Retry requirements and the way that your function scales to handle large numbers of events can vary.

A trigger is used to invoke the Lambda function automatically. A trigger can be an AWS service that is configured to invoke the Lambda function to respond to lifecycle events, or on a schedule. AWS services such as CloudWatch Events can invoke the function based on a schedule. AWS S3 invokes your Lambda function when an object is created or updated.

To trigger a Lambda function from a stream or queue, you need to create an event source mapping. It reads items from an SQS queue, Kinesis stream, or DynamoDB stream, and sends them to the Lambda function.

Now, let’s see how asynchronous invocation works for a Lambda function.

Async Lambda Invocations

In an asynchronous flow, you don’t need to wait for the response from a Lambda function. AWS services such as AWS S3 or SNS invoke Lambda functions asynchronously by passing events. With Lambda Destinations, a new feature launched by AWS, it can send invocation records to downstream services to chain together.

Asynchronous invocation for a Lambda function requires the invocation type parameter as “Event”. It involves two steps –

The Lambda service places an event in a queue and returns a “success” response. It won’t have any information other than a status code.
A separate process reads events from the queue and sends them to your function.

$ aws lambda invoke –function-name async-function –invocation-type Event –payload ‘{ “name”: “value” }’ output.json
{
“StatusCode”: 202
}
If it is able to put the event in queue successfully, the output JSON will have a status code of 202, otherwise, it will show an error message. The Lambda service manages the function’s asynchronous event queue and when it fails with errors, it attempts a retry. It tries to run it two more times, with a one-minute wait between the first two attempts, and two minutes between the second and third attempts. These errors can be either returned by the code or runtime.

There might be instances when the function doesn’t have enough concurrency available to process all events. In those cases, additional requests are throttled. If it returns throttling errors (429) and server errors (50X series), Lambda returns the event to the queue and attempts to run the function again for up to 6 hours.

AWS Lambda Async with Node.js Promises and Await Syntax

Lambda functions can be written in several languages such as Java, NodeJS, Python, and more. If you use NodeJS and want to implement an asynchronous flow, you would need to use async handlers along with await. AWS Lambda’s async await feature was added after the launch of NodeJS 8.10 runtime. It enables users to enjoy the benefits of Promises using the async/await syntax and write easy-to-maintain clean code.

AWS Lambda async handlers automatically implement the await for Promises so developers don’t have to explicitly use this keyword in the code.

Below is a simple example of await in a Lambda function:

export const handler = async (event, context) => {
const response = await someAsyncFunc();
return response;
}

This is equivalent to this code:

export const handler = async (event, context) => {
return someAsyncFunc();
}

Using the Amazon Node.js Lambda Function Handler

Here are a few important details about how the Node.js Lambda function handler works:

Every time a function is invoked, Lambda runs the handler method.
The handler can accept one event at a time – when it exits or returns a response, it can handle another event.
The value of the handler setting is defined as {file-name}.{exported-handler-method}. For example, a handler method exported from an example.js file is defined as example.handler.
The handler method takes three arguments – an event object, which is a JSON-formatted string passed when the function was invoked, a context object, which provides details about the invocation, function, and execution environment, and a callback. In an async handler, you do not use callback. Instead, return a response, error, or Promise to the runtime.
If you are performing an asynchronous task in your code, return a Node.js Promise to ensure it finishes running. When a Promise is resolved or rejected, Lambda will send the response back to the invoker.

Here is an example of a JavaScript file performing an HTTP request with an async handler, using Promises. The code was shared in the Amazon documentation.

const https = require('https')
let url = "https://docs.aws.amazon.com/lambda/latest/dg/welcome.html"  

exports.handler = async function(event) {
  const promise = new Promise(function(resolve, reject) {
    https.get(url, (res) => {
        resolve(res.statusCode)
      }).on('error', (e) => {
        reject(Error(e))
      })
    })
  return promise
}

Invoking Async Lambdas via Asynchronous Invocation Configuration API

If you are using AWS CLI or AWS SDK to invoke an AWS Lambda function, you need to use configuration APIs to manage the asynchronous invocation settings.

You need to use the following API operations:

PutFunctionEventInvokeConfig – Used to create a configuration for invoking functions. It overwrites any existing configuration on the function, version, or alias.

$ aws lambda put-function-event-invoke-config --function-name error \
--maximum-event-age-in-seconds 3600 --maximum-retry-attempts 2
{
    "LastModified": 1175686212.411,
    "FunctionArn": "arn:aws:lambda:ap-east-2:112233445566:function:error:$LATEST",
    "MaximumRetryAttempts": 0,
    "MaximumEventAgeInSeconds": 3600,
    "DestinationConfig": {
        "OnSuccess": {},
        "OnFailure": {}
    }
}

GetFunctionEventInvokeConfig – Used to get the details of the existing configuration for the AWS Lambda invoke.
UpdateFunctionEventInvokeConfig – Also updates the configuration, but doesn’t override the existing configuration.

$ aws lambda update-function-event-invoke-config --function-name error \
--destination-config '{"OnFailure":{"Destination": "arn:aws:sqs:us-east-2:112233445566:destination"}}'
{
    "LastModified": 1175691862.482,
    "FunctionArn": "arn:aws:lambda:us-east-2:112233445566:function:error:$LATEST",
    "MaximumRetryAttempts": 0,
    "MaximumEventAgeInSeconds": 3600,
    "DestinationConfig": {
        "OnSuccess": {},
        "OnFailure": {
            "Destination": "arn:aws:sqs:ap-east-2:112233445566:destination"
        }
    }
}

ListFunctionEventInvokeConfigs – Used to retrieve a list of configurations for an asynchronous invocation for a function.
DeleteFunctionEventInvokeConfig – Used to delete the configuration for an asynchronous invocation for a function, version, or alias.

AWS Lambda Destinations – Where Can You Send Records of Async Invocations?

As mentioned above, with an AWS Lambda asynchronous flow, the Lambda service responds back immediately with the status code 202, as long as it is able to place the event in the queue successfully. If, however, anything goes wrong, it doesn’t notify the client. With Lambda Destinations, you can send the records of asynchronous invocations to another service. The Lambda destinations feature enables you to take action for both success and failure conditions. You can choose an SQS queue, an SNS topic, a Lambda function, or EventBridge as destinations.

The benefit of this feature is that you can connect to a destination service from a function without writing a single line of code. The alternative solution is to use AWS Step Functions, which is a bit of overkill for such a simple use case.

Dead-Letter Queues

An AWS Lambda dead-letter queue works similarly to an on-failure destination. You can configure the function to send discarded events to the dead-letter queue and process them again later. Dead-letter queue is attached with the function’s version configuration so once you publish a version, it’s locked with it. A DLQ can be either an SQS queue or an SNS topic.

Amazon SQS queue – Holds the failed events until they’re retrieved. To retrieve the events, you can configure a Lambda function to read them from the queue and invoke a function.

Amazon SNS topic – Doesn’t have any persistence, so it relays failed events to destinations such as an email address, a Lambda function, or an HTTP endpoint.

Summary

Event-driven is the most prevalent architecture used in cloud platforms since the inception of microservice programming, and AWS Lambda is helping to enable passing these events to the integrated services with minimal code written. The AWS Lambda asynchronous features enable implementing more and more use cases to automate data flows based on events. With the destination and dead-letter queue features, it becomes easy to implement error handling and retries without the need to write extra lines of code.