Lambda Logs API: a new way to process Lambda logs in real-time

AWS released Lambda Extensions just over a month ago, and we explained what they are and why they matter. Today, AWS has announced AWS Lambda Logs API, a way for extensions to subscribe to and process Lambda logs in real-time.

The most obvious use case for this new feature is collecting and forwarding your Lambda logs to other services in real-time, without going through the subscription filters of Amazon CloudWatch Logs. As an AWS launch partner for this new feature, we at Lumigo are proud to launch an open-source extension that you can use to ship your Lambda logs to S3. From there, you can perform additional post-processing or use Athena to query them at scale or forward them anywhere else you want. In the future, it will support shipping to other destinations out of the box. And of course, you’re welcome to contribute to the project and add support for your destinations.

How it works

A Lambda extension can subscribe to three different types of log streams:

1. Lambda function logs – basically anything that’s written to stdout or stderr
2. Platform logs – the START, END, and REPORT messages you see for every invocation in CloudWatch Logs
3. Extension logs – logs from the extensions themselves

To subscribe, the extension makes a HTTP PUT request to http://${AWS_LAMBDA_RUNTIME_API}/2020-08-15/runtime/logs. In the HTTP body, it can configure the subscription:

• The buffering configuration – max bytes, max items, and timeout (ms).
• Event types – the aforementioned “lambda”, ”platform”, or ”extension” logs. Note that this is an array so that you can subscribe to all three log streams in one request.
• Destination – you can ask the logs to be delivered to either a local TCP port in NDJSON format or to a local HTTP (no HTTPS!) endpoint in JSON.

The subscription must be made during the initialization phase of the extension’s lifecycle. After the initialization phase is over, the Lambda Logs API will not process any subscription requests from the extension.

Also, if an extension subscribes to the extension logs (for example, to collect logs from other extensions) and logs something every time it receives some events, then it will receive those logs from the Logs API too. This can result in an endless stream of frivolous log messages. To stop this from happening, consider buffering and sending the extension’s logs to your own backend instead.

Lumigo extension: lambda-log-shipper

Our open-source extension ships your logs to an S3 bucket of your choosing, and it’ll follow the same folder structure as Kinesis Firehose. If you’re collecting and processing application events with Kinesis Firehose, Glue and Athena already, then this should feel very familiar.

The files use a newline-delimited format and can be processed easily.

But hey, why should you use this instead of using CloudWatch subscription filters with a Lambda function? Well, here are a couple of reasons:

1. You can only have two subscription filters per log group, and you might be using them already for other purposes.
2. You don’t need a separate Lambda function for shipping your logs anymore, and it helps keep your project more self-contained.
3. You don’t incur the extra cost for invoking a separate Lambda function to ship your logs.
4. You don’t incur the extra concurrent executions for invoking a separate Lambda function to ship your logs. Therefore one less risk of hitting the regional concurrent executions limit unwittingly.

And of course, there are trade-offs:

1. Every function needs to install this extension, which requires extra configuration for the extension via environment variables.
2. Log aggregation is often a platform decision and should not be done differently on each function. Enforcing a consistent approach across an entire organization is difficult when every function is responsible for shipping its logs to [insert name of vendor]. If you choose to use this extension, it will go a long way to standardize it in your organization so that all your teams follow the same consistent approach.
3. The extension shares CPU, network, and memory with your function code. So for CPU or memory-intensive functions, running the extension might have an impact on performance.

Get the open-source lambda-log-shipper from GitHub. I hope you find it useful. And please check out our other open-source tools while you’re here, including the Lumigo CLI and the Lambda Janitor.

You can also get a free Lumigo account with its rich capabilities for monitoring and debugging serverless applications.