AWS released Lambda Extensions just over a month ago, and we explained what they are and why they matter. Today, AWS has announced AWS Lambda Logs API, a way for extensions to subscribe to and process Lambda logs in real-time.
The most obvious use case for this new feature is collecting and forwarding your Lambda logs to other services in real-time, without going through the subscription filters of Amazon CloudWatch Logs. As an AWS launch partner for this new feature, we at Lumigo are proud to launch an open-source extension that you can use to ship your Lambda logs to S3. From there, you can perform additional post-processing or use Athena to query them at scale or forward them anywhere else you want. In the future, it will support shipping to other destinations out of the box. And of course, you’re welcome to contribute to the project and add support for your destinations.
A Lambda extension can subscribe to three different types of log streams:
To subscribe, the extension makes a HTTP PUT request to http://${AWS_LAMBDA_RUNTIME_API}/2020-08-15/runtime/logs. In the HTTP body, it can configure the subscription:
The subscription must be made during the initialization phase of the extension’s lifecycle. After the initialization phase is over, the Lambda Logs API will not process any subscription requests from the extension.
Also, if an extension subscribes to the extension logs (for example, to collect logs from other extensions) and logs something every time it receives some events, then it will receive those logs from the Logs API too. This can result in an endless stream of frivolous log messages. To stop this from happening, consider buffering and sending the extension’s logs to your own backend instead.
Our open-source extension ships your logs to an S3 bucket of your choosing, and it’ll follow the same folder structure as Kinesis Firehose. If you’re collecting and processing application events with Kinesis Firehose, Glue and Athena already, then this should feel very familiar.
The files use a newline-delimited format and can be processed easily.
But hey, why should you use this instead of using CloudWatch subscription filters with a Lambda function? Well, here are a couple of reasons:
And of course, there are trade-offs:
Get the open-source lambda-log-shipper from GitHub. I hope you find it useful. And please check out our other open-source tools while you’re here, including the Lumigo CLI and the Lambda Janitor.
You can also get a free Lumigo account with its rich capabilities for monitoring and debugging serverless applications.