Lambda Logs: A Complete Guide

Serverless has become a popular deployment pattern for cloud applications. AWS Lambda has become the de-facto platform for building serverless applications. Every developer wants to get his hands dirty with Lambda; quickly build function code and run it. However, serverless monitoring and debugging a Lambda function can be a tricky task as most of the integration testing happens in a cloud environment.

AWS has come up with the SAM framework for locally debugging functions but it has limitations as it cannot spin up all the AWS services to integrate and test. So, most of the time debugging happens only in the AWS cloud and that requires a good logging mechanism.

AWS CloudWatch Logs is an AWS service for logging Lambda functions. In this article, we will discuss how Lambda function logging can be enabled in CloudWatch Logs and how to use CloudWatch features to get most of the information from these logs.

Lambda Function Logging

Lambda functions support several languages including Java, Python, Go, NodeJS, and .Net for building serverless applications. To output logs from function code, we need to use a logging library that writes to stdout or stderr. For example, the code below uses a logging library in Python to log events and environment variables:

import json import os import logging logger = logging.getLogger() logger.setLevel(logging.INFO) def lambda_handler(event, context):     logger.info('## ENVIRONMENT VARIABLES')    logger.info(os.environ)     logger.info('## EVENT')     logger.info(event)

The Python runtime logs provide START, END, and REPORT lines for each invocation. Here is an example for the above Python code:

START RequestId: 4c1df9d1-xmpl-46da-9773-411e1eca8655 Version: $LATEST [INFO]  2020-14-61T21:10:18.534Z    4c1df9d1-xmpl-46da-9773-411e1eca8655    ## ENVIRONMENT VARIABLES [INFO]  2020-14-61T21:10:18.534Z    4c1df9d1-xmpl-46da-9773-411e1eca8655    environ({'AWS_LAMBDA_LOG_GROUP_NAME': '/aws/lambda/my-function', 'AWS_LAMBDA_LOG_STREAM_NAME': '2020/01/31/[$LATEST]1bbe51xmplb34a2788dbaa7433b0aa4d', 'AWS_LAMBDA_FUNCTION_NAME': 'my-function', ...}) [INFO]  2020-14-61T21:10:18.535Z    4c1df9d1-xmpl-46da-9773-411e1eca8655    ## EVENT [INFO]  2020-14-61T21:10:18.535Z    4c1df9d1-xmpl-46da-9773-411e1eca8655    {'key': 'value'} END RequestId: 4c1df9d1-xmpl-46da-9773-411e1eca8655 REPORT RequestId: 4c1df9d1-xmpl-46da-9773-411e1eca8655  Duration: 2.45 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 46 MB  Init Duration: 113.51 ms

• RequestId – The unique request ID for each invocation.
• Duration – The time required for the processing of the event.
• Billed Duration – The duration of the invocation.
• Memory Size – The amount of memory allocated to the function.
• Max Memory Used – The amount of memory used by the function.
• Init Duration – Initialization time for the first request served. Time took by the runtime to load the function and run code (code outside of the handler method).

This logging format is the same for each language supported by Lambda.

Tips from the experts

1. Implement structured logging

   Use structured logging (e.g., JSON format) to make your logs machine-readable and easily searchable in CloudWatch Logs Insights. This allows you to extract key-value pairs and create more insightful queries and dashboards, making debugging and monitoring more effective.

2. Minimize logging verbosity in production

   Excessive logging can increase latency and incur higher costs. Set log levels appropriately; use DEBUG or INFO levels during development and reduce to WARN or ERROR in production environments. This helps to keep the log noise down and focuses on critical issues.

3. Use log retention policies

   Set your Logs retention policies to automatically delete older logs that are no longer needed. This helps to manage storage costs and keeps log groups manageable. Retention periods should be aligned with compliance and operational requirements.

4. Create custom metrics from logs

   Use metric filters to extract specific data points from logs, such as error rates, latencies, or custom events, and create metrics. These metrics can then trigger alarms or be used in dashboards, providing real-time insights without sifting through logs manually.

5. Reduce cold start log noise

   Cold starts are inherent in serverless applications, but they can clutter logs. Monitor cold start occurrences and analyze them separately using Lambda metrics (Init Duration) rather than log events. This approach reduces noise and allows a more targeted performance optimization strategy.

6. Integrate with third-party log management tools

   Consider integrating CloudWatch Logs with third-party log management and monitoring tools like Lumigo for enhanced features like alerting, anomaly detection, and advanced analytics. These tools can provide a more unified and enriched view of logs across all services.

7. Optimize log storage costs with Log Archive and Compression

   Use AWS S3 to archive old logs that you need to retain for compliance or long-term analysis. Configure CloudWatch Logs to export logs to S3, and enable compression to reduce storage costs significantly. Automate this with AWS Lambda to run at scheduled intervals.

8. Use asynchronous logging for performance optimization

   When dealing with high-throughput applications, synchronous logging can slow down function execution. Implement asynchronous logging solutions or batch log entries to CloudWatch Logs to minimize performance impact, ensuring that your Lambda functions remain efficient and responsive.

Aviad Mor

CTO

Aviad Mor is the Co-Founder & CTO at Lumigo. Lumigo’s SaaS platform helps companies monitor and troubleshoot microservices applications while providing actionable insights that prevent business disruptions. Aviad has over a decade of experience in technology leadership, heading the development of core products in Check Point from inception to wide adoption.

CloudWatch Logs

As described above, Lambda is integrated with the CloudWatch service by default. We just need to ensure the Function is attached with an IAM role that provides it with access to CloudWatch Logs. Before we jump into how Lambda logs are captured in CloudWatch Logs, let’s first understand how CloudWatch Logs works.

CloudWatch Logs has three major components:

Log Event

A log event is a record of an event that happened at a particular timestamp. A log event generally is made of timestamp and message.

Log Stream 

A log stream is a sequence of log events that are emitted from the same source. In AWS, each service creates a separate log stream in CloudWatch Logs.

Log Group

A log group is a group of log streams that share the same retention, monitoring, and access control settings. Log groups can be configured to select which streams to put together. There is no limit on the number of log streams that can be grouped into one log group.

Lambda Function Logs

The Lambda service creates a container for each function invocation. For each container invoked, it will create a new log stream. The log stream naming convention is data/[version]uuid.

Lambda creates a new log group for each function and maps all the log streams to it based on the relevant containers of the function. The log group naming convention is: aws/lambda/[Function Name ]

The log events from the Lambda function’s stdout and stderr are automatically redirected asynchronously to the CloudWatch Log group.

View CloudWatch Logs

In CloudWatch Logs, we can go to the log group for a particular function and search for particular events by using the Search log group functionality.

CloudWatch Logs has an Insights service that can be used to query the log data and identify potential issues. It includes a purpose-built query language with a few commands. It also provides sample queries, query auto-completion, command descriptions, and log field discovery.

Insights automatically discovers fields in logs from AWS services such as AWS Lambda, AWS CloudTrail, Amazon VPC, and Amazon Route 53, which emits log events as JSON. We can save queries and run complex queries on a need basis, without having to re-create them each time.

Once the CloudWatch Logs agent publishes log data to CloudWatch Logs, we can search and filter the log data by creating one or more metric filters.

Metric filters define the terms and patterns to look for in the CloudWatch log data. These filters are used to turn log data into numerical CloudWatch metrics and then create graphs or set an alert based on these metrics. CloudWatch supports various types of statistics, including percentile statistics, for analytics, or for setting alerts.

X-Ray Tracing

Distributed applications need to have tracing enabled for debugging issues. The same applies to Lambda functions. Lambda integrates with several AWS services such as DynamoDB, S3, and many others. To find out how a particular request is performing across all these services, AWS launched the X-Ray service.

X-Ray first collects data from each of the services, combines the gathered data into a single unit known as a trace. Then, using this data, it builds a service map that shows each request trace and displays the latency, HTTP status code, and other metadata information.

AWS X-Ray receives data from services as segments. X-Ray then groups segments that have a common request into traces. X-Ray processes the traces to generate a service graph that provides a visual representation of the Lambda-based application.

Once X-Ray instrumentation is done, we’ll be able to see the X-Ray traceId, segmentId, and sampled in CloudWatch Logs as well.

3rd-Party Lambda Monitoring Services

There are several 3rd-party providers of Lambda and serverless monitoring, logging, and debugging platform. They typically provide capabilities that go above and beyond those provided by CloudWatch and X-Ray. Lumigo is one such service, which features one-click automated tracing. It easily organizes everything related to a Lambda function in one place, showing only the most relevant information. This saves a lot of time in debugging errors or performance issues.

Summary

Logging is an essential part of application development. It is a must-have for debugging production issues. As Lambda is a serverless service, it becomes even more important, as most of the integration testing happens only in the AWS cloud. CloudWatch Logs supports Lambda logging natively. It provides several features to view the logs, query them, and generate dashboards.