Lambda Logs
Serverless Monitoring Guide
Lambda Logs
Lambda CloudWatch Metrics

Lambda Logs: A Complete Guide

Serverless has become a popular deployment pattern for cloud applications. AWS Lambda has become the de-facto platform for building serverless applications. Every developer wants to get his hands dirty with Lambda; quickly build function code and run it. However, debugging a Lambda function can be a tricky task as most of the integration testing happens in a cloud environment.

AWS has come up with the SAM framework for locally debugging functions but it has limitations as it cannot spin up all the AWS services to integrate and test. So, most of the time debugging happens only in the AWS cloud and that requires a good logging mechanism.

AWS CloudWatch Logs is an AWS service for logging Lambda functions. In this article, we will discuss how Lambda function logging can be enabled in CloudWatch Logs and how to use CloudWatch features to get most of the information from these logs.

Lambda Function Logging

Lambda functions support several languages including Java, Python, Go, NodeJS, and .Net for building serverless applications. To output logs from function code, we need to use a logging library that writes to stdout or stderr. For example, the code below uses a logging library in Python to log events and environment variables:

import json
import os
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    logger.info('## ENVIRONMENT VARIABLES')
    logger.info(os.environ)
    logger.info('## EVENT')
    logger.info(event)

Every Lambda function has a default integration with a CloudWatch Logs log group. When a Lambda function spins up multiple container instances, it creates a log stream for each instance. The runtime sends details about each invocation to the log stream and relays logs and other output from the function’s code.

The Python runtime logs provide START, END, and REPORT lines for each invocation. Here is an example for the above Python code:

START RequestId: 4c1df9d1-xmpl-46da-9773-411e1eca8655 Version: $LATEST
[INFO]  2020-14-61T21:10:18.534Z    4c1df9d1-xmpl-46da-9773-411e1eca8655    ## ENVIRONMENT VARIABLES

[INFO]  2020-14-61T21:10:18.534Z    4c1df9d1-xmpl-46da-9773-411e1eca8655    
environ({'AWS_LAMBDA_LOG_GROUP_NAME': '/aws/lambda/my-function', 'AWS_LAMBDA_LOG_STREAM_NAME': 
'2020/01/31/[$LATEST]1bbe51xmplb34a2788dbaa7433b0aa4d', 'AWS_LAMBDA_FUNCTION_NAME': 'my-function', ...})

[INFO]  2020-14-61T21:10:18.535Z    4c1df9d1-xmpl-46da-9773-411e1eca8655    ## EVENT

[INFO]  2020-14-61T21:10:18.535Z    4c1df9d1-xmpl-46da-9773-411e1eca8655    {'key': 'value'}

END RequestId: 4c1df9d1-xmpl-46da-9773-411e1eca8655

REPORT RequestId: 4c1df9d1-xmpl-46da-9773-411e1eca8655  Duration: 2.45 ms   
Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 46 MB  Init Duration: 113.51 ms    

Report logs have various attributes that provide a lot of information about how the function is behaving.

  • RequestId – The unique request ID for each invocation.
  • Duration – The time required for the processing of the event.
  • Billed Duration – The duration of the invocation.
  • Memory Size – The amount of memory allocated to the function.
  • Max Memory Used – The amount of memory used by the function.
  • Init Duration – Initialization time for the first request served. Time took by the runtime to load the function and run code (code outside of the handler method).

This logging format is the same for each language supported by Lambda.

CloudWatch Logs

As described above, Lambda is integrated with the CloudWatch service by default. We just need to ensure the Function is attached with an IAM role that provides it with access to CloudWatch Logs. Before we jump into how Lambda logs are captured in CloudWatch Logs, let’s first understand how CloudWatch Logs works.

CloudWatch Logs has three major components:

Log Event

A log event is a record of an event that happened at a particular timestamp. A log event generally is made of timestamp and message.

Log Stream 

A log stream is a sequence of log events that are emitted from the same source. In AWS, each service creates a separate log stream in CloudWatch Logs.

Log Group

A log group is a group of log streams that share the same retention, monitoring, and access control settings. Log groups can be configured to select which streams to put together. There is no limit on the number of log streams that can be grouped into one log group.

Lambda Function Logs

The Lambda service creates a container for each function invocation. For each container invoked, it will create a new log stream. The log stream naming convention is data/[version]uuid.

Lambda creates a new log group for each function and maps all the log streams to it based on the relevant containers of the function. The log group naming convention is: aws/lambda/[Function Name ]

The log events from the Lambda function’s stdout and stderr are automatically redirected asynchronously to the CloudWatch Log group.

View CloudWatch Logs

In CloudWatch Logs, we can go to the log group for a particular function and search for particular events by using the Search log group functionality.

CloudWatch Logs has an Insights service that can be used to query the log data and identify potential issues. It includes a purpose-built query language with a few commands. It also provides sample queries, query auto-completion, command descriptions, and log field discovery.

Insights automatically discovers fields in logs from AWS services such as AWS Lambda, AWS CloudTrail, Amazon VPC, and Amazon Route 53, which emits log events as JSON. We can save queries and run complex queries on a need basis, without having to re-create them each time.

Once the CloudWatch Logs agent publishes log data to CloudWatch Logs, we can search and filter the log data by creating one or more metric filters.

Metric filters define the terms and patterns to look for in the CloudWatch log data. These filters are used to turn log data into numerical CloudWatch metrics and then create graphs or set an alert based on these metrics. CloudWatch supports various types of statistics, including percentile statistics, for analytics, or for setting alerts.

X-Ray Tracing

Distributed applications need to have tracing enabled for debugging issues. The same applies to Lambda functions. Lambda integrates with several AWS services such as DynamoDB, S3, and many others. To find out how a particular request is performing across all these services, AWS launched the X-Ray service.

X-Ray first collects data from each of the services, combines the gathered data into a single unit known as a trace. Then, using this data, it builds a service map that shows each request trace and displays the latency, HTTP status code, and other metadata information.

AWS X-Ray receives data from services as segments. X-Ray then groups segments that have a common request into traces. X-Ray processes the traces to generate a service graph that provides a visual representation of the Lambda-based application.

Once X-Ray instrumentation is done, we’ll be able to see the X-Ray traceId, segmentId, and sampled in CloudWatch Logs as well.

3rd-Party Lambda Monitoring Services

There are several 3rd-party providers of Lambda and serverless monitoring, logging, and debugging platform. They typically provide capabilities that go above and beyond those provided by CloudWatch and X-Ray. Lumigo is one such service, which features one-click automated tracing. It easily organizes everything related to a Lambda function in one place, showing only the most relevant information. This saves a lot of time in debugging errors or performance issues.

Summary

Logging is an essential part of application development. It is a must-have for debugging production issues. As Lambda is a serverless service, it becomes even more important, as most of the integration testing happens only in the AWS cloud. CloudWatch Logs supports Lambda logging natively. It provides several features to view the logs, query them, and generate dashboards.