Lambda Logging

AWS Lambda Python Logging: A Quick Guide

The AWS Lambda framework is one of the most used services consumed by AWS customers. It is used to build event-driven architecture and serverless applications. It supports various different languages like Java, Python, NodeJS, and many more. Logging is one of the important mechanisms to debug and triage production issues of a function so, in this article, we are going to talk through how logging is enabled for an AWS Lambda Python function.

This is part of a series of articles about AWS lambda monitoring.

AWS Lambda Logging With CloudWatch

A Lambda function inherently comes with a CloudWatch Logs log group and each instance of your function has a log stream. When a function is invoked, the runtime (Python, Java, etc..) sends details about each invocation to the log stream. It relays logs and other output from the function’s code.

There are two ways through which you can output logs from your function code:

• Use the print method – print(event)
• Use any logging library that writes to stdout or stderr – logging, aws-lambda-logging

Below is a format for a log example:

START RequestId: 7f231cfc-xmpl-2647-b01a-bc68ec115c16 Version: 1.2
## ENVIRONMENT VARIABLES
environ({'AWS_LAMBDA_LOG_GROUP_NAME': '/aws/lambda/log-function', 'AWS_LAMBDA_LOG_STREAM_NAME': '2020/09/25/[1.2]3893xmpl7fac4513b41bb42b621a213c', 'AWS_LAMBDA_FUNCTION_NAME': 'log-function', ...})
## EVENT
{'key': 'value'}
END RequestId: 7f231cfc-xmpl-2647-b01a-bc68ec115c16
REPORT RequestId: 87f231cfc-xmpl-2647-b01a-bc68ec115c16  Duration: 10.14 ms  Billed Duration: 90 ms Memory Size: 128 MB Max Memory Used: 66 MB  Init Duration: 120.19 ms   
XRAY TraceId: 1-3e54a324-10brweplf1fb22f01bc555a1   SegmentId: 01f1xmpl2d1f3f63 Sampled: true

When you use Python runtime, it logs the START, END, and REPORT lines for each AWS Lambda Python function invocation. The start line displays the request id and version of the function. The end line displays the request id and the report line provides a little more details:

Report Log

• RequestId – The unique request ID for AWS Lambda python function invocation.
• Duration – The amount of time of your function’s handler method spent to process the event.
• Billed Duration – The amount of execution time you will be billed for the current invocation.
• Memory Size – The amount of memory allocated to the function.
• Max Memory Used – The amount of max memory used by the function.
• Init Duration – This provides information for the first request served. The amount of time the runtime took to load the function and run code outside of the handler method.
• XRAY TraceId – This is logged only when you enable XRay for the Lambda function. It displays the AWS X-Ray trace ID.
  • SegmentId – For traced requests, the X-Ray segment ID.
  • Sampled – For traced requests, the sampling result.

There are various ways you can view logs for an AWS Lambda Python function – in the Lambda console, in the CloudWatch Logs console, or from the command line.

As described earlier in this article, you can either use the print method or library for logging. The best approach is a library as it provides several features such as enable or disable logs at runtime. On top of that, it emits logs in the form of stdout or stderr that can be used to flush out these logs to any third-party tools.

Structured Logging with JSON: Generating Readable Logs from Python Lambda Functions

This is based on the tutorial and Github repo by Vincent Van der Kussen. It will show you how to structure your Lambda logs in Python to generate a clear, readable log format in CloudWatch.

Lambda’s default log format is not ideal for analysis. Let’s see how to convert it to a readable JSON format.

Step 1: Create a custom formatter

You’ll need a custom JSON formatter – a small piece of code that formats logs in a readable way. You can get a simple formatter called json_logger here.

Note that this formatter disables the default Python Lambda log handler. To turn it back on, set logger.propagate = True in the setup_logger() function.

Step 2: Import the formatter into your Lambda functions

To use the formatter, import it and initiate an instance in each of your Lambda functions, and use it to log events, like this:

Step 4: See the result in the CloudWatch console

Here is how the default log handler shows an event:

And here is the nicely formatted JSON event:

Image Source: ZooLite

Using Logging Libraries with Lambda

You can do more with logging using specialized logging libraries. Below we describe the use of the Python community Logging library, which is not specialized for AWS Lambda, and the aws-lambda-logging library, which is.

Python Logging Library

Logging library is provided and maintained by the Python community itself. The advantage of this library module is that all Python modules can participate in logging. Your application log can include your customized messages integrated with messages from third-party modules.

This module provides a few of the basic classes, together with their functions:

• Loggers – expose the interface that function code directly uses. To use logger, you need to use logger.getLogger(). This method uses a singleton pattern and always returns the same object reference. Once it’s initialized, you can use logger.debug(), logger.info(), logger.error() and many other methods.
• Handlers – send the log records that are created by loggers to the appropriate destination. Handler is never instantiated directly; it acts as a base class for other useful subclasses. However, the __init__() method in subclasses needs to call Handler.__init__().
• Filters – provide a finer grained facility for filtering and determining which log records to be sent for output. For example, a filter initialized with ‘X.Y’ will allow events logged by loggers ‘X.Y’, ‘X.Y.Z’, ‘X.Y.C.D’, ‘X.Y.D’ etc. but not ‘X.YY’, ‘Y.X.Y’ etc.
• Formatters – specify the layout and format of log records in the final output so that can be interpreted by either a human or an external system. You have to use this method for formatting – logging.Formatter(fmt=None, datefmt=None, style=’%’).

Below is a basic example of usage of logging library in Python:

import os
import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

def lambda_handler(event, context):
    logger.info('## ENVIRONMENT VARIABLES')
    logger.info(os.environ)
    logger.debug('## EVENT')
    logger.debug(event)

AWS Lambda Logging Library

Python Package Index, a very well known repository, provides the aws-lambda-logging library.

This library has been built to provide better logging for AWS Lambda running on a Python runtime environment. Most production-like environments use third party tools and systems that parse your logs and show metrics, analytics, and other features. This library gives a highly opinionated JSON formatting to ease the parsing.

To use this library, you need to apply this code:

import aws_lambda_logging

def handler(event, context):
    aws_lambda_logging.setup(level='DEBUG')

Now, if you are using Boto3 AWS SDK for Python programming, you can set the level of boto3 separately:

  aws_lambda_logging.setup(level='DEBUG', boto_level='CRITICAL')

Let’s take a simple example of AWS Lambda logging and see what will be the output:

import aws_lambda_logging

def handler(event, context):
    aws_lambda_logging.setup(level='DEBUG', aws_request_id=context.get('aws_request_id'))
    log.debug('first example')
    log.debug('{"Inputs": [“param1”,”param2”,”param3”]}')

The output will be in a JSON formatted message:

{
    "level": "DEBUG",
    "timestamp": "2020-09-25 11:37:22,428",
    "apigw_request_id": "513fde81-536d-12e1-c7ed-1d374ea70152",
    "location": "root.handler:6",
    "message": {
"first example",
"Inputs": [
            "param1",
            "param2",
            "param3"
        ]
    }
}

Summary

AWS Lambda is one of the most used services to implement serverless architecture patterns. And Python being the most used language in recent years, it’s natural that developers use either Python directly or AWS boto3 SDK. However, debugging serverless applications is the most tedious task and having a good logging library helps you to make this job easier. In this article, we saw both a plain logging library and aws-lambda-logging library that helps to relay application information in human-readable format and make debugging easier.

Want a much easier and faster way to monitor and debug your serverless applications? Sign up for a free Lumigo account.