Debugging AWS Lambda & Serverless Applications

AWS Lambda and serverless computing may offer several advantages over what came before, but ease of debugging is not one of them. Here we take an in-depth look at the challenges, best practices and recommended tools for AWS Lambda debugging.

Serverless computing has changed the way enterprises design and build their production applications. We’ve moved beyond the hype stage to the point where it can be considered a mainstream application development technology. It’s easy to see why. Serverless enables developers to build and ship faster, allowing them to concentrate on writing business logic rather than worrying about server provisioning, maintenance, idle capacity management, auto scaling and all those other tasks that eat away at our time.

Although there are a variety of serverless computing options, including some interesting cloud-agnostic ones, we’re going to focus on AWS serverless applications. It’s what we used to build our platform, and now – five years in – it dominates the space.

An AWS serverless application is comprised of several components, with the AWS Lambda function serving as the glue. Other AWS services interact with each other through Function, completing the serverless workflow.

 

separator

1. The Basic Components of a Serverless System

Before going further, let’s first remind ourselves of the basic components involved in building a serverless application.

Basic components of a serverless system

 

Event Source

Lambda Functions don’t have a REST API endpoint, which means they cannot be called directly from an outside client. A Lambda needs event-triggering to be invoked. An event could be:

  • A data record change of a DynamoDB table
  • A request to a REST endpoint through API Gateway
  • A change in the state of a resource (e.g. file uploaded to the S3 Bucket)

It can also be triggered through CloudWatch Events. As Lambda has evolved, there has been a  constant stream of new types of events that can serve as a trigger.

Lambda Function

All the functional logic goes here. Lambda functions can be written in a variety of languages, including but not limited to Java, Go, Node.js and Python. Functions are triggered by an event source and then connect to other AWS cloud services to create a serverless pattern.

Services

These can be either AWS services or external services. These services would be required either to persist or retrieve the data from other systems. Some of the services like SQS and SNS complement serverless flows with event-driven architecture.

Basic stuff, but we need to keep these components in mind while understanding the debugging of an AWS serverless application.

separator

2. Debugging Monolithic Apps vs Serverless Apps

With all of the talk these days about microservices and serverless applications, the monolithic application has become the black sheep of cloud system design. But despite that, there are certain aspects of application design where the humble monolith still does better than microservices and serverless. And ease of debugging is one of them. 

Microservices and serverless never promised to address any of the existing debugging challenges of monolith applications, and they don’t. If anything, they have introduced some major new challenges.

Debugging a Monolithic App
  • Debugging a monolith application conjures specific images in any experienced developer’s mind. Most likely their local environment using their favourite IDE ( Eclipse, VSC, intellij, etc.). Then it was as simple as creating a breakpoint at the line of code that needed to be inspected and then running the application in debug mode.
  • In monoliths, most issues are debugged locally as there are typically only a few failure points: UI/Backend or database behaviour. And in most cases, these can be set up in a local environment. 
  • Usually you would utilize eclipse/intellij to do remote debugging as well. It could be done from Eclipse, VSC or STS IDE. It would certainly require privileged access to connect to the remote servers .Even then it would only be used in exceptional cases where simulating the same scenario was not at all possible in local.
  • Testing and debugging a monolithic application is simpler since all tasks run on a single process.
  • Debugging a monolith may be easier but it takes a lot of effort. A large application may take minutes instead of seconds to start up, which can slow down debugging efforts. Also, reaching a debugging breakpoint takes time if the app flow is huge.
What’s Different About Debugging a Serverless Application?

Serverless has changed the way we design and build applications, and that changes the debugging process as well.

  • With serverless applications, logging has become the most common way of debugging issues. It has become important to log everything the service does so we developers understand real application behavior.
  • Serverless flows generally have more components involved, hence, more failure points. One of the key differences between debugging a microservices application and a serverless one, is that it’s simply not possible to set up all of these components in a local environment. 
  • Serverless applications by their nature, offer no option for remote debugging.
  • Hopping across process, machine, and networking boundaries introduces plenty of new variables and opportunities for things to go wrong – many of which are out of the developer’s control.
  • Loose dependency between components makes it harder to determine when compatibility or interface contracts are broken. 

So, as you can see, there are many differences in the way debugging works in monolithic and serverless applications. Let’s delve a little deeper into the challenges serverless introduces in debugging applications.

separator

3. The Challenges of Debugging a Serverless Application

A serverless application in AWS has event sources like API Gateway, DynamoDB, SQS or Kinesis invoking a Lambda Function. Then, the function makes calls to DBs, SNS/SQS, a workload on ECS/EKS, or even services running outside AWS, such as Twilio or Stripe.

It’s a veritable spider’s web of components, and an issue may require you, the developer, to debug any one (or more) of these many components. Here are a few common challenges:

Distributed Components

Serverless application components are highly distributed. Those services could be native to AWS or third-party. Simulating these locally may not always be possible, and doing debugging on a live environment is neither cheap nor easy.

Lack of Remote Debugging

Serverless applications by nature don’t give access to server and OS level, so remote debugging is not an option. For rare cases, where a developer is not able to understand why the issue is happening and cannot do live debugging this can be a real handicap.

Ephemerality

Servers are ephemeral by design in serverless computing and that gives very limited opportunity to identify root cause to debug issues.

High Cost

In serverless, most debugging is done in a live environment, and your monthly bill will reflect this, which is something to consider when it comes to the bottom line of your business.

Lack of Frameworks For Local Setup

There are very few frameworks available to set up Lambda in local and those that do come with limitations.

separator

4. Debugging Serverless Offline With AWS Serverless Application Model (SAM)

Running and debugging an application in local is a critical part of software development. Understanding that, AWS and the open source community have come up with several tools that enable developers to run and debug Lambda functions and serverless workflow in local.

AWS SAM is an open-source development framework used to build serverless applications on AWS. SAM has two major components:

SAM Template Specification

We use this specification to define our serverless application. We can describe the functions, APIs, permissions, configurations, and events in the specification. SAM templates are an extension of AWS CloudFormation templates, with some additional components that make them easier to work with. 

This is the YAML template for a SAM specification:

An example of a yaml template for AWS SAM

 

SAM Command Line Interface (SAM CLI)

We use SAM CLI to build serverless applications defined by AWS SAM templates. The CLI provides commands that enable us to:

  • Ensure that SAM template files match specification
  • Locally invoke functions 
  • Perform step-through debugging on functions
  • Package and deploy to the cloud
Process to Set Up the SAM CLI Debugger

1. You need to run the following commands to start and debug Lambda in local:

  • sam init – This generates a preconfigured AWS SAM template and example application code in the language of our choosing.
  • sam local invoke and sam local start-api – We use this command to test our application code locally. By adding a parameter -d we can enable a debug port as well.

e.g. sam local invoke -d 9999 -e event.json HelloWorldFunction

2. Once the function is up and running, the debugger is listening to localhost:9999. The event.json file has the event details configuration that is used to trigger the Lambda function. Now, we can use VSC or any other IDE to debug the code. Let’s take a quick walkthrough with VSC here. 

3. When we hit the F5 button, VSC opens up the launch configuration. This configuration needs to be updated so that it enables the debugger UI to debug at breakpoint. Below is a sample for the .vscode/launch.json configuration:

.vscode/launch.json configuration sample

4. Once we run the debugger UI with the above launch.json, VSCode will attach itself to the SAM CLI process and we should see the breakpoint code as highlighted. We can also use the inspect option to see the detail values of an object being debugged.

The debugger UI showing the breakpoint code

5. Alternatively, we can install the AWS Toolkit in VSCode. This tool enables the links for running the function locally. It also enables the debug option. To supply the event information, we use the Configure indicator from the CodeLens line, like so:

Installing the AWS Toolkit in VSCode

separator

5. Debugging Serverless Offline With Serverless Framework

Serverless Framework  is the most widely used open-source framework for developing and deploying serverless applications. The benefit of this framework is that it is not tightly coupled with AWS and can be used for other serverless providers like Azure Functions. Serverless Framework comes with an open-source CLI which provides support for commands to create and build serverless applications using a .yml file template.

How to Set Up the Serverless Framework Debugger

1. Install the serverless CLI using the following command:

npm install -g serverless

2. Create a new Serverless Service/Project:

serverless create –template aws-nodejs –path my-service

3. This will generate the scaffold for a Node.js function code with a handler.js file along with a serverless.yml file.

The handler.js file contains the handler method which will be invoked through events. Business logic will reside in this file. 

The serverless.yml file has configuration for all the Functions, Events and resources required for a serverless application flow.

4. Now, as our application is ready to run, we need to install a plugin called Serverless Offline.This plugin emulates AWS Lambda and API Gateway on your local machine to speed up your development cycles. 

5. Add serverless-offline to your project using the following command:

npm install serverless-offline –save-dev

6. Then, inside your project’s serverless.yml file, add plugin entry. If there is no existing plugin in the yml file, then add a new entry:

plugins:

  – serverless-offline

To test if the plugin was installed successfully, you can run this command:

serverless offline

It should print all the options for the plugin to run:

7. Now, run a Node.js application in debug mode.

Update the script section of the package.json file residing in the root folder:

“scripts”: {

“start”: “./node_modules/.bin/serverless offline -s dev”,

“debug”: “export SLS_DEBUG=* && node –debug ./node_modules/.bin/serverless offline -s dev”,

“test”: “mocha”

}

Note: This script is for linux environment. For Windows, we need to replace the “debug” parameter with:

 “debug”: “SET SLS_DEBUG=* && node –debug %USERPROFILE%\\AppData\\Roaming\\npm\\node_modules\\serverless\\bin\\serverless offline -s dev”

In the above code, we need to pay attention only to the debug section. The SLS_DEBUG environment variable is set to let Serverless Framework know that this application needs to be running in debug mode. And once node executes the node –debug command, it will use the serverless-offline plugin to run the function locally. We can also define the port in serverless.yml file which we want to map with this Node.js code.

8. Now, we just need to tell VSCode to run this code and attach a debugger to it by hitting F5. This will prompt the setup of a launch configuration. We need to use:

node.js launch via NPM

.vscode/launch.json needs to have the below configuration:

vscode launch json needs to have the following configuration

9. Now we simply hit F5 or click debug > Choose Debug and click play, and our Lambda functions will run locally with a debugger attached. If you’ve been following along, this is what you should see:

separator

6. How to Debug Serverless in the Cloud

So far, we’ve looked at how to debug a serverless application locally before we deploy the application to an AWS environment. However, once the application runs in a live environment, new issues may be discovered, as your functions will now connect to real AWS services, some of which we could not even test locally. 

So, let’s take a look at the steps required to debug a serverless application on a live AWS environment.

Debugging With API Testing Tools

To debug a serverless application on AWS, we may have to start with API Gateway testing. There are various clients available which can be used to test an end-point. The most common tools are Postman and SoapUI. Curl can also be used for simple testing. 

Postman provides options for setting the request type, headers, SSL certificates, body, and tests. This approach can be used for functions tied to API endpoints as event triggers, such as API Gateway. But if a function is triggered asynchronously by other services, we’ll need to rely on different tools to debug.

Debugging With the AWS Console

The AWS Console provides a useful option for testing Lambda functions directly without the involvement of an event source. It is very useful for checking real-time issues while deploying a Lambda for the first time.

First, it requires that test events be created. You can then run those events directly on the function while you are writing and building your application. These test events can emulate what is expected from any AWS service. Here’s an example used to create a thumbnail from an image whenever it gets uploaded to the S3 bucket:

Configuring a test event in AWS Console

When we hit the Test button, the output will be generated as follows:

AWS Console - test output

We can further check metrics using the “Monitoring” tab. It has out-of-the-box metrics to check error rate, throttling and more, which can be very helpful for debugging issues. Detail logs can also be viewed by clicking on the “View logs in CloudWatch” button.

 

Debugging With AWS CloudWatch Logs

CloudWatch Logs is the main source of information on how an AWS application is behaving, and that applies to Lambda Functions, too. 

Lambda Functions, by default, send log data to CloudWatch. CloudWatch then creates a LogGroup for each Lambda Function. 

Inside a LogGroup, there will be multiple Log Streams collating the logs that get generated from a particular Lambda instance (Lambda concurrency allows multiple instances created for each Lambda function).

Log Streams contain log events, and clicking on each event will provide detailed information for a particular Lambda function invocation.

So, whenever a Lambda function has an issue, we now know the steps to follow in order to dig through  the logs and see what’s happening for a particular invocation. Logs appearing in CloudWatch Logs are almost real time so collect as many logs as you need to debug an issue. 

That being said, there are a few caveats to using CloudWatch Logs:

  • AWS charges based on the size of log data being stored in it. So, it is recommended to remove unnecessary log statements once a bug is fixed. It may also increase the size of your Lambda function as well, which may cause a delay in loading and starting a function.
  • Many enterprises work with multiple clouds and/or on-premise platforms, so would likely be using Splunk or ELK for log aggregation. CloudWatch, being very tightly coupled with AWS services, would not be preferred by those organizations. In that case, logs from CloudWatch Logs are streamed to these commercial tools through streaming services like Kinesis. These tools also have the benefit of providing a visualization layer by allowing you to create several dashboards to meet your needs. 
  • CloudWatch Logs can become costly over time as AWS charges more for higher storage requirements. ELK, being open-source, is preferred by many to save costs.

Here’s an example of a typical flow used to store and view logs by these enterprises:

 

Debugging Using AWS X-Ray

Despite having rich features, CloudWatch Logs is limited to providing information for only one component – the Lambda Function. But, for a typical serverless flow, a function would be calling a variety of other AWS services. 

What if we need to get more details around these integrations, such as how they’re performing and what failures are causing performance issues? 

To help answer these questions, AWS introduced X-Ray, which helps developers analyze and debug distributed applications. X-Ray provides end-to-end tracing of requests, and generates a map of application components. It can be used to analyze applications in development and in production.

separator

7. Commercial Tools for Debugging Serverless on AWS Cloud

So, as you can see, debugging an application on AWS cloud is perfectly possible to do using native tools like SAM, CloudWatch and X-Ray, or open source tools like Serverless Framework.

However, we may end up having to call upon multiple tools to gather all the information we need to get the job done. 

That’s where commercial tools like Lumigo come in. For one thing, they’re much simpler to set up, and they provide a seamless experience for monitoring, tracing and debugging not only Lambda functions but also AWS and third-party managed services.

When an issue occurs, a developer can quickly zoom in to see an individual invocation of a Lambda function in the context of an end-to-end transaction flow.

We can then dig into the details for functions and services – including stack trace, inputs, outputs, parameters, environment variables, and correlated logs – in order to isolate root cause.

separator

8. Summary

Serverless has already shown its potential to revolutionize the way we build and deploy cloud applications. But while it solves a lot of problems for developers, particularly in the management of infrastructure, it introduces some new ones of its own, particularly in the areas of monitoring, debugging, testing and tracing. That’s why we see cloud providers and commercial vendors focusing on building the tools to address these problems.

SHARE WITH A FRIEND: