AWS Lambda and serverless computing may offer several advantages over what came before, but easy debugging is not one of them. Here we take an in-depth look at the challenges, best practices, and recommended tools for AWS Lambda debugging.
Serverless computing has changed the way enterprises design and build their production applications. We’ve moved beyond the hype stage to the point where it can be considered a mainstream application development technology. It’s easy to see why. Serverless enables developers to build and ship faster, allowing them to concentrate on writing business logic rather than worrying about server provisioning, maintenance, idle capacity management, auto-scaling, and all those other tasks that eat away at our time.
Although there are a variety of serverless computing options, including some interesting cloud-agnostic ones, we’re going to focus on AWS serverless applications. It’s what we used to build our platform, and now – five years in – it dominates the space.
An AWS serverless application is comprised of several components, with the AWS Lambda function serving as the glue. Other AWS services interact with each other through Function, completing the serverless workflow.
This is part of our comprehensive guide to performance testing in a cloud native world.
Before going further, let’s first remind ourselves of the basic components involved in building a serverless application.
Lambda Functions don’t have a REST API endpoint, which means they cannot be called directly from an outside client. A Lambda needs event-triggering to be invoked. An event could be:
These can be either AWS services or external services. These services would be required either to persist or retrieve the data from other systems. Some of the services like SQS and SNS complement serverless flows with event-driven architecture.
Basic stuff, but we need to keep these components in mind while understanding the debugging of an AWS serverless application.
With all of the talk these days about microservices and serverless applications, the monolithic application has become the black sheep of cloud system design. But despite that, there are certain aspects of application design where the humble monolith still does better than microservices and serverless. And ease of debugging is one of them.
Microservices and serverless never promised to address any of the existing debugging challenges of monolith applications, and they don’t. If anything, they have introduced some major new challenges.
Serverless has changed the way we design and build applications, and that changes the debugging process as well.
So, as you can see, there are many differences in the way debugging works in monolithic and serverless applications. Let’s delve a little deeper into the challenges serverless introduces in debugging applications.
A serverless application in AWS has event sources like API Gateway, DynamoDB, SQS or Kinesis invoking a Lambda Function. Then, the function makes calls to DBs, SNS/SQS, a workload on ECS/EKS, or even services running outside AWS, such as Twilio or Stripe.
It’s a veritable spider’s web of components, and an issue may require you, the developer, to debug any one (or more) of these many components. Here are a few common challenges:
Serverless application components are highly distributed. Those services could be native to AWS or third-party. Simulating these locally may not always be possible, and doing debugging on a live environment is neither cheap nor easy.
Serverless applications by nature don’t give access to server and OS level, so remote debugging is not an option. For rare cases, where a developer is not able to understand why the issue is happening and cannot do live debugging this can be a real handicap.
Servers are ephemeral by design in serverless computing and that gives very limited opportunity to identify the root cause to debug issues.
In serverless, most debugging is done in a live environment, and your monthly bill will reflect this, which is something to consider when it comes to the bottom line of your business.
There are very few frameworks available to set up Lambda locally and those that exist come with limitations.
Running and debugging an application in local is a critical part of software development. Understanding that, AWS and the open-source community have come up with several tools that enable developers to run and debug Lambda functions and serverless workflow in local.
AWS SAM is an open-source development framework used to build serverless applications on AWS. SAM has two major components:
We use this specification to define our serverless application. We can describe the functions, APIs, permissions, configurations, and events in the specification. SAM templates are an extension of AWS CloudFormation templates, with some additional components that make them easier to work with.
This is the YAML template for a SAM specification:
We use SAM CLI to build serverless applications defined by AWS SAM templates. The CLI provides commands that enable us to:
1. You need to run the following commands to start and debug Lambda in local:
e.g. sam local invoke -d 9999 -e event.json HelloWorldFunction
2. Once the function is up and running, the debugger is listening to localhost:9999. The event.json file has the event details configuration that is used to trigger the Lambda function. Now, we can use VSC or any other IDE to debug the code. Let’s take a quick walkthrough with VSC here.
3. When we hit the F5 button, VSC opens up the launch configuration. This configuration needs to be updated so that it enables the debugger UI to debug at breakpoint. Below is a sample for the .vscode/launch.json configuration:
4. Once we run the debugger UI with the above launch.json, VSCode will attach itself to the SAM CLI process and we should see the breakpoint code as highlighted. We can also use the inspect option to see the detail values of an object being debugged.
5. Alternatively, we can install the AWS Toolkit in VSCode. This tool enables the links for running the function locally. It also enables the debug option. To supply the event information, we use the Configure indicator from the CodeLens line, like so:
Serverless Framework is the most widely used open-source framework for developing and deploying serverless applications. The benefit of this framework is that it is not tightly coupled with AWS and can be used for other serverless providers like Azure Functions. Serverless Framework comes with an open-source CLI which provides support for commands to create and build serverless applications using a .yml file template.
1. Install the serverless CLI using the following command:
npm install -g serverless
2. Create a new Serverless Service/Project:
serverless create –template aws-nodejs –path my-service
3. This will generate the scaffold for a Node.js function code with a handler.js file along with a serverless.yml file.
The handler.js file contains the handler method which will be invoked through events. Business logic will reside in this file.
The serverless.yml file has a configuration for all the Functions, Events, and resources required for a serverless application flow.
4. Now, as our application is ready to run, we need to install a plugin called Serverless Offline.This plugin emulates AWS Lambda and API Gateway on your local machine to speed up your development cycles.
5. Add serverless-offline to your project using the following command:
npm install serverless-offline –save-dev
6. Then, inside your project’s serverless.yml file, add plugin entry. If there is no existing plugin in the yml file, then add a new entry:
To test if the plugin was installed successfully, you can run this command:
It should print all the options for the plugin to run:
7. Now, run a Node.js application in debug mode.
Update the script section of the package.json file residing in the root folder:
“start”: “./node_modules/.bin/serverless offline -s dev”,
“debug”: “export SLS_DEBUG=* && node –debug ./node_modules/.bin/serverless offline -s dev”,
Note: This script is for linux environment. For Windows, we need to replace the “debug” parameter with:
“debug”: “SET SLS_DEBUG=* && node –debug %USERPROFILE%\\AppData\\Roaming\\npm\\node_modules\\serverless\\bin\\serverless offline -s dev”
In the above code, we need to pay attention only to the debug section. The SLS_DEBUG environment variable is set to let Serverless Framework know that this application needs to be running in debug mode. And once node executes the node –debug command, it will use the serverless-offline plugin to run the function locally. We can also define the port in serverless.yml file which we want to map with this Node.js code.
8. Now, we just need to tell VSCode to run this code and attach a debugger to it by hitting F5. This will prompt the setup of a launch configuration. We need to use:
.vscode/launch.json needs to have the below configuration:
9. Now we simply hit F5 or click debug > Choose Debug and click play, and our Lambda functions will run locally with a debugger attached. If you’ve been following along, this is what you should see:
So far, we’ve looked at how to debug a serverless application locally before we deploy the application to an AWS environment. However, once the application runs in a live environment, new issues may be discovered, as your functions will now connect to real AWS services, some of which we could not even test locally.
So, let’s take a look at the steps required to debug a serverless application on a live AWS environment.
To debug a serverless application on AWS, we may have to start with API Gateway testing. There are various clients available that can be used to test an end-point. The most common tools are Postman and SoapUI. Curl can also be used for simple testing.
Postman provides options for setting the request type, headers, SSL certificates, body, and tests. This approach can be used for functions tied to API endpoints as event triggers, such as API Gateway. But if a function is triggered asynchronously by other services, we’ll need to rely on different tools to debug.
The AWS Console provides a useful option for testing Lambda functions directly without the involvement of an event source. It is very useful for checking real-time issues while deploying a Lambda for the first time.
First, it requires that test events be created. You can then run those events directly on the function while you are writing and building your application. These test events can emulate what is expected from any AWS service. Here’s an example used to create a thumbnail from an image whenever it gets uploaded to the S3 bucket:
When we hit the Test button, the output will be generated as follows:
We can further check metrics using the “Monitoring” tab. It has out-of-the-box metrics to check error rate, throttling and more, which can be very helpful for debugging issues. Detail logs can also be viewed by clicking on the “View logs in CloudWatch” button.
CloudWatch Logs is the main source of information on how an AWS application is behaving, and that applies to Lambda Functions, too.
Lambda Functions, by default, send log data to CloudWatch. CloudWatch then creates a LogGroup for each Lambda Function.
Inside a LogGroup, there will be multiple Log Streams collating the logs that get generated from a particular Lambda instance (Lambda concurrency allows multiple instances created for each Lambda function).
Log Streams contain log events, and clicking on each event will provide detailed information for a particular Lambda function invocation.
So, whenever a Lambda function has an issue, we now know the steps to follow in order to dig through the logs and see what’s happening for a particular invocation. Logs appearing in CloudWatch Logs are almost real-time so collect as many logs as you need to debug an issue.
That being said, there are a few caveats to using CloudWatch Logs:
Here’s an example of a typical flow used to store and view logs by these enterprises:
Despite having rich features, CloudWatch Logs is limited to providing information for only one component – the Lambda Function. But, for a typical serverless flow, a function will call a variety of other AWS services.
What if we need to get more details around these integrations, such as how they’re performing and what failures are causing performance issues?
To help answer these questions, Amazon introduced AWS X-Ray, which helps developers analyze and debug distributed applications. X-Ray provides end-to-end tracing of requests and generates a map of application components. It can be used to analyze applications in development and in production.
So, as you can see, debugging an application on AWS cloud is possible to do using native tools like SAM, CloudWatch, and X-Ray, or open-source tools like Serverless Framework.
However, we will have to call upon multiple tools to gather all the information we need to get the job done — and it will involve a lot of tedious, manual work and code changes.
That’s where tools like Lumigo come in. For one thing, they’re much simpler to set up, and they provide a seamless experience for monitoring, tracing, and debugging not only Lambda functions but also AWS and third-party managed services.
When an issue occurs, a developer can quickly zoom in to see an individual invocation of a Lambda function in the context of end-to-end transaction flow.
We can then dig into the details for functions and services – including stack trace, inputs, outputs, parameters, environment variables, and correlated logs – in order to isolate the root cause.
Serverless has already shown its potential to revolutionize the way we build and deploy cloud applications. But while it solves a lot of problems for developers, particularly in the management of infrastructure, it introduces some new ones of its own, particularly in the areas of monitoring, debugging, testing, and tracing. That’s why we see cloud providers and commercial vendors focusing on building the tools to address these problems.
We have authored in-depth guides on several other topics that can be useful as you explore the world of performance testing.
Lambda functions can be extremely convenient for modern, microservices applications. However, they raise new performance issues, which can be hard to identify and debug. Learn the root causes of Lambda performance issues and how to improve performance for users.
See top articles in our Lambda performance guide:
Discover how to monitor serverless environments using logs, first party cloud monitoring tools like Amazon CloudWatch, and dedicated serverless monitoring solutions like Lumigo.
See top articles in our serverless monitoring guide:
Authored by Tigera
Learn how to monitor Kubernetes, the world’s leading container orchestrator, to improve the performance and security of large-scale containerized applications.