Serverless Debugging: An Introduction

Lambda Functions don’t have a REST API endpoint, which means they cannot be called directly from an outside client. A Lambda needs event-triggering to be invoked. An event could be:

• A data record change of a DynamoDB table

• A request to a REST endpoint through API Gateway

• A change in the state of a resource (e.g. file uploaded to the S3 Bucket)

It can also be triggered through CloudWatch Events. As Lambda has evolved, there has been a  constant stream of new types of events that can serve as a trigger.

All the functional logic goes here. Lambda functions can be written in a variety of languages, including but not limited to Java, Go, Node.js and Python. Functions are triggered by an event source and then connect to other AWS cloud services to create a serverless pattern.

These can be either AWS services or external services. These services would be required either to persist or retrieve the data from other systems. Some of the services like SQS and SNS complement serverless flows with event-driven architecture.

Basic stuff, but we need to keep these components in mind while understanding the debugging of an AWS serverless application.

With all of the talk these days about microservices and serverless applications, the monolithic application has become the black sheep of cloud system design. But despite that, there are certain aspects of application design where the humble monolith still does better than microservices and serverless. And ease of debugging is one of them.

Microservices and serverless never promised to address any of the existing debugging challenges of monolith applications, and they don’t. If anything, they have introduced some major new challenges.

• Debugging a monolith application conjures specific images in any experienced developer’s mind. Most likely their local environment using their favorite IDE ( Eclipse, VSC, IntelliJ, etc.). Then it was as simple as creating a breakpoint at the line of code that needed to be inspected and then running the application in debug mode.
• In monoliths, most issues are debugged locally as there are typically only a few failure points: UI/Backend or database behavior. And in most cases, these can be set up in a local environment.
• Usually, you would utilize Eclipse/IntelliJ to do remote debugging as well. It could be done from Eclipse, VSC, or STS IDE. It would certainly require privileged access to connect to the remote servers. Even then it would only be used in exceptional cases where simulating the same scenario was not at all possible in local.
• Testing and debugging a monolithic application is simpler since all tasks run on a single process.
• Debugging a monolith may be easier but it takes a lot of effort. A large application may take minutes instead of seconds to start up, which can slow down debugging efforts. Also, reaching a debugging breakpoint takes time if the app flow is huge.

Serverless has changed the way we design and build applications, and that changes the debugging process as well.

• With serverless applications, logging has become the most common way of debugging issues. It has become important to log everything the service does so we developers understand real application behavior.
• Serverless flows generally have more components involved, hence, more failure points. One of the key differences between debugging a microservices application and a serverless one is that it’s simply not possible to set up all of these components in a local environment.
• Serverless applications by their nature, offer no option for remote debugging.
• Hopping across process, machine, and networking boundaries introduces plenty of new variables and opportunities for things to go wrong – many of which are out of the developer’s control.
• Loose dependency between components makes it harder to determine when compatibility or interface contracts are broken.

So, as you can see, there are many differences in the way debugging works in monolithic and serverless applications. Let’s delve a little deeper into the challenges serverless introduces in debugging applications.

A serverless application in AWS has event sources like API Gateway, DynamoDB, SQS or Kinesis invoking a Lambda Function. Then, the function makes calls to DBs, SNS/SQS, a workload on ECS/EKS, or even services running outside AWS, such as Twilio or Stripe.

It’s a veritable spider’s web of components, and an issue may require you, the developer, to debug any one (or more) of these many components. Here are a few common challenges:

Serverless application components are highly distributed. Those services could be native to AWS or third-party. Simulating these locally may not always be possible, and doing debugging on a live environment is neither cheap nor easy.

Serverless applications by nature don’t give access to server and OS level, so remote debugging is not an option. For rare cases, where a developer is not able to understand why the issue is happening and cannot do live debugging this can be a real handicap.

Servers are ephemeral by design in serverless computing and that gives very limited opportunity to identify the root cause to debug issues.

In serverless, most debugging is done in a live environment, and your monthly bill will reflect this, which is something to consider when it comes to the bottom line of your business.

There are very few frameworks available to set up Lambda locally and those that exist come with limitations.

Running and debugging an application in local is a critical part of software development. Understanding that, AWS and the open-source community have come up with several tools that enable developers to run and debug Lambda functions and serverless workflow in local.

AWS SAM is an open-source development framework used to build serverless applications on AWS. SAM has two major components:

We use this specification to define our serverless application. We can describe the functions, APIs, permissions, configurations, and events in the specification. SAM templates are an extension of AWS CloudFormation templates, with some additional components that make them easier to work with.

This is the YAML template for a SAM specification:

We use SAM CLI to build serverless applications defined by AWS SAM templates. The CLI provides commands that enable us to:

• Ensure that SAM template files match specification
• Locally invoke functions
• Perform step-through debugging on functions
• Package and deploy to the cloud

1. You need to run the following commands to start and debug Lambda in local:

• sam init – This generates a preconfigured AWS SAM template and example application code in the language of our choosing.
• sam local invoke and sam local start-api – We use this command to test our application code locally. By adding a parameter -d we can enable a debug port as well.

e.g. sam local invoke -d 9999 -e event.json HelloWorldFunction

2. Once the function is up and running, the debugger is listening to localhost:9999. The event.json file has the event details configuration that is used to trigger the Lambda function. Now, we can use VSC or any other IDE to debug the code. Let’s take a quick walkthrough with VSC here.

3. When we hit the F5 button, VSC opens up the launch configuration. This configuration needs to be updated so that it enables the debugger UI to debug at breakpoint. Below is a sample for the .vscode/launch.json configuration:

4. Once we run the debugger UI with the above launch.json, VSCode will attach itself to the SAM CLI process and we should see the breakpoint code as highlighted. We can also use the inspect option to see the detail values of an object being debugged.

5. Alternatively, we can install the AWS Toolkit in VSCode. This tool enables the links for running the function locally. It also enables the debug option. To supply the event information, we use the Configure indicator from the CodeLens line, like so:

Serverless Framework is the most widely used open-source framework for developing and deploying serverless applications. The benefit of this framework is that it is not tightly coupled with AWS and can be used for other serverless providers like Azure Functions. Serverless Framework comes with an open-source CLI which provides support for commands to create and build serverless applications using a .yml file template.

1. Install the serverless CLI using the following command:

npm install -g serverless

2. Create a new Serverless Service/Project:

serverless create –template aws-nodejs –path my-service

3. This will generate the scaffold for a Node.js function code with a handler.js file along with a serverless.yml file.

The handler.js file contains the handler method which will be invoked through events. Business logic will reside in this file.

The serverless.yml file has a configuration for all the Functions, Events, and resources required for a serverless application flow.

4. Now, as our application is ready to run, we need to install a plugin called Serverless Offline.This plugin emulates AWS Lambda and API Gateway on your local machine to speed up your development cycles.

5. Add serverless-offline to your project using the following command:

npm install serverless-offline –save-dev

6. Then, inside your project’s serverless.yml file, add plugin entry. If there is no existing plugin in the yml file, then add a new entry:

plugins:

  – serverless-offline

To test if the plugin was installed successfully, you can run this command:

serverless offline

It should print all the options for the plugin to run:

7. Now, run a Node.js application in debug mode.

Update the script section of the package.json file residing in the root folder:

“scripts”: {

“start”: “./node_modules/.bin/serverless offline -s dev”,

“debug”: “export SLS_DEBUG=* && node –debug ./node_modules/.bin/serverless offline -s dev”,

“test”: “mocha”

}

Note: This script is for linux environment. For Windows, we need to replace the “debug” parameter with:

“debug”: “SET SLS_DEBUG=* && node –debug %USERPROFILE%\\AppData\\Roaming\\npm\\node_modules\\serverless\\bin\\serverless offline -s dev”

In the above code, we need to pay attention only to the debug section. The SLS_DEBUG environment variable is set to let Serverless Framework know that this application needs to be running in debug mode. And once node executes the node –debug command, it will use the serverless-offline plugin to run the function locally. We can also define the port in serverless.yml file which we want to map with this Node.js code.

8. Now, we just need to tell VSCode to run this code and attach a debugger to it by hitting F5. This will prompt the setup of a launch configuration. We need to use:

.vscode/launch.json needs to have the below configuration:

9. Now we simply hit F5 or click debug > Choose Debug and click play, and our Lambda functions will run locally with a debugger attached. If you’ve been following along, this is what you should see:

So far, we’ve looked at how to debug a serverless application locally before we deploy the application to an AWS environment. However, once the application runs in a live environment, new issues may be discovered, as your functions will now connect to real AWS services, some of which we could not even test locally.

So, let’s take a look at the steps required to debug a serverless application on a live AWS environment.

To debug a serverless application on AWS, we may have to start with API Gateway testing. There are various clients available that can be used to test an end-point. The most common tools are Postman and SoapUI. Curl can also be used for simple testing.

Postman provides options for setting the request type, headers, SSL certificates, body, and tests. This approach can be used for functions tied to API endpoints as event triggers, such as API Gateway. But if a function is triggered asynchronously by other services, we’ll need to rely on different tools to debug.

The AWS Console provides a useful option for testing Lambda functions directly without the involvement of an event source. It is very useful for checking real-time issues while deploying a Lambda for the first time.

First, it requires that test events be created. You can then run those events directly on the function while you are writing and building your application. These test events can emulate what is expected from any AWS service. Here’s an example used to create a thumbnail from an image whenever it gets uploaded to the S3 bucket:

When we hit the Test button, the output will be generated as follows:

We can further check metrics using the “Monitoring” tab. It has out-of-the-box metrics to check error rate, throttling and more, which can be very helpful for debugging issues. Detail logs can also be viewed by clicking on the “View logs in CloudWatch” button.

CloudWatch Logs is the main source of information on how an AWS application is behaving, and that applies to Lambda Functions, too.

Lambda Functions, by default, send log data to CloudWatch. CloudWatch then creates a LogGroup for each Lambda Function.

Inside a LogGroup, there will be multiple Log Streams collating the logs that get generated from a particular Lambda instance (Lambda concurrency allows multiple instances created for each Lambda function).

Log Streams contain log events, and clicking on each event will provide detailed information for a particular Lambda function invocation.

So, whenever a Lambda function has an issue, we now know the steps to follow in order to dig through the logs and see what’s happening for a particular invocation. Logs appearing in CloudWatch Logs are almost real-time so collect as many logs as you need to debug an issue.

That being said, there are a few caveats to using CloudWatch Logs:

• AWS charges based on the size of the log data being stored in it. So, it is recommended to remove unnecessary log statements once a bug is fixed. It may also increase the size of your Lambda function as well, which may cause a delay in loading and starting a function.
• Many enterprises work with multiple clouds and/or on-premise platforms, so would likely be using Splunk or ELK for log aggregation. CloudWatch, being very tightly coupled with AWS services, would not be preferred by those organizations. In that case, logs from CloudWatch Logs are streamed to these commercial tools through streaming services like Kinesis. These tools also have the benefit of providing a visualization layer by allowing you to create several dashboards to meet your needs.
• CloudWatch Logs can become costly over time as AWS charges more for higher storage requirements. ELK, being open-source, is preferred by many to save costs.

Here’s an example of a typical flow used to store and view logs by these enterprises:

Despite having rich features, CloudWatch Logs is limited to providing information for only one component – the Lambda Function. But, for a typical serverless flow, a function will call a variety of other AWS services.

What if we need to get more details around these integrations, such as how they’re performing and what failures are causing performance issues?

To help answer these questions, Amazon introduced AWS X-Ray, which helps developers analyze and debug distributed applications. X-Ray provides end-to-end tracing of requests and generates a map of application components. It can be used to analyze applications in development and in production.

So, as you can see, debugging an application on AWS cloud is possible to do using native tools like SAM, CloudWatch, and X-Ray, or open-source tools like Serverless Framework.

However, we will have to call upon multiple tools to gather all the information we need to get the job done — and it will involve a lot of tedious, manual work and code changes.

That’s where tools like Lumigo come in. For one thing, they’re much simpler to set up, and they provide a seamless experience for monitoring, tracing, and debugging not only Lambda functions but also AWS and third-party managed services.

When an issue occurs, a developer can quickly zoom in to see an individual invocation of a Lambda function in the context of end-to-end transaction flow.

We can then dig into the details for functions and services – including stack trace, inputs, outputs, parameters, environment variables, and correlated logs – in order to isolate the root cause.

Serverless has already shown its potential to revolutionize the way we build and deploy cloud applications. But while it solves a lot of problems for developers, particularly in the management of infrastructure, it introduces some new ones of its own, particularly in the areas of monitoring, debugging, testing, and tracing. That’s why we see cloud providers and commercial vendors focusing on building the tools to address these problems.

See Additional Guides on Key Performance Testing Topics

Together with our content partners, we have authored in-depth guides on several other topics that can also be useful as you explore the world of performance testing.

Lambda Performance

Authored by Lumigo

• AWS Lambda Cold Starts: Solving the Problem 
• AWS Lambda Concurrency 
• AWS Lambda Async 

Optimizing Python

Authored by Granulate

• Profiling in Python: Top Tools and 5 Tips for Success
• Optimizing Python: Why Python Is Slow & 4 Optimization Methods
• 5 Key Tips for Python Performance Optimization

Application Performance Monitoring

Authored by Granulate

• Mastering Performance Optimization – The Basic Metrics And What’S Wrong
• Application Performance Monitoring (APM): Metrics, Tools & Tips
• Optimizing Kafka Performance – Granulate