Automating Serverless Tasks with the Lumigo CLI

Here at Lumigo, our mission is to help customers succeed with serverless by solving the observability problem and letting them focus on adding business value with serverless technologies. And to do that, we love to eat our own dogfood – be it using the same serverless technologies that our customers use and even using Lumigo to monitor Lumigo itself! That way, we feel your pain and we find solutions to problems that you care about.

When you’re working with AWS, and by extension, serverless technologies on AWS such as Lambda, EventBridge and SQS, there are many boring manual tasks. For example, how do you replay messages from a SQS dead-letter queue (DLQ) back to the main queue? Or how to peek at events that are captured by an EventBridge bus or rule?

We created the Lumigo CLI, a free command-line tool that will automate these boring tasks so you can focus on building the things that make your customers happy!

In this post, we will cover some of the most popular commands available in the Lumigo CLI.

Right-size Lambda memory setting

When it comes to Lambda functions, you have a single dial for controlling a function’s power (in terms of CPU and network bandwidth) and cost per ms of execution time. More memory equals more power, but also high cost per ms of execution time.

For CPU intensive workloads, you might want to allocate more memory to boost the function’s performance. But if your function spends most of its time waiting for DynamoDB operations to complete, then giving it more power is not likely going to do you any good.

To help you right-size the memory setting for your workload, you can use the command lumigo-cli powertune-lambda.

This command utilizes Alex Casalboni’s excellent aws-lambda-power-tuning state machine and manages the deployment and update of the state machine for you. It also makes it easy for you to right-size the function as part of your CI/CD pipeline so you can keep your function optimized at all times. See this blog post for more detail.

Tail EventBridge/Kinesis/DynamoDB/SNS/SQS

A common challenge with building event-driven systems is that there is no easy way to peek at the events that are captured. Services such as EventBridge, Kinesis Data Streams, DynamoDB Streams and SNS don’t have any built-in APIs for tailing them.

This is such a common problem that we often find customers create their own workarounds. So we added a number of tail commands that lets you watch events as they flow into these services:

• lumigo-cli tail-dynamodb to tail a DynamoDB Stream
• lumigo-cli tail-kinesis to tail a Kinesis Data Stream
• lumigo-cli tail-eventbridge-bus to tail an EventBridge bus
• lumigo-cli tail-eventbridge-rule to tail a specific EventBridge rule
• lumigo-cli tail-sns to tail an SNS topic
• lumigo-cli tail-sqs to tail an SQS queue

Replay SQS DLQ messages

When you use SQS, it’s recommended that you set up a redrive policy to capture poison messages in a dead-letter queue. However, how you reprocess these DLQ messages is a problem that you have to solve yourself!

Often times, DLQs end up capturing messages that can’t be processed because of temporary problems as opposed to true poison messages. Maybe a downstream system is experiencing elevated error rate or latency, or maybe a bug was introduced to your processing logic. In these cases, the dead-letter messages can be safely replayed once the underlying problem was addressed.

At that point, you can just run the lumigo-cli replay-sqs-dlq command to replay those DLQ messages to an SQS queue, an SNS topic, or a Kinesis Data Stream.

Often times, we want to replay the DLQ messages to an SQS queue in a staging environment first before we replay them in production. The lumigo-cli replay-sqs-dlq command lets you replay messages to a different region and/or AWS account, and you have to option to keep the messages in the DLQ so you replay them in production once you have verified your fix in staging.

Switching between AWS profiles

The AWS CLI lets you configure multiple profiles and you can use the AWS_PROFILE environment variable to choose which profile to use. However, when you’re dealing with many AWS accounts, switching between all these different profiles is definitely a chore of the “death by a thousand cuts” variety!

You can use the lumigo-cli switch-profile command to quickly switch to another AWS profile, and use the lumigo-cli whoami command to see which profile you’re currently using.

And much more!

In this post, we have mentioned some of the most popular lumigo-cli commands, but there are much more where they came from. Amongst others, there are also commands to:

• Send data from local files to SNS/SQS
• Measure Lambda cold start durations
• Run S3 select against a batch of objects
• List Lambda functions in all regions
• Analyze lambda cost and cold starts
• List Kinesis streams and shards
• Wipe an account (super useful when you have playground or workshop accounts!)

If you have ideas for other commands then please let us know in our github repo!

See the lumigo-cli in action!

Seeing is believing. If you want to see the Lumigo CLI in action view this webinar I ran on Feb 16, 2021 on YouTube: