Lambda Events and Event Source Mapping: A Practical Guide

What Are Lambda Events? 

Lambda events are defined as JSON-formatted documents that trigger AWS Lambda functions to execute. These documents originate from other AWS services or external sources and serve as input data driving the function’s behavior. Lambda can handle various application scenarios such as real-time file processing, streaming data analysis, and backend server tasks.

Lambda events are processed asynchronously or in response to direct service requests. AWS Lambda maintains a queue to handle asynchronous events, ensuring each event is processed at least once. In the case of synchronous invocation, such as via API Gateway, the function is triggered directly and must respond in real-time.

What Is Lambda Event Source Mapping? 

A Lambda event source mapping is a feature within AWS Lambda that allows developers to connect an event source to a Lambda function directly from the AWS console without writing additional integration code. It defines how data from Amazon Kinesis, DynamoDB, or SQS is batched and delivered to a Lambda function. 

This connection is managed internally by AWS, providing a seamless data flow from the source to the function. Event source mappings simplify the architecture and manage the invocation and error handling between the event source and Lambda. They enable efficient handling of high-volume streams and database changes, adjusting polling automatically.

Related content: Read our guide to Lambda architecture

What Is Lambda Event Filtering?

Lambda event filtering is an AWS feature that allows developers to specify JSON-based filter criteria directly on the event source mapping. This reduces the need for filtering logic within the Lambda function, enhancing performance by executing the function only when relevant events occur. 

Filters can be set based on attributes present in the incoming event, ensuring that only pertinent data triggers the Lambda function. Event filtering enhances resource utilization and reduces costs by preventing unnecessary Lambda invocations. This is crucial in event-driven architectures, where only a subset of events require reaction.

Which Services Can AWS Lambda Read Events From?

Lambda can read data from the following services.

Amazon DynamoDB

DynamoDB streams integrate with AWS Lambda, automatically triggering functions for each item-level change in the database table. Lambda functions can interact with insert, modify, and remove events, handling data synchronization, replication, or triggering complex workflows. This ensures real-time processing of database changes for dynamic applications.

Additionally, the setup is managed through Lambda event source mappings, creating a direct connection between DynamoDB and Lambda without the need to manage the underlying infrastructure. This setup benefits developers by simplifying the data flow and decreasing the development time.

Amazon Kinesis

With Amazon Kinesis, AWS Lambda functions can process streaming data in real time by utilizing an event source mapping that binds a Kinesis data stream to a Lambda function. Once established, the function is triggered each time new data enters the stream, suitable for analytics, real-time monitoring, and other time-sensitive applications.

Lambda handles data as it arrives in batches, providing a scalable and manageable approach to streaming data. With built-in retry mechanics and error handling, this integration supports the capability to analyze and respond to data trends quickly.

Amazon MQ

Amazon MQ, a managed message broker service for ActiveMQ, can trigger AWS Lambda functions by pushing messages directly to them. This setup is appropriate for applications requiring cross-application message processing and integration with existing systems using traditional messaging models.

Messages in Amazon MQ queues or topics can be seamlessly redirected to Lambda functions through event source mapping. This makes it easier to implement decoupled microservices architectures while leveraging AWS’s scalability and serverless execution model.

Amazon MSK

Amazon Managed Streaming for Apache Kafka (Amazon MSK) allows Lambda to directly process messages from Kafka topics. This integration helps developers build applications that react to real-time data streams from various sources, including website clickstreams, financial transactions, and social media feeds.

Configuring a Lambda function to consume data from MSK involves setting up an event source mapping that specifies the Kafka topics and other parameters, such as the batch size. This provides a serverless way to process streaming data, eliminating the overhead of provisioning and managing servers.

Self-Managed Apache Kafka

For self-managed Apache Kafka clusters, AWS Lambda offers integration capabilities that allow the processing of data streams without managing the underlying compute resources. Lambda functions can be triggered by records on Kafka topics, enabling real-time processing and response.

This connection is handled through event source mappings, which define the relationship between the Kafka cluster and the Lambda function. The configuration includes details such as the VPC, security groups, and access policies, ensuring secure and efficient data handling.

Amazon SQS

AWS Lambda integrates with Amazon Simple Queue Service (SQS) for applications needing to process message queues. Lambda functions can be triggered by messages in SQS queues, suitable for transactional systems, batch processes, or workload decoupling.

Lambda directly pulls messages from the queue based on the source mapping configuration, processing them individually or in batches. This integration simplifies queue management and automatically scales with the volume of incoming messages, providing a solution for handling asynchronous workloads.

Amazon DocumentDB

Amazon DocumentDB’s integration with AWS Lambda allows the manipulation and response to JSON data stored within the DocumentDB environment. Lambda can react to changes or updates made within DocumentDB collections, enabling tasks such as data validation, transformation, and real-time analytics.

The setup between DocumentDB and Lambda is characterized by an event source mapping that directs data changes to the Lambda function. This minimizes the complexity and enhances the manageability of real-time database applications.

Tutorial: Managing an Event Source Mapping in Lambda 

Create an Event Source Mapping

Note: Before proceeding, please ensure that the Execution Role for the Lambda function has relevant execution permissions.

To create an event source mapping in AWS Lambda, add the necessary permissions to the Lambda function’s execution role. For some sources like Amazon SQS, you can use an AWS-managed policy that already includes the required permissions.

Once the permissions are set, you can proceed with creating the trigger:

1. Navigate to the Lambda console and open the Functions page.
2. Select the function you want to link to an event source.
3. Under the Function overview, click on Add trigger.
4. Choose the type of trigger you want to add.

Configure the trigger options according to your needs and click Add to finalize the creation of the event source mapping.

You can use the AWS CLI to create an event source mapping if you prefer the command line. Here’s how you can map a Lambda function to a DynamoDB stream:

aws lambda create-event-source-mapping --function-name my-function --batch-size 1000 --maximum-batching-window-in-seconds 10 --starting-position LATEST --event-source-arn arn:aws:dynamodb:us-east-2:123456789012:table/my-table/stream/2024-04-12T15:28:14.526

The output should look like this:

This command creates a new mapping with a batch size of 1000 and a maximum batching window of 10 seconds.

Update an Event Source Mapping

To update the event source mapping via the console:

1. Open the Lambda console and go to the Functions page.
2. Select the function associated with the event source mapping.
3. Click on Configuration and then on Triggers.
4. Select the trigger you want to edit and click on Edit.

Alternatively, you can update the event source mapping using the AWS CLI. For example, to configure maximum concurrency for an Amazon SQS source, you would use:

aws lambda update-event-source-mapping --uuid "692e257c-7966-4928-b5c8-f05ed091057a" --parallelization-factor 10

This command adjusts the maximum concurrency settings for your event source mapping.

Delete an Event Source Mapping

To delete a mapping via the Lambda console:

1. Go to the Event source mappings page of the Lambda console.
2. Select the mappings you wish to delete.
3. In the Delete event source mappings box, type “delete” and click Delete.

For CLI users, the deletion can be performed with the following command:

aws lambda delete-event-source-mapping --uuid a1b2c3d4-5678-90ab-cdef-11111EXAMPLE

The output should look like this:

This command deletes the specified event source mapping, removing the link between your event source and the Lambda function.

AWS Lambda Observability, Debugging, and Performance Made Easy with Lumigo

Lumigo is a serverless monitoring platform that lets developers effortlessly find Lambda cold starts, understand their impact, and fix them.

Lumigo can help you:

• Solve cold starts – easily obtain cold start-related metrics for your Lambda functions, including cold start % and average cold duration, and enable provisioned concurrency. Generate real-time alerts on cold starts, so you’ll know instantly when a function is under-provisioned and can adjust provisioned concurrency.
• Find and fix issues in seconds with visual debugging – Lumigo builds a virtual stack trace of all services participating in the transaction. Everything is displayed in a visual map that can be searched and filtered.
• Automatic distributed tracing – with one click and no manual code changes, Lumigo visualizes your entire environment, including your Lambdas, other AWS services, and every API call and external SaaS service.
• Identify and remove performance bottlenecks – see the end-to-end execution duration of each service and which services run sequentially and in parallel. Lumigo automatically identifies your worst latency offenders, including AWS Lambda cold starts.
• Serverless-specific smart alerts – using machine learning, Lumigo’s predictive analytics identifies issues before they impact application performance or costs, including alerts about AWS Lambda cold starts.

Get a free account with Lumigo to resolve Lambda issues in seconds.