Guide Content

Guide Content

An Introduction to the AWS Serverless Ecosystem

With the rapid growth of serverless development in recent years, the services available to developers have grown in number and scope. The best example of this is the AWS serverless ecosystem, which now encompasses dozens of services.

In this article

In an environment as vast and confusing as serverless, we at Lumigo wanted to provide a guide to this environment, shining a light into the corners to see what we can find. We’ll get our bearings in the larger AWS ecosystem by evaluating each of the services, starting with the core Lambda functionality, and expanding outwards to AWS Lambda monitoring and function trigger sources. Once we’re done, you’ll have a clear map of which services lie where, and how AWS can better help you on your serverless journey.

This is part of an extensive series of guides about managed services.

The core: AWS Lambda and related tools

AWS Lambda Icon Logo Serverless computing was defined with the launch of AWS Lambda in 2014 at re:Invent, AWS’s annual developer conference. Providing developers with the ability to run custom functionality independently of the need for infrastructure proved a powerful tool in redefining traditional microservice applications. The ability to run code on-demand without a dedicated container or server allows your development team to focus on rapid feature delivery. It reduces your overall time-to-market by bypassing the traditional infrastructure management layer of a microservice architecture.

Building off the wild success of serverless development with Lambda, AWS introduced Lambda@Edge. This service, a feature of Amazon CloudFront, is designed to shorten the round-trip time between your custom functionality and the calling application. This opened up AWS Lambda to an entirely new tier of responsibility, easing localization implementation and allowing true flexibility in internationalization by running as close to the user as possible. These Lambda functions, which run at the “edge” of the network on the CloudFront CDN, can respond to user interface requests very quickly, letting you perform tasks such as internationalization without the need for a dedicated load balancer.

AWS Lambda provides a powerful tool for building applications without the need for infrastructure. However, the mechanical process of deploying Lambda functions can be heavily resource- and time-dependent, as the process relies on the physical copying of code files. This practice can lead to extensive code duplication, particularly when working with serverless applications of any significant complexity level. AWS Lambda Layers reduce this complexity by allowing you to create common elements that can be quickly incorporated into your Lambda functions. The code and configuration sharing allowed by Lambda Layers can be a crucial factor in reducing Lambda function complexity.

AWS Step Functions Icon Logo State machines are a frequent feature of multi-page user workflows, requiring clear functionality that operates in a specific sequence with configurable outputs. AWS recognized this need as a common pattern and presented Step Functions as the solution. AWS Step Functions allow you to manage the deployment and usage of your lambda functions via configurable state machines. These can manage application state, a traditional problem in serverless applications where the individual components are intended to be atomic. Using the powerful user interface in the AWS dashboard, you can easily configure your step function-driven state machines using a convenient function designer, or dive deep with the Amazon States Language.

Learn more about AWS Step Functions limits, use cases, and best practices.

Event-driven solutions

As an on-demand infrastructure, AWS Lambda operates with an event-driven architecture. These events originate in a number of different places, but generally are dependent on either a notification of activity or the modification of data in a system store. In this section, we’ll give a high-level overview of these triggers by breaking them down into three categories: data-driven events, notification-driven events, and other types of events that don’t fall into a neat category.

Data-driven events

Data-driven events are provided by sources that predominantly operate on user data. This data can be a cached set of application parameters, a database being updated, or a new file on S3 – anything that modifies user data through these services can result in a trigger event being generated for AWS Lambda. Below we’ll cover five of the more popular data-driven event sources that integrate with Lambda.

AMazon RDS Icon Logo The first is the Relational Database Service, or Amazon RDS. This event source reacts to activity in a database hosted on Amazon RDS. The RDS service generates a notification on the Simple Notification Service (more on this later), which can then be configured to call a custom AWS Lambda function. This allows you to create fully customized workflows based upon activity in your relational databases.

Amazon DynamoDB icon Logo The ability to drive activity based on low-level data modification is not restricted to relational databases. Amazon DynamoDB offers a NoSQL-driven database system that can generate AWS Lambda events based on data activity. This activity is captured in DynamoDB Streams, which represents a time-ordered sequence of modifications presented at the item-level for any DynamoDB table. This information, which is stored additionally in a log for up to 24 hours, can capture the entire transition state of the data it represents. By linking your DynamoDB Streams with AWS Lambda, you can create dynamic workflows that trigger off of activity in your NoSQL-based data.

Next in the list of data-driven event sources is the Amazon Simple Storage Service, or Amazon S3. This high-capacity, high-availability file storage solution is crucial for providing a scalable storage approach that can handle web traffic at all levels. Amazon S3 allows you to dynamically integrate with AWS Lambda via event generation upon the creation and deletion of resources in a given S3 bucket. This can be used to create complex file modification flows, such as video transcoders and log transformers.

AMazon Kinesis Data Streams Icon Logo When you’re looking to capture gigabytes of data from thousands of sources every second, you’ll want to use a high-volume real-time data streaming service like Amazon Kinesis Data Streams, or Amazon KDS. The high scalability of Amazon KDS enables real-time analytics on user behavior and other activity streams, letting you process data that is collected in milliseconds and spread out to numerous dashboards, fraud detection systems, and more. AWS Lambda can plug into this flow synchronously, allowing you to react dynamically to the critical stream of data from your configured sources. This allows you to read and process records in batches, collating, and dispatching the stream data as appropriate.

Amazon ElastiCache Icon Logo The last data-driven event source we’ll mention is Amazon ElastiCache. Amazon ElastiCache provides you with a scalable in-memory data storage solution that is easy to set up, configurable, and runs without a hitch. This allows you to institute caching throughout your application with ease, as ElastiCache is purpose-built to provide analytics and retrieval on your data as quickly as possible. AWS Lambda can run in response to events in your ElastiCache system, allowing you to quickly react to cache events as needed.

While the above list seems long, we did not touch on every AWS service that can generate data-driven events for AWS Lambda. The set of AWS features and capabilities grows daily, so we felt it was important to stick to the few well-known services that can have an impact on serverless development from day one. With careful application of the above data stores, you can confidently respond to user activity as it happens.

Notification-driven

Next, we take a look at notification-driven event generating services for AWS Lambda. While these services may deal with significant amounts of data, their primary purpose is often the simple routing of information from one system to another. We’ll take a look at the four most popular of these notification-driven event sources, and see what they have to bring to the AWS Lambda serverless ecosystem.

Amazon API Gateway Icon Logo The first service we’ll look at is Amazon API Gateway. API Gateway lets you build an HTTP application by providing a nascent HTTP server application with a generalized translation layer. In essence, this tool lets you stand up a URL that you can hit to independently invoke your AWS Lambda functions. This tool relies on heavy usage of architectural design when building your serverless application, and in many cases, API Gateway will comprise the majority of the entry points to your user-centric web applications driven by AWS Lambda.

Amazon-Simple-Email-Service-SES Icon Logo Moving on from HTTP-driven behavior, we take a look at SMTP triggers with Amazon Simple Email Service, or Amazon SES. This integration once again leverages the power of the Amazon Simple Notification Service, which we’ll cover next. AWS Lambda can integrate with your Amazon SES-based email receipt system, allowing you to trigger dynamic functionality whenever you receive a message through Amazon SES.

Amazon-Simple-Notification-Service-SNS Icon Logo Amazon’s Simple Notification Service, or Amazon SNS, is a messaging service with a focus on high availability, durability, and security. It is a fully managed pub/sub messaging service that provides high-throughput many-to-many messaging. Amazon SNS integrates with AWS Lambda at the message level, letting you set AWS Lambda functions as targets for specific notification messages. This lets you build custom asynchronous event-driven pipelines that can be used for anything from analyzing user data to reporting metrics on application behavior.

Amazon-Simple-Queue-Service-SQS Icon Logo Wrapping up the more popular notification-based event sources is Amazon’s Simple Queue Service or Amazon SQS. This messaging service allows you to create dedicated queues for transmitting information throughout your application architecture. This removes the complexity and overhead of custom-built systems and allows your developers to focus on business value instead of pure technical minutiae. AWS Lambda can be a consumer of any queue in Amazon SQS, letting you build your application’s state flow the way you want it to be.

While these are not the only notification-based event sources available to serverless developers, most applications of significant size will end up using one of these services at some point. Knowing how you can leverage the asynchronous messaging inherent in a notification-based pattern can help you get a better handle on your application’s scalability, building comfort with high-availability and high-throughput systems that are invaluable as your organization grows.

Other AWS Serverless Ecosystem Services

The final set of event triggers we’ll discuss don’t have a straightforward categorization – more accurately, there weren’t enough tools in each category to really justify breaking them out into their own subsections. These tools are no less powerful for the omission, however, we simply did not want the article to stretch into novel-length. Below we’ll cover three additional event sources: Amazon EC2, Amazon Lex, and Alexa.

Amazon EC2 Icon Logo Amazon Elastic Compute Cloud, or Amazon EC2, is a service dedicated to providing cloud-based compute capacity on-demand. With the goal of making web-scale cloud computing easier for developers, EC2 has a simple web service interface that removes most of the friction from obtaining and configuring compute capacity for your application. AWS Lambda can process lifecycle events from Amazon EC2, letting you dynamically manage compute resources based upon CloudWatch events you receive from the EC2 service as compute capacity is adjusted.

Amazon Lex Icon Logo The next service, Amazon Lex, is unique in that while it is a form of notification-driven event generation, the general goal of the interface is to create conversation trees similar to the step function definitions we explored before. Amazon Lex bots can be configured with AWS Lambda to perform validation of the conversational content, fulfillment of the query, or both. With the clever application of AWS Lambda functions, you can make your user interface bots as efficient and accurate as possible.

Alexa, Amazon’s Echo-based voice assistant, allows for extensive customization through the creation of dedicated skills. These skills can be based around AWS Lambda functions, which can be called based upon the voice input received from your application’s users. With AWS Lambda and Alexa, you can easily create voice interfaces for your applications that wow your users and sound great to potential investors.

While the list of event sources above is extensive, it is by no means exhaustive. AWS continues to expand the list of potential AWS Lambda event triggers, adding new services to the list every day.

Serverless Application Support

The next set of tools we’ll cover are resource and scaling solutions that can be used for any application based in the AWS ecosystem. These services also have deep ties into AWS Lambda, and can be customized using serverless functions as needed. In this section we’ll cover AWS AppSync, Amazon Aurora Serverless, and AWS Fargate, critical tools in building scalable user-driven applications.

AWS Appsync is a tool that helps developers build data-driven scalable applications. Providing developers the ability to create a flexible API out of a collection of data sources, AppSync lets you build real-time applications that scale with your needs, providing instantaneous updates from your data pipeline as they happen. Building on this flexibility, AppSync allows for AWS Lambda functions to serve as an event source, letting you integrate AppSync easily with your serverless data processing pipeline.

Amazon-Aurora Serverless Icon Logo The scalability of your database infrastructure is as important, if not more so than the scalability of your application infrastructure. Amazon Aurora Serverless provides an on-demand auto-scaling configuration for your Amazon Aurora instances. AWS Aurora Serverless scales your database as needed in response to traffic increases, and offers seamless integration with AWS AppSync to provide fully-powered data pipelines driven by your AWS Lambda functions.

AWS-Fargate Icon Logo When working with container-based systems, AWS Fargate is a critical component in being able to handle application demand as it scales. AWS Fargate is a container engine, providing scalable infrastructure for applications built in AWS Elastic Container Service (ECS) and Elastic Kubernetes Service (EKS). Fargate automates maintaining container infrastructure, allowing you to focus on application functionality instead of container scaling and related infrastructure.

Monitoring, Observability, and General Support

Once you have written your Lambda functions and configured their triggers, the real work of maintaining your application begins. AWS offers several tools that help you develop a true sense of the health of your application, giving you the monitoring and observability you need to have confidence in your serverless stack. We’ll cover four of the more popular support tools available in AWS below.

Amazon-CloudWatch Icon Logo The first, and most important, is Amazon CloudWatch. CloudWatch is a service dedicated to monitoring and observability and integrates natively with your AWS Lambda functions as they operate. By default, all of your application’s debugging and trace output will end up as an artifact in either CloudWatch Logs or CloudWatch metrics. Collating and organizing your CloudWatch logs can prove a daunting challenge at times, but if you can work through the morass you will have all the data you need on the health of your application.

AWS-CodePipeline Icon Logo Next is AWS CodePipeline. AWS CodePipeline allows you to create continuous delivery pipelines for any applications that run on AWS. This means that you can not only use CodePipeline to deploy your serverless applications, but you can also invoke the same functions that drive your dynamic infrastructure, letting you build dynamic deployment characteristics into your development pipeline.

AWS X-Ray is another pillar of our serverless observability suite. AWS X-Ray is a configurable distributed tracing platform that you can use to get a real-time picture of your serverless application’s operation. Through its integration with the general AWS platform, X-Ray receives trace messages from your AWS Lambda functions automatically. This lets you build a AWS-X-Ray Icon Logo visualization of your system that you can use to drive refinement and optimization as performance degrades. The comprehensive view that X-Ray can provide is invaluable, allowing you to build a quick picture of your serverless architecture at a glance.

The final tool we’ll discuss is the AWS Serverless Application Model, or AWS SAM. This is not a technological solution, but rather a templating engine and organization tool that you can use to turn your collection of disparate Lambda functions into a cohesive application. AWS SAM features a set of command-line tools that let you automate your testing and deployment, letting you get to a continual delivery state more quickly than would otherwise be feasible.

Closing thoughts

The AWS serverless landscape is constantly evolving. As the technology continues to move forward and drive improvements to cold-start times, response times, and service integrations, the capabilities available to serverless developers grow. Expansions in capabilities of existing tools, integrations with additional services, and additional platform refinements continue to make serverless development with AWS Lambda a strong choice. However, developing in AWS Lambda alone is not without its pitfalls. Tools like Lumigo can help you to close the gaps in your serverless maintainability and observability plans, giving you the confidence you need in your application to deploy new features rapidly.

Learn More About the AWS Serverless Ecosystem

There’s a lot more to learn about AWS serverless. To continue your research, take a look at the rest of our blogs on this topic:

AWS Step Functions – Limits, Use Cases, Best Practices

AWS Step Functions is an orchestrator that helps developers design and implement complex workflows. When you need to build a workflow, or have multiple tasks that need orchestration, Step Functions coordinates between those tasks. This makes it simple to build multi-step systems.

What is AWS X-Ray?

AWS X-Ray enables developers to debug distributed microservice-based applications and conduct performance analysis. X-Ray consolidates information about an application’s architecture into a Service Map, which also includes connections to components and a dependency tree. It enables easier root cause analysis by highlighting problematic parts of an application.

AWS AppSync: Five Reasons You Should Consider It Over API Gateway

In AppSync, AWS has a GraphQL-as-a-Service offering that makes it easy to build scalable and resilient GraphQL APIs in the cloud. You don’t need to run any servers yourself. Simply configure your GraphQL resolvers, and you have a GraphQL API that can scale to millions of users and offers multi-AZ redundancy out-of-the-box

See Additional Guides on Key Managed Services Topics

Together with our content partners, we have authored in-depth guides on several other topics that can also be useful as you explore the world of managed services.