AWS Lambda is a widely used cloud computing service that allows developers to run code in a serverless cloud. It eliminates the need to provision or manage servers. The Relational Database Service (RDS) is a popular cloud service that provides managed SQL databases. RDS is server-based, so it requires provisioning and managing a server.
AWS Lambda and RDS can theoretically work together, but it is not straightforward to make them connect seamlessly. The different architectures (serverless vs. server-based) mean they have different requirements and need to be carefully configured.
This is part of our series of articles about AWS Lambda deployment.
In this article
To create the execution role needed for granting permissions:
This tutorial uses an Amazon RDS database instance. The Lambda function will facilitate access to this instance while in the Amazon VPC. The Lambda function will create a table called demoEmployee,insert some records, and retrieve them. The schema of the tables is as follows:
To add records to the Employee table:
aws rds create-db-instance --db-name ExampleDB --engine MySQL \ --db-instance-identifier demoDatabase --backup-retention-period 3 \ --db-instance-class db.t2.micro --allocated-storage 5 --no-publicly-accessible \ --master-username demoUsername --master-user-password demoPassword
Keep the username, password, and the database’s name safe. Also note down the database instance’s endpoint from the RDS console.
2. Use the following code in a file named app.py to create a table in the ExampleDB database, adds records, and retrieve it:
import sys import logging import rds_config import pymysql demo_rds_host = rds_enpoint demo_db_username = rds_config.db_username demo_db_password = rds_config.db_password demo_db_name = rds_config.db_name logger = logging.getLogger() logger.setLevel(logging.INFO) try: conn = pymysql.connect(host=demo_rds_host, user=demo_db_username, passwd=demo_db_username, db=demo_db_name, connect_timeout=5) except pymysql.MySQLError as e: logger.error("Connection to MySQL database unsuccessful") logger.error(e) sys.exit() logger.info("Successful to MySQL RDS instance") def demo_handler(event, context): item_count = 0 with conn.cursor() as cur: cur.execute("create table demoEmployee ( demoEmployeeID int NOT NULL, Name varchar(255) NOT NULL, PRIMARY KEY (EmpID))") cur.execute('insert into demoEmployee (demoEmployeeID, Name) values(1, "Jack")') cur.execute('insert into demoEmployee (demoEmployeeID, Name) values(2, "Jill")') cur.execute('insert into demoEmployee (demoEmployeeID, Name) values(3, "Abbott")') conn.commit() cur.execute("select * from demoEmployee") for row in cur: item_count += 1 logger.info(row) conn.commit() return “%d items added to the table" %(item_count)
3. Store the connection information for the Lambda function in a file named rds_config.py:
db_username = "demoUsername" db_password = "demoPassword" db_name = "ExampleDB"
4. Zip the file with function code, config, and dependency information. Here, the code has just one dependency, the pymysql library, for accessing the MySQL instance. Name the zip file as demoApp.zip.
To create and test the Lambda function:
aws lambda create-function --function-name demoLambdaFunction --runtime python3.8 \ --zip-file fileb://demoApp.zip --handler app.handler \ --role arn:aws:iam::123456789012:role/lambda-demo-role \ --vpc-config SubnetIds=subnet-0532bb6758ce7c71f,subnet-d6b7fda068036e11f,SecurityGroupIds=sg-0897d5f549934c2fb
2. Once it’s created, use the following command to invoke the function:
aws lambda invoke --function-name demoLambdaFunction output.txt
If the function is successful, it will generate the output.txt. Go through this file, results in the AWS Lambda console, and logs on CloudWatch logs to ensure the function ran successfully.
Amazon RDS Proxy can act as an intermediary between your application and an RDS database. It establishes and manages the connection pools to your database so that an application creates fewer database connections. This is especially important for Lambda functions, because they can create a large number of connections in a short period of time, overwhelming your RDS database.
Lambda functions interact with RDS Proxy instead of directly with an RDS database instance. RDS Proxy handles the connection pooling for many simultaneous connections created by Lambda functions. This allows Lambda applications to reuse existing connections, avoiding the need to create a new connection for each Lambda invocation. Let’s see how it works.
You can use an existing database (it must be Aurora MySQL or Amazon RDS MySQL). Store the database credentials as secrets using AWS Secrets Manager. Create an IAM policy to enable RDS Proxy to read the secrets.
To create a secret in Secrets Manager:
Source for this and the following images: AWS
5. Enter a name for the secret and select Next.
6. Accept all default settings and select Store. Remember Amazon Resource Name (ARN) assigned to the secret – it will be necessary later.
7. Create a new IAM role allowing RDS Proxy to read the new secret. RDS Proxy will use the secret to connect to the relevant database. Next, go to the IAM console to create a new IAM role. Attach a policy to provide Secrets Manager permissions to your newly created secret. You can see an example of policies for this and the next step in this Amazon blog post.
8. Add a trust policy to enable RDS to assume the new IAM role. Save this role and remember the IAM role’s ARN for later.
You can then use the Lambda console to add a database proxy to your chosen Lambda function:
4. After several minutes, the RDS Proxy will be ready, and the status update will become Available.
5. Select the proxy to view your details. Remember the proxy endpoint for later use in your Lambda function’s code.
Once your Lambda function has permission to use your newly configured proxy, you can connect to RDS Proxy.
You can connect to an RDS proxy endpoint rather than directly to an RDS instance. There are two security options when doing this: IAM authentication or the native database credentials from Secrets Manager. IAM authentication is the better option because it eliminates the need to read or embed credentials in the Lambda function’s code.
For example, in Node.js, you can perform the following steps to use the RDS proxy from your application:
You can skip this step if you connect to the proxy using native database credentials. To check if your RDS proxy uses IAM authentication:
Here are some of the challenges associated with connecting a Lambda function to an RDS instance.
An RDS instance usually runs in a subnet in your VPC. This approach ensures security by keeping the RDS instance isolated, but it also prevents Lambda functions from accessing it.
Conversely, a Lambda function runs in an isolated VPC managed by AWS, accessible only to the Lambda service. It can access the Internet and call any AWS service or publicly available third-party resource, but it cannot access a private resource like an RDS instance.
You must establish a connection between the Lambda and RDS VPCs, specifying the subnet/VPC configuration. There must be an elastic network interface (ENI) for each Lambda execution environment to prevent performance issues—creating and attaching this interface takes time and slows down the function’s cold start.
Another issue with Lambda is its scalability through different execution environments. Because you have to create dedicated ENIs for each environment, you can quickly reach your account’s ENI assignment limit.
AWS offers a VPC networking option for Lambda that leverages a networking virtualization system called Hyperlane to connect VPCs. Hyperlane creates a single VPC NAT instance when you create a function or change the VPC configurations, so you don’t need to create additional ENIs.
This update accelerates Lambda cold starts, minimizing the impact of connecting to private resources like an RDS instance. However, you still create the Hyperlane-managed ENI inside the VPC using the subnet’s IP address. Alternatively, you can use provisioned concurrency to prevent cold starts.
Another potential issue when connecting Lambda functions to RDS involves a large number of DB connections. A standard application has a small number of connections to a database instance, stored in a shared pool for all requests—the active connections don’t exceed the specified limit even during heavy workloads.
However, the isolation of Lambda execution environments prevents them from sharing data, requiring a dedicated DB connection for each function to connect to RDS. During heavy workloads, Lambda spawns thousands of functions with individual DB connections. The short lifecycle of Lambda functions results in constantly opening and closing the connections.
A standard SQL database engine cannot handle these workloads, impacting stability. Typically, you would mitigate this issue by limiting Lambda’s concurrency. A better approach is to use a proxy server to manage the connections. AWS offers a managed RDS Proxy service to help address this issue.
Lambda functions can connect to RDS instances via RDS Proxy rather than establishing direct connections. RDS Proxy manages connections and deals with Lambda request spikes—it queues requests or rejects them if they exceed the DB connection limit. It can result in rejected requests but improves the application’s resiliency.
RDS Proxy is transparent to the application code. During database failures, it can switch to new RDS instances to maintain connectivity. RDS Proxy supports authorization with IAM roles instead of DB credentials, reducing the risk of leaked credentials.
Lumigo is a cloud native observability tool, purpose-built for distributed systems like serverless applications and supports the suite of AWS managed serverless services, including Lambda and RDS. Lumigo connects Lambda invocations through asynchronous event sources and can trace requests to RDS in your transactions. With Lumigo, you can capture the request and response to RDS, as well as other services like DynamoDB, SNS, SQS and Kinesis, making debugging complex serverless applications easy.