Lambda and RDS: Connecting Lambda Functions to RDS Instances

What Are AWS Lambda and RDS?

AWS Lambda is a widely used cloud computing service that allows developers to run code in a serverless cloud. It eliminates the need to provision or manage servers. The Relational Database Service (RDS) is a popular cloud service that provides managed SQL databases. RDS is server-based, so it requires provisioning and managing a server. 

AWS Lambda and RDS can theoretically work together, but it is not straightforward to make them connect seamlessly. The different architectures (serverless vs. server-based) mean they have different requirements and need to be carefully configured. 

This is part of our series of articles about AWS Lambda deployment.

Configuring a Lambda Function to Access Amazon RDS in an Amazon VPC

Create Role to Give Permissions

To create the execution role needed for granting permissions:

1. Go to the IAM console on the AWS management console and click Roles on the left sidebar.
2. Click the Choose role button.
3. From the options under the Select type of trusted entity section, choose AWS Service, and choose Lambda under Choose the service that will use the role.
4. Give lambda-demo-role as the role name. 

Create Database Instance

This tutorial uses an Amazon RDS database instance. The Lambda function will facilitate access to this instance while in the Amazon VPC. The Lambda function will create a table called demoEmployee,insert some records, and retrieve them. The schema of the tables is as follows:

demoEmployee(demoEmployeeID, employeeName)

Here,

demoEmployeeID

To add records to the Employee table:

1. In your default VPC, launch an RDS MySQL with the ExampleDB database with the following command on the AWS CLI:

aws rds create-db-instance --db-name ExampleDB --engine MySQL \

--db-instance-identifier demoDatabase --backup-retention-period 3 \

--db-instance-class db.t2.micro --allocated-storage 5 --no-publicly-accessible \

--master-username demoUsername --master-user-password demoPassword

Keep the username, password, and the database’s name safe. Also note down the database instance’s endpoint from the RDS console.

2. Use the following code in a file named app.py to create a table in the ExampleDB database, adds records, and retrieve it:

import sys
import logging
import rds_config
import pymysql
demo_rds_host  = rds_enpoint
demo_db_username = rds_config.db_username
demo_db_password = rds_config.db_password
demo_db_name = rds_config.db_name

logger = logging.getLogger()
logger.setLevel(logging.INFO)

try:
    conn = pymysql.connect(host=demo_rds_host, user=demo_db_username, passwd=demo_db_username, db=demo_db_name, connect_timeout=5)
except pymysql.MySQLError as e:
    logger.error("Connection to MySQL database unsuccessful")
    logger.error(e)
    sys.exit()

logger.info("Successful to MySQL RDS instance")
def demo_handler(event, context):
    item_count = 0

    with conn.cursor() as cur:
        cur.execute("create table demoEmployee ( demoEmployeeID  int NOT NULL, Name varchar(255) NOT NULL, PRIMARY KEY (EmpID))")
        cur.execute('insert into demoEmployee (demoEmployeeID, Name) values(1, "Jack")')
        cur.execute('insert into demoEmployee (demoEmployeeID, Name) values(2, "Jill")')
        cur.execute('insert into demoEmployee (demoEmployeeID, Name) values(3, "Abbott")')
        conn.commit()
        cur.execute("select * from demoEmployee")
        for row in cur:
            item_count += 1
            logger.info(row)
    conn.commit()

    return “%d items added to the table" %(item_count)

3. Store the connection information for the Lambda function in a file named rds_config.py:

db_username = "demoUsername"

db_password = "demoPassword"

db_name = "ExampleDB"

4. Zip the file with function code, config, and dependency information. Here, the code has just one dependency, the pymysql library, for accessing the MySQL instance. Name the zip file as demoApp.zip.

Create and Test Lambda Function

To create and test the Lambda function:

1. Use the following command to generate the Lambda function:

aws lambda create-function --function-name  demoLambdaFunction --runtime python3.8 \
--zip-file fileb://demoApp.zip --handler app.handler \
--role arn:aws:iam::123456789012:role/lambda-demo-role \
--vpc-config SubnetIds=subnet-0532bb6758ce7c71f,subnet-d6b7fda068036e11f,SecurityGroupIds=sg-0897d5f549934c2fb

2. Once it’s created, use the following command to invoke the function:

aws lambda invoke --function-name demoLambdaFunction output.txt

If the function is successful, it will generate the output.txt. Go through this file, results in the AWS Lambda console, and logs on CloudWatch logs to ensure the function ran successfully.

Using Amazon RDS Proxy with AWS Lambda

Amazon RDS Proxy can act as an intermediary between your application and an RDS database. It establishes and manages the connection pools to your database so that an application creates fewer database connections. This is especially important for Lambda functions, because they can create a large number of connections in a short period of time, overwhelming your RDS database.

Lambda functions interact with RDS Proxy instead of directly with an RDS database instance. RDS Proxy handles the connection pooling for many simultaneous connections created by Lambda functions. This allows Lambda applications to reuse existing connections, avoiding the need to create a new connection for each Lambda invocation. Let’s see how it works.

Prerequisites

You can use an existing database (it must be Aurora MySQL or Amazon RDS MySQL). Store the database credentials as secrets using AWS Secrets Manager. Create an IAM policy to enable RDS Proxy to read the secrets.

To create a secret in Secrets Manager:

1. Go to AWS Secrets Manager and select Store a new secret.
2. Under the secret type, select Credentials for RDS database.
3. Enter your username and password.
4. Choose the RDS database to which the secret relates. Select Next.

Source for this and the following images: AWS

5. Enter a name for the secret and select Next.

6. Accept all default settings and select Store. Remember Amazon Resource Name (ARN) assigned to the secret – it will be necessary later.

7. Create a new IAM role allowing RDS Proxy to read the new secret. RDS Proxy will use the secret to connect to the relevant database. Next, go to the IAM console to create a new IAM role. Attach a policy to provide Secrets Manager permissions to your newly created secret. You can see an example of policies for this and the next step in this Amazon blog post.

8. Add a trust policy to enable RDS to assume the new IAM role. Save this role and remember the IAM role’s ARN for later. 

Creating and Attaching a Proxy to a Lambda Function

You can then use the Lambda console to add a database proxy to your chosen Lambda function:

1. Go to the AWS Lambda console and select the Lambda function you want to enable RDS Proxy. The Lambda function must have a configuration that allows access to the same subnets and VPC as the RDS database. 
2. Go to the bottom of the Lambda configuration page and select Add database proxy.
3. Go through the database proxy adding wizard, enter the proxy identifier in the relevant field, and select the relevant RDS database. Select your Secrets Manager secret and IAM role (created in the previous step). RDS Proxy will use the secret to connect to the database. Select Add.

4. After several minutes, the RDS Proxy will be ready, and the status update will become Available.

5. Select the proxy to view your details. Remember the proxy endpoint for later use in your Lambda function’s code.

Once your Lambda function has permission to use your newly configured proxy, you can connect to RDS Proxy.

Using the RDS Proxy

You can connect to an RDS proxy endpoint rather than directly to an RDS instance. There are two security options when doing this: IAM authentication or the native database credentials from Secrets Manager. IAM authentication is the better option because it eliminates the need to read or embed credentials in the Lambda function’s code. 

For example, in Node.js, you can perform the following steps to use the RDS proxy from your application: 

• Package your MySQL client module (in NodeJS) with the chosen function. 
• Use the standard npm command to install and save the MySQL module and include its dependencies in the Lambda package. 
• You can store connection data using Lambda environment variables – this is the best way to set your database configurations because you can easily change the details without updating the code. 
• The endpoint environment variable will be the RDS proxy endpoint from earlier. The username and password will be your database credentials, while the database variable (DB) will be the name of your database schema.
• Ensure that the Lambda execution role covers the necessary connection permissions between RDS and the database. The Lambda console should do this automatically.

Confirming Your RDS Proxy Uses IAM Authentication

You can skip this step if you connect to the proxy using native database credentials. To check if your RDS proxy uses IAM authentication:

1. Go to the RDS Proxy console.
2. Select the desired proxy and click on Actions. 
3. Select Modify.
4. Under Connectivity, make sure the IAM Authentication option setting is Required.

Challenges of Connecting Lambda Functions to RDS

Here are some of the challenges associated with connecting a Lambda function to an RDS instance.

Running Lambda in a VPC

An RDS instance usually runs in a subnet in your VPC. This approach ensures security by keeping the RDS instance isolated, but it also prevents Lambda functions from accessing it. 

Conversely, a Lambda function runs in an isolated VPC managed by AWS, accessible only to the Lambda service. It can access the Internet and call any AWS service or publicly available third-party resource, but it cannot access a private resource like an RDS instance. 

You must establish a connection between the Lambda and RDS VPCs, specifying the subnet/VPC configuration. There must be an elastic network interface (ENI) for each Lambda execution environment to prevent performance issues—creating and attaching this interface takes time and slows down the function’s cold start. 

Another issue with Lambda is its scalability through different execution environments. Because you have to create dedicated ENIs for each environment, you can quickly reach your account’s ENI assignment limit.

AWS offers a VPC networking option for Lambda that leverages a networking virtualization system called Hyperlane to connect VPCs. Hyperlane creates a single VPC NAT instance when you create a function or change the VPC configurations, so you don’t need to create additional ENIs.

This update accelerates Lambda cold starts, minimizing the impact of connecting to private resources like an RDS instance. However, you still create the Hyperlane-managed ENI inside the VPC using the subnet’s IP address. Alternatively, you can use provisioned concurrency to prevent cold starts.

Having Too Many Concurrent DB Connections

Another potential issue when connecting Lambda functions to RDS involves a large number of DB connections. A standard application has a small number of connections to a database instance, stored in a shared pool for all requests—the active connections don’t exceed the specified limit even during heavy workloads. 

However, the isolation of Lambda execution environments prevents them from sharing data, requiring a dedicated DB connection for each function to connect to RDS. During heavy workloads, Lambda spawns thousands of functions with individual DB connections. The short lifecycle of Lambda functions results in constantly opening and closing the connections.

A standard SQL database engine cannot handle these workloads, impacting stability. Typically, you would mitigate this issue by limiting Lambda’s concurrency. A better approach is to use a proxy server to manage the connections. AWS offers a managed RDS Proxy service to help address this issue. 

Lambda functions can connect to RDS instances via RDS Proxy rather than establishing direct connections. RDS Proxy manages connections and deals with Lambda request spikes—it queues requests or rejects them if they exceed the DB connection limit. It can result in rejected requests but improves the application’s resiliency.

RDS Proxy is transparent to the application code. During database failures, it can switch to new RDS instances to maintain connectivity. RDS Proxy supports authorization with IAM roles instead of DB credentials, reducing the risk of leaked credentials.

Lambda, RDS and Lumigo

Lumigo is a cloud native observability tool, purpose-built for distributed systems like serverless applications and supports the suite of AWS managed serverless services, including Lambda and RDS. Lumigo connects Lambda invocations through asynchronous event sources and can trace requests to RDS in your transactions. With Lumigo, you can capture the request and response to RDS, as well as other services like DynamoDB, SNS, SQS and Kinesis, making debugging complex serverless applications easy.