Building, deploying and observing SDKs as a Service - Part 1

Home Blog Building, deploying and observing SDKs as a Service - Part 1

An API, or application programming interface, is a set of protocols and instructions that allows two software applications to communicate with one other. APIs can be implemented in a number of architectural styles. One of the most popular styles is REST (representational state transfer,) which allows server and client interaction in a stateless manner.

In modern software development, APIs are important because they allow developers from different platforms—such as web applications, mobile applications, or embedded applications—to focus on implementing the user interface and user experience of the application, as opposed to spending time duplicating and implementing the same business logic across every platform. As such, in order to work efficiently with customers and application data, developers rely heavily on APIs. Besides helping internal product developers to build the internal applications, certain APIs can also be exposed to public usage, so that third party developers, or developers interested in using the APIs to build their own applications, can interact with the API.

It’s important to think carefully about how APIs are designed in order to create them in such a way that offers developers an enjoyable user experience. In addition, you also need to observe and improve all APIs that you have built continuously, as well as taking lessons for future API builds. This article will show you how to build and observe APIs so that you can maximize the value that the API brings to integrators.

What Are the Properties of a Good API?

Implementing APIs for your web application is not particularly difficult. But creating APIs that are able to scale with a growing workload and can be reused across different platforms is not easy at all. When designing and crafting your APIs, you need to take into account a number of essential properties.

Easy to Use

You are probably not the only one using the APIs that you create. More often than not, frontend developers, mobile developers, and even community developers will be using your APIs to integrate with their applications. This is why you need to make sure that your API is easy to use by providing high quality and up-to-date documentation, and following best practices when defining API paths and their methods.

Broad Use Cases

Because you need to serve different types of platforms when creating APIs, making sure that your APIs can support mobile applications, embedded applications, and web applications is vital. Otherwise, you will need to rewrite the whole API for each platform.

Reliable

The most common reason that community developers avoid a particular API is unreliability. For example, while your APIs may have worked well when the community developers integrated them into their system, when running them in production on the next day they showed a status code error even with the same input because of a recent update you made in the API. Unreliable APIs can lead to severe bugs and are often unacceptable, especially in domains such as finance, e-commerce, and healthcare. For this reason, it’s worth putting substantial effort into making sure that your API is reliable and works as expected after a new feature is added or an update is made.

Strong Security

Software security is rightly becoming a prominent consideration both for developers and customers. When building your APIs, you need to consider how to apply authentication and authorization. The goal is to allow only certain users to access APIs that manipulate sensitive and critical information in your system. Whether you should use token-based or role-based access control depends on your specific use cases, but should be planned early on before development begins.

Scalable

The ability to handle traffic spikes and load balancing is essential for ensuring that your APIs can handle a growing number of requests. One way to achieve scalability is by utilizing cloud solutions such as auto-scaling databases or Kubernetes auto-scaling pods. These solutions automatically scale up system resources when the number of user requests increases, ensuring that your APIs can handle the increased demand. 

Versioning Support

It’s not unusual for APIs to need to support both legacy and current versions of an application. There are a number of reasons behind this, such as old user devices not supporting the latest application version. However, old application versions may use an old API, whereas the current version requires an updated API. This allows for a smooth transition for applications and helps avoid issues with compatibility.

Demonstration: Interacting with APIs

To better understand how to build and observe APIs properly, let’s deploy a set of APIs for managing pet stores. To keep the article short and concise, make use of this existing pet store API provided by the Swagger team. You’ll create AWS Lambda functions that call the existing pet store APIs, then create an API gateway to trigger these Lambda functions.

Introduction: Pet Store APIs

Let’s say you’re trying to manage the pets in our store. The pet store APIs allow us to create, update, and delete pets from the store.

Figure 1: Example API request to the pet store API v3

Figure 1: Example API request to the pet store API v3

Above is an example of an API for finding pets by available status.

Create Lambda Functions

Let’s try to deploy APIs as you would in real projects by creating Lambda functions that wrap the existing pet store APIs. You need to use an HTTP client to make API requests to the real pet store APIs; use the axios library to do that. The Lambda runtime for Node.js in AWS does not have axios. You need to prepare the runtime environment locally first, then import it to AWS Lambda.

Create a Local Runtime Environment

First, initiate a new Node.js project on your machine:

cd ~

mkdir Projects

cd Projects

mkdir pet-store-lambda

cd pet-store-lambda

npm init

Then, install the axios library by running:

Zip the pet-store-lambda directory to the pet-store-lambda.zip file:

cd ..

zip -r pet-store-lambda.zip pet-store-lambda

Import the Local Runtime to AWS Lambda Node.js

From the AWS Lambda page, create a new Lambda function using Node.js runtime. 

Figure 2: Create a new Lambda function

Figure 2: Create a new Lambda function

Note: Node.js version 16 is more stable than version 18 at the time of writing this article, so let’s stick with that for now.

After creating a new Lambda function, import the zip file you created earlier into the function. The code structure of your AWS Lambda function should look like this:

Figure 3: Code source tree after importing the zip file

Figure 3: Code source tree after importing the zip file

You need to move the node_modules directory to under managingPetStores directory and remove the pet-store-lambda directory. Then create a new index.mjs file inside the managingPetStores directory.

Your code source structure should look like this:

Figure 4: The code source should look like this after your modification

Figure 4: The code source should look like this after your modification

Add the Code for AWS Lambda Function

The index.mjs is an entry point of our AWS Lambda function. Add the following code to the index.mjs file:

import axios from “axios”;

const domain = “https://petstore3.swagger.io/api/v3”;

 

export const handler = async (event) => {

    

    if(event.httpMethod === ‘POST’){

      

      let postRequestJSON = JSON.parse(event.body);

        const response = await axios

          .post(domain+’/pet’,postRequestJSON);

          console.log(postRequestJSON)

          return {

            statusCode: 200,

            body: JSON.stringify({

              message:”Success”,

              details:response.data

            })

        }

    }

    if(event.httpMethod === ‘DELETE’){

      const res = await axios

          .delete(domain+’/pet/’+event.pathParameters.id);

          console.log(res);

        return {

            statusCode: 200,

            body: JSON.stringify(“Delete successfully”)

        };

    }

    if(event.httpMethod === ‘PUT’){

      let putRequestJSON = JSON.parse(event.body);

        const response =  await axios

          .put(domain+’/pet’,putRequestJSON);

          console.log(putRequestJSON)

          return {

            statusCode: 200,

            body: JSON.stringify({

              message:”Success”,

              details: response.data

            })

        };

    }

    if(event.httpMethod === ‘GET’ && event.resource === ‘/pet/{id}’){

      

      const response = await axios.get(domain +’/pet/’+event.pathParameters.id);

      return {

        statusCode: 200,

        body: JSON.stringify({

          message:”Successful retrieving”,

          details:response.data

        })

      }

    }

    

    if(event.httpMethod === ‘GET’ && event.resource === ‘/pet/findByStatus’){

      

      const response = await axios.get(domain +’/pet/findByStatus’,{params: {

      status: event.queryStringParameters.status

    }

    });

      return {

        statusCode: 200,

        body: JSON.stringify({

          message:”Successful retrieving”,

          details:response.data

        })

      }

    }

    

 

};

Put simply, the above code checks for event triggers from AWS API Gateway. If an event has a method with, let’s say, ‘GET’ (event.httpMethod === ‘GET’,) and has a resource matched “/pet/findByStatus” (event.resource === ‘pet/findByStatus’,) the Lambda function will send a new request to the existing API pet store and get its API response.

Click “Deploy” to apply this change.

Figure 5: Add code to the index.mjs file and deploy the Lambda function

Figure 5: Add code to the index.mjs file and deploy the Lambda function

Add API Gateway

From the AWS API Gateway page, create a new REST API Gateway with the following structure:

Figure 6: Add API specs to the Amazon API Gateway

Figure 6: Add API specs to the Amazon API Gateway

Note that when creating the API gateway items, you need to add your Lambda function and tick “Use Lambda Proxy integration”:

Figure 7: Add Lambda function and enable “Use Lambda Proxy Integration” when creating a new API in the API Gateway

Figure 7: Add Lambda function and enable “Use Lambda Proxy Integration” when creating a new API in the API Gateway

Deploying the API is simple: click “Deploy” on the Deploy page.

Figure 8: Deploy the new API Gateway stage named “Production”

Figure 8: Deploy the new API Gateway stage named “Production”

You successfully implemented your API! Now, let’s try to interact with it.

Interact with the APIs

First, grab the AWS API Gateway invoking the URL from the Stage page’s Settings tab:

Figure 9: Grab the API Gateway

Figure 9: Grab the API Gateway

Then, open up Postman and try to add a new pet to the store.

Figure 10: Add a new pet to the store using via your newly created API

Figure 10: Add a new pet to the store using via your newly created API

The API works as expected. Now, try to remove the pet from the store.

Figure 11: Remove a pet from the store using via your newly created API

Figure 11: Remove a pet from the store using via your newly created API

The pet can be removed successfully.

Now that our APIs are up and running, another important step in their implementation is monitoring and debugging them. Implementing the monitoring and logging components from scratch is time-consuming and could mean missing out on best practices, which in turn could lead to difficulties in setting up and maintenance of system resources. This is where you need a product like Lumigo to monitor your API.

Using Lumigo to Monitor the API

To observe how the deployed pet store API works, let’s add Lumigo monitoring and observability support to our AWS environment.

From the Lumigo page, connect to your AWS account.

Figure 12: Connect Lumigo to your AWS account

Figure 12: Connect Lumigo to your AWS account

Then, enable CloudWatch Logs for your API Gateway.

Figure 13: Enable Logs/Tracing in AWS API Gateway

Figure 13: Enable Logs/Tracing in AWS API Gateway

At this stage, you need to provide log destination arn value. To do so, first create a new role for accessing AmazonAPIGateWay logs from CloudWatch.

Figure 14: Create a new role for IAM account to get logs from API Gateway to CloudWatch

Figure 14: Create a new role for IAM account to get logs from API Gateway to CloudWatch

Copy the ARN value into the AWS CloudWatch setting.

Figure 15: Copy the ARN address from the new role

Figure 15: Copy the ARN address from the new role

Copy to console the ARN log group from CloudWatch Log group for Lambda function “managingPetStores.”

Figure 16: Copy the ARN log group to clipboard

Figure 16: Copy the ARN log group to clipboard

Next, paste the value in the above step to Logs/Tracing page in API Gateway (you need to remove the final * character in the string.)

Figure 17: Enter the ARN Access Log and Log Format information in the Logs/Tracing tab of API Gateway

Figure 17: Enter the ARN Access Log and Log Format information in the Logs/Tracing tab of API Gateway

Finally, update the code in your AWS Lambda function to allow Lumigo to collect Lambda function metrics. To do that, first install the @lumigo/tracer, and then add the wrapper code for the Lumigo tracer. To install the @lumigo/tracer in AWS Lambda function, install the @lumigo/tracer library in your local environment first.

cd pet-store-lambda

npm install @lumigo/tracer

Then zip the project and upload it to AWS Lambda function just like you already did when adding the axios library to AWS function runtime. You also need to add the following code in index.mjs file inside AWS Lambda function.

import lumigo from ‘@lumigo/tracer’;

 

const tracer = lumigo({ token: ‘YOUR-TOKEN-HERE’ });

 

export const handler = tracer.trace(async (event, context) => {

  …

});

Note that the Lumigo token may be found on the Lumigo settings page as below:

Figure 18: Grab the token for manual tracing from the Lumigo Settings page

Figure 18: Grab the token for manual tracing from the Lumigo Settings page

Deploying the change in the AWS Lambda function. You’ve now successfully set up the Lumigo connection for monitoring the AWS Lambda function. Let’s move on and see the invocations, metrics, and logs generated by AWS Lambda functions, which you can view on the Lumigo dashboard.

Monitor AWS Lambda Functions Using Lumigo

From Lumigo dashboard, click on Functions on the left panel, and you can see the recent invocations from the “managingPetStores” Lambda function.

Figure 19: Overview of the collected invocations from Lumigo dashboard

Figure 19: Overview of the collected invocations from Lumigo dashboard

Click on the “Metrics” tab to see the invocations successes and failures, along with other metrics like consumed memory and duration of each request that the Lambda function executed.

Figure 20: Overview of metrics for “managingPetStores” Lambda function

Figure 20: Overview of metrics for “managingPetStores” Lambda function

Click each recorded invocation to see how the components in the system interacted with one another inside the AWS Lambda function.

Figure 21: Visualization of how components interact with each other inside an invocation

Figure 21: Visualization of how components interact with each other inside an invocation

In this invocation, the API gateway for API “PUT /Production/pet” made a request to “managingPetStores” Lambda function, and the Lambda function then requested to the actual “petstore3.swagger.io” API server.

On the other side of the screen, you can see the event data that the Lambda function received from API gateway, along with the return value from the Lambda function.

Figure 22: Details of the event body and return value from AWS Lambda function inside an invocation

Figure 22: Details of the event body and return value from AWS Lambda function inside an invocation

And that’s how Lumigo steps in to help us monitor APIs easily and efficiently.

Conclusion: Better API Monitoring with Lumigo

This article reviewed how to build and observe APIs so that your APIs can both satisfy business requirements and allow developers to get started using them. Implementing APIs that work at a basic level is easy, but implementing them such that they can be used by thousands of developers and be applied on multiple platforms is not a trivial task.

Monitoring APIs is essential to maintaining their functionality and performance. Lumigo helps you to find and fix API issues in serverless and distributed environments in a matter of minutes, and saves you time and energy by offering a pre-built observing tool for all your API needs.

Stay tuned for part 2 of this article where we will take it a step further by building, deploying, and observing SDKs as a service. We will cover how to generate SDKs to make it even easier for developers to use your APIs, as well as how to monitor and maintain the performance of your SDKs.