OpenTelemetry on AWS: AWS Distro for OTel (ADOT) & How to Use It

What Is OpenTelemetry? 

OpenTelemetry is an open-source observability framework that provides tools, APIs, and SDKs to instrument, generate, collect, and export telemetry data (metrics, logs, and traces). It provides a unified approach to application observability, enabling developers and operations teams to understand their software’s performance and behavior in real time. 

OpenTelemetry simplifies the collection and analysis of telemetry data, making it easier for teams to monitor applications across various environments, from development to production. By standardizing how data is collected and shared, OpenTelemetry facilitates interoperability between different tools and platforms, enhancing the observability ecosystem.

The project is a merger of two former projects, OpenCensus and OpenTracing. It supports a wide variety of programming languages plus integrates with many popular backend deployment options. The framework’s extensible design enables the ability to customize data collection to meet specific application needs, allowing for the capture of information to diagnose and resolve issues.

What Is AWS Distro for OpenTelemetry (ADOT)? 

AWS Distro for OpenTelemetry (ADOT) is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project. It provides enhanced support for AWS services and monitoring tools, such as Amazon CloudWatch, AWS X-Ray, and third-party service providers. 

ADOT allows AWS customers to implement observability across their applications without the complexities of integrating multiple observability tools. Users can collect, analyze, and export telemetry data from their applications and AWS infrastructure, providing detailed insights into application performance and user experiences.

ADOT makes it easier to integrate AWS services and ensure compatibility with OpenTelemetry APIs and SDKs, simplifying the setup and configuration process for collecting telemetry data from AWS infrastructure.

Source: AWS

Components of AWS Distro for OpenTelemetry 

OpenTelemetry SDK

The OpenTelemetry SDK provides a standardized approach to API utilization needed to instrument applications for telemetry data collection. They offer a framework for capturing metrics, logs, and traces within applications, enabling the monitoring of application performance and easier troubleshooting when identifying issues. 

The SDK supports multiple programming languages, making it versatile for diverse application ecosystems. They can be configured for data collection according to their specific requirements, ensuring that only the most relevant insights are gathered from within the codebase.

Auto-Instrumentation Agent

The auto-instrumentation agent in ADOT simplifies the process of instrumenting applications by automatically utilizing the necessary telemetry collection code best suited to the application to be monitored. This agent is particularly useful for complex applications or environments where manual instrumentation would be time-consuming or challenging. 

By utilizing the auto-instrumentation agent within applications you can quickly achieve comprehensive observability across deployments with minimal configuration. The agent detects and collects data on incoming and outgoing requests, database calls, and other interactions, providing a detailed view of application and deployment performance. 

OpenTelemetry Collector

The OpenTelemetry Collector is a central component for processing and exporting telemetry data. It can receive data from various sources, process and transform it according to configurable rules, and then export it to one or more destinations. The collector is designed to support agent and standalone deployment modes to fit different architectural requirements.

The OpenTelemetry Collector in ADOT allows for efficient aggregation, processing, and routing of telemetry data, ensuring it reaches the appropriate monitoring tools and services. It supports a wide range of processors and exporters, enabling users to tailor the data flow to their specific observability needs. 

AWS Backends Supported by ADOT

Here are the primary observability backends in the AWS ecosystem that can receive telemetry data from ADOT:

AWS X-Ray

AWS X-Ray is a service that offers detailed insights into the performance and health of applications. With the integration of AWS Distro for OpenTelemetry (ADOT) applications can send traces and metrics to X-Ray, enabling a more comprehensive view of distributed applications. This includes tracing requests as they travel through your applications, understanding dependencies, and identifying performance bottlenecks.

Amazon CloudWatch

Amazon CloudWatch is a platform for monitoring and operational data collection, including logs, metrics, and events. Through ADOT, users can easily route telemetry data to CloudWatch, allowing for real-time monitoring of application performance, system health, and operational activities. This integration supports proactive issue resolution and system optimization. 

Amazon Managed Service for Prometheus

Prometheus is a popular open-source cloud-native monitoring project managed by the Cloud Native Computing Foundation (CNCF). The integration with Amazon Managed Service for Prometheus offers a fully managed Prometheus-compatible monitoring service for containerized and microservices-based environments. By using ADOT, applications can forward metrics to this service, simplifying observability for Kubernetes and other containerized application environments.

Key AWS Services Integrated with ADOT 

AWS Lambda

The AWS Lambda integration with ADOT enables instrumentation for serverless applications. This integration makes it possible to monitor the performance and behavior of Lambda functions including the capture of detailed telemetry data without managing servers or agents. 

This capability allows for real-time visibility into function executions, response times, error rates, and other critical metrics, which helps in optimizing serverless application code for performance and cost.

Amazon Elastic Container Service (ECS)

The ECS integration with ADOT enhances the observability of containerized applications. It allows for the gathering of telemetry data from containers running on ECS, including metrics, logs, and traces, providing a comprehensive view of application performance. 

The combination of ADOT and Amazon ECS supports dynamic, scalable container management, offering detailed insights into container performance, resource utilization, and operational issues. This visibility is crucial for optimizing container deployments and troubleshooting problems quickly.

AWS App Runner

App Runner automatically handles the deployment, scaling, and security of containerized applications, and with ADOT, it provides detailed telemetry data that helps in the understanding of application performance deployed to the AWS cloud. This integration ensures that applications deployed using App Runner are easy to manage and are thoroughly observable.

By leveraging ADOT with App Runner, teams can focus on building and enhancing their applications, relying on App Runner for operational aspects and ADOT for observability. This combination reduces the complexity of managing application deployments and monitoring.

ADOT Tutorial: Auto-Instrumenting Java Traces and Metrics and Sending Them to Amazon X-Ray 

Here an example workflow showing how to perform auto-instrumentation for a Java application (version 8+), using the Java Auto-Instrumentation Agent provided by ADOT, and sending the traces and metrics to X-Ray, Amazon’s distributed tracing service. Similar workflows exist for other popular programming languages and observability backends, both Amazon-specific and from other providers.

Prerequisites: Before proceeding, you’ll need to run an OpenTelemetry collector. While that is not covered in this write-up, you can see Amazon’s instructions for running a collector with Docker, ECS, EKS, or EC2.

Installing the Agent

Download the latest version of the Java agent. Alternatively, include it in your Java project using the following Maven code, adjusting <version> to the latest version:

dependencies {
    implementation("software.amazon.opentelemetry:aws-opentelemetry-agent:1.32.1")
}

<dependencies>
  <dependency>
      <groupId>software.amazon.opentelemetry</groupId>
      <artifactId>aws-opentelemetry-agent</artifactId>
      <version>1.32.1</version>
  </dependency>
</dependencies>

Using Auto-Instrumentation in Your Java Application

To apply the agent to your application, use the -javaagent flag at the start of your application code, referencing the downloaded agent’s Java Archive (JAR) artifact. This flag, like regular system properties, should be positioned before -jar or your main class name.

Most tracing systems require a defined service name and service namespace to recognize your application. You can use the OTEL_RESOURCE_ATTRIBUTES environment variable and service.name / service.namespace attribute keys.

The following command will provide the relevant attribute keys, initialize your application with the agent, and activate the instrumentation:

OTEL_RESOURCE_ATTRIBUTES=service.name=MyApp,service.namespace=MyTeam java -javaagent:path/to/aws-opentelemetry-agent.jar -jar myapp.jar

Configuring Auto-Instrumentation

Now that instrumentation is running, you can define and configure how it works. Out of the box, the OpenTelemetry Java agent uses the OTLP exporter to transmit data to an OpenTelemetry collector found at http://localhost:4317. This pathway is the same for both metrics and trace data.

You can customize the Java agent’s behavior by manipulating its parameters using standard OpenTelemetry configurations (see the OpenTelemetry Java documentation). These settings can be changed either through environment variables or through system properties. 

For example, to set the sampling rate for trace creation to 25% set the OTEL_TRACES_SAMPLER environment variable to parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG to 0.25.

For debugging or development purposes, you might want to log your traces and metrics. To do this, set the OTEL_TRACES_EXPORTER and OTEL_METRICS_EXPORTER variables to logging.

Using X-Ray Remote Sampling

Now, we can configure the ADOT Java Auto-Instrumentation Agent to work with X-Ray remote sampling, by modifying the OTEL_TRACES_SAMPLER environment variable to xray. You will need to configure your OpenTelemetry collector to allow the application to fetch the sampling configuration.

By default, the sampler sends requests to http://localhost:2000. To modify the endpoint with which the sampler communicates, change the OTEL_TRACES_SAMPLER_ARG environment variable. For instance, to set the endpoint to http://localhost:4000, use the following setting:

OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:4000.

That’s it! Your Java application is now auto-instrumented to send traces to Amazon X-Ray.

Microservices Monitoring with Lumigo

Lumigo is cloud native observability tool that provides automated distributed tracing of microservice applications and supports OpenTelemetry for reporting of tracing data and resources. With Lumigo, users can:

• See the end-to-end path of a transaction and full system map of applications.
• Monitor and debug third-party APIs, SDKs, and managed services (ex. Amazon DynamoDB, Twilio, Stripe).
• Go from alert to root cause analysis in one click.
• Understand system behavior and explore performance and cost issues.
• Group services into business contexts.

Get started with a free trial of Lumigo for your microservice applications