OpenTelemetry is an open-source observability project overseen by the Cloud Native Computing Foundation (CNCF). It makes it easy to instrument applications and collect logs, traces, and metrics.
OpenTelemetry can collect logs through a combination of SDKs, agents, and exporters. It provides libraries and agents for multiple programming languages and frameworks, allowing the integration of OpenTelemetry logging capabilities directly into applications:
In this article
The OpenTelemetry log data model offers several benefits that enhance observability and debugging capabilities compared to traditional logging methods:
OpenTelemetry collects three main types of logs:
System format logs refer to the logs generated by the operating system and other system-level software running on the host computer. These logs are important for monitoring the system’s health and performance. They provide valuable information on system events like startup and shutdown, hardware and software errors, and security events.
OpenTelemetry logging can collect these system format logs and present them in a consolidated view alongside logs from applications. This is especially useful in distributed systems where logs from multiple hosts need to be analyzed together.
Third-party application logs are those generated by software that is not developed by the entity operating the system. This includes databases, web servers, and other middleware. These logs can provide insights into the operation of these software components and can help diagnose issues that affect the overall system. OpenTelemetry can collect third-party application logs, parse them, and present them in a structured format that makes analysis easier.
First-party application logs are those generated by software developed by the entity operating the system. These logs can provide detailed information about the operation of the application and can be invaluable in debugging and optimizing the application.
OpenTelemetry can collect first-party application logs, enrich them with context from the resource and InstrumentationScope, and present them alongside system and third-party application logs. This provides a comprehensive system view and facilitates efficient troubleshooting and analysis.
The OpenTelemetry log data model provides a structured format designed to capture as much information about an event as possible. It has several key components, each capturing a unique aspect of the event.
The timestamp is an important record that denotes the exact moment when an event occurs. It plays a crucial role in understanding the order of events and in coordinating logs with traces and metrics. Accurate timestamps help diagnose issues by identifying if a particular event consistently precedes a failure or if there’s a significant delay between two related events. They provide a temporal context to the events.
When considering the timing of an event, it’s important to understand the concept of ObservedTimestamp, which refers to the exact moment when an event is noticed or recorded within a system and marks the point of recognition by the monitoring tools rather than the actual occurrence of the event. The importance of this distinction lies in the potential time gap between the event’s occurrence and its subsequent observation. This gap can be influenced by factors such as system latency, network delays, or the processing time required by the monitoring tools.
In contrast, timestamps found in logs traditionally denote the exact time an event occurred, not when it was detected or recorded by an observing system. This fundamental difference means that while log timestamps aim to provide a historical record of events as they happened, ObservedTimestamps offer a real-time perspective of when those events were perceived by the system. Understanding the distinction between these two types of timestamps is essential for accurate event analysis and troubleshooting, as it helps identify potential delays or issues within the monitoring process.
Trace context fields in the OpenTelemetry log data model capture information about the trace and span associated with the event. A trace represents a single operation or transaction, like a user request, and a span represents a specific unit of work done within that operation.
By including the trace context fields in the log data, you can easily correlate logs with traces. This allows a greater understanding of the broader context of an event, i.e., how it fits into the overall operation or transaction. This ability to correlate logs with traces is one of the key benefits of OpenTelemetry logging, as it significantly enhances software observability.
Severity fields capture the severity or the importance of the event. They can be either textual or numeric. For example, severity might range from DEBUG, which represents detailed information useful for debugging, to FATAL, which represents severe error events.
While different systems might use different notations for severity, OpenTelemetry creates unified severity fields that make it possible to analyze log severity across the entire environment.
The body is the primary content of the log entry. It usually contains a human-readable message that communicates the details of an event. The body can also contain structured data. For instance, it may include a stack trace, which is a report of the active stack frames at a particular point in time during the execution of a program. This can be useful in debugging.
The resource component of the OpenTelemetry logging data model is an immutable description of the entity that is generating the log. This could be a service, a host, a container, or any other unit of software or hardware that produces logs.
The resource is defined at startup and does not change during the existence of the entity. It contains attributes that describe the entity. For example, the resource may contain attributes such as the pod name, namespace, and labels in a Kubernetes pod. The resource is critical in correlating logs produced by the same entity and distinguishing logs from different entities.
The InstrumentationScope is a component that describes the source of the log within the entity that is generating it. While the resource is immutable and describes the entity as a whole, the InstrumentationScope can change during the execution of the entity. It provides additional context to the log, such as the part of the code that generated the log or the request that triggered the log. This makes the InstrumentationScope invaluable in diagnosing issues in complex systems where various components may generate logs.
Attributes are key-value pairs that provide additional context to the log. They can be associated with the body, the resource, or the InstrumentationScope. Attributes can include details such as the severity of the event, the timestamp of the event, and the thread that generated the event. They can also include any other information that may be useful in understanding the event or in filtering and aggregating logs. The flexibility of attributes makes them a powerful tool in enriching logs.
Related content: Read our guide to OpenTelemetry architecture
Prerequisites: To run the code below, please use opentelemetry-sdk version 1.14.0 and install the following two libraries:
pip3 install opentelemetry-api opentelemetry-sdk
In each of the examples below, save the Python code to a file, such as app.py and run it using python app.py. This will execute the sample code once; for simplicity, the examples do not support continuous log monitoring, but this can easily be added.
In some cases, your application might be writing logs to a regular JSON file. OpenTelemetry makes it easy to tail these logs. Let’s start by looking at a simple example of how you can tail a JSON file using OpenTelemetry logging.
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
SimpleSpanProcessor(ConsoleSpanExporter())
)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("foo"):
with tracer.start_as_current_span("bar"):
with tracer.start_as_current_span("baz"):
print("Hello world from OpenTelemetry Python!")
In this example, we set up a trace provider, create a span processor, and initiate a tracer. We then start a few spans and print a message. When you run this code, OpenTelemetry will automatically tail the logs being written to the console
Syslog is a standard for message logging, often used in systems and network administration. OpenTelemetry provides an easy way to collect and integrate these logs into your observability stack.
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
import logging
trace.set_tracer_provider(TracerProvider(resource=Resource.create({ "service.name" : "my-service" } )))
logging.basicConfig()
logging.root.setLevel(logging.NOTSET)
logging.basicConfig(level=logging.NOTSET)
logger = logging.getLogger("my-logger")
logger.info("Starting up")
with trace.get_tracer("my-application").start_as_current_span("my-span"):
logger.info("Hello, world!")
In the above example, we set up a tracer provider with a service name. We then get a logger and log a startup message. We start a span and log a message within that span. OpenTelemetry will automatically collect these logs and associate them with the relevant traces.
Here is how to set up a log emitter provider with a Kubernetes service name. OpenTelemetry will then be able to associate the logs with that Kubernetes service.
import logging
from opentelemetry.sdk._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk.resources import Resource
logger_provider = LoggerProvider(
resource=Resource.create(
{
"service.name": "my-service"
}
),
)
set_logger_provider(logger_provider)
handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)
logging.getLogger().addHandler(handler)
logger = logging.getLogger("my-logger")
logger.info("Starting up")
logger.info("Hello, world!")
logger.info("Goodbye, world!")
In this example, we get a logger and log a startup message. We then log a couple of messages within a block of code where we are catching logs. OpenTelemetry will automatically collect these logs and associate them with the relevant service.
The code below shows how to set up a log emitter provider to log messages representing Kubernetes events. The messages contain a Kubernetes service name, and as above, OpenTelemetry can then associate them with the Kubernetes service.
import logging
from opentelemetry.sdk._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk.resources import Resource
logger_provider = LoggerProvider(
resource=Resource.create(
{
"service.name": "my-service"
}
),
)
set_logger_provider(logger_provider)
handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)
logging.getLogger().addHandler(handler)
logger = logging.getLogger("my-logger")
logger.info("Starting up")
logger.info("Hello, world!")
logger.info("Goodbye, world!")
In the above example, we get a logger and log a startup message. We then log a couple of messages within a code block where we are catching logs. These messages represent Kubernetes events. OpenTelemetry will automatically collect these events and associate them with the relevant service.
Lumigo is cloud native observability tool that provides automated distributed tracing of microservice applications and supports OpenTelemetry for reporting tracing data and resources. With Lumigo, users can:
Get started with a free trial of Lumigo for your microservice applications