What Is Container Monitoring?

Topics

Container Monitoring: Challenges, Tools, and 4 Tips for Success

Container monitoring is a way to collect metrics and track the health of containerized applications and microservices architectures. This process can be difficult due to the ephemeral nature of containers and the limitations of traditional application performance monitoring tools.

The goal of container monitoring is to ensure that container workloads are performing as expected and running smoothly. Container monitoring is a subset of container observability, which also includes log analysis, notifications, and tracing.

This is part of an extensive series of guides about [microservices].

Why Is Container Monitoring Important?

As containers grow in popularity, new challenges emerge that can expose businesses to financial damages, lost productivity, and regulatory penalties. Monitoring can help with early identification of performance issues, software bugs, and security incidents, and provide the information teams need to remediate these issues.

Container monitoring also helps prevent outages by reducing the mean time to recovery (MTTR) of performance issues and providing data to support the overall health of your applications. The ability to automatically raise alerts, monitor time series data, and troubleshoot issues improves the user experience and ultimately, business outcomes.

Monitoring provides a holistic view of containerized infrastructure. Through data aggregation and visualization, you can optimize functionality and better identify the root cause of performance issues.

What Is Container Logging and Observability?

Logging is an essential part of any computing system. Without logging, it is difficult to identify and troubleshoot problems. A log is a record of events that occurred over a period of time.

Logging is a mechanism for capturing and recording information about a program so that it can be monitored and debugged, while the program is running, and afterwards. Logging helps developers understand what their code is doing, and helps application owners and maintainers understand how systems operate and behave at runtime.

There are several logging methods for containers, including Docker’s logging driver, Fluentd, and rsyslog. These tools can be used to troubleshoot container or log data issues.

Observability builds on logging to provide deep visibility into distributed systems, automating and accelerating problem identification and resolution. Observability platforms continuously instrument and collect telemetry data, and make it easy to add instrumentation to existing applications and infrastructure components.

Learn more in our detailed guide to container observability (coming soon)

Container Monitoring Challenges

Containers Add a New Layer to Infrastructure

Containers add a new dynamic layer to your infrastructure. Performance monitoring tools should be able to automatically discover all running containers, pick up container deployment changes immediately, and update them in real time to map hosts.

Dynamic Deployment and Orchestration

A container orchestration tool, such as Kubernetes, deploys containers on the most appropriate hosts in your cluster. Containers move from one host to another for the purposes of horizontally scaling. This means that monitoring tools need to determine which host is running which container, and be able to persist log data even after containers shut down.

Complex Resource Management

Containers use multiple tiers of resources, and it can be complex to monitor and identify resource utilization issues. Containers often run within clusters managed by an orchestrator like Kubernetes. In this case, there are multiple resource constraints:

The number of nodes in the cluster
The specific resource requirements of a container
Resource availability on a specific node / host
Resource requirements of specific applications running within a container

When containerized applications experience performance issues or downtime, it can sometimes be difficult to pinpoint the cause, because these problems can originate from one or even several components in the containerized environment.

Managing Logs in Container Clusters

Unlike traditional application logs, container logs use console output streams called stdout and stderr. Containers use a logging driver to collect logs and deliver them to a destination.

When containers run in a cluster, each container has a different log stream stderr and stdout. To monitor application logs, you must parse them individually and combine them to get a full picture of the cluster. It is also important to identify the source of the logs (i.e. which log belongs to which container) and add necessary metadata such as container ID and container name.

Learn more in our detailed guide to container logging (coming soon)

What Is Distributed Tracing and Why Is it Important for Container Monitoring?

Distributed tracing is a technique for tracking the execution of a distributed system, such as a microservices architecture or a containerized environment. It allows administrators to understand how different components of the system interact with each other and how requests are processed as they flow through the system.

Distributed tracing is important for container monitoring because it can help administrators understand the performance and behavior of containerized applications and the underlying infrastructure. By tracking the execution of a distributed system, administrators can identify bottlenecks and latency issues, understand the impact of changes to the system, and troubleshoot problems more effectively.

Distributed tracing works by inserting trace instrumentation into the code of the system being monitored. This instrumentation generates trace data as the system executes, which is then collected and analyzed by the tracing system. Distributed tracing tools typically provide features such as visualization and analysis, which can help administrators understand and analyze the trace data.

What Are Container Monitoring Tools?

Container monitoring tools monitor running containers, collect container activity logs, and analyze data to provide insight into container performance. These tools typically provide the following features and capabilities:

Dashboards and visualizations—present container data visually, making it easy for users to analyze without advanced analytical knowledge.
Architecture visualization—providing a graphical representation of services, integrations, and infrastructure related to the container ecosystem.
Anomaly detection—allows users to automate their systems to continuously monitor activity and compare it against baseline patterns.
Performance baselines—baselines and benchmarks are used to establish standard performance levels and compare them to actual application and infrastructure activity.
Alerts—allows team members to receive relevant information about events in the container ecosystem in a timely manner.
API monitoring—API monitoring tracks connections to containerized environments and detects anomalies in functionality, user access, and traffic.
Configuration monitoring—allows users to monitor configuration rule sets, enforce policy actions, and log changes to maintain regulatory compliance.
Improvement suggestions—provides suggestions for potential solutions or enhancements to address issues like slowdowns, errors or outages.
Automation—performs changes to containerized resources in real time to address issues discovered in the environment.

Benefits of Container Monitoring Tools

Benefits that container monitoring solutions provide include:

Resolve issues faster and more proactively—container monitoring tools work by collecting application metrics and dependencies. They use this information to benchmark performance metrics and help identify anomalies. Container monitoring solutions provide real-time monitoring of application and infrastructure performance by alerting administrators when problems arise.
Detailed visualizations—allow users to quickly drill down to the root cause of any issues they encounter, improving the team’s ability to fix issues quickly and minimize impact to end users.
Improved performance—provide visibility into resource usage, redundancies, and inefficiencies. Container monitoring allows teams to evaluate their containerized applications and fine-tune them for optimal performance.
Change management—all changes deployed by the development team are continuously monitored. The tool instantly detects issues and vulnerabilities and notifies developers so the team can take immediate action to fix them.

Learn more in our detailed guide to container monitoring tools (coming soon)

What to Look for in a Container Monitoring Tool

A good monitoring system should provide an overview of the entire application with information about each component. Here are a few things to consider when choosing a container monitoring solution:

Ability to combine and correlate metrics and logs from a variety of data sources
Ability to see overall application performance across multiple platforms
Ability to correlate events and logs to spot anomalies
Ability to proactively prevent events, and reactively respond to minimize damage
Ability to drill down into each component and layer to isolate and determine root cause
Ability to easily add instrumentation for existing components
Ability to easily set up alerts and automations

Large enterprises may need to use more than one tool to monitor different containerized applications. Before deciding which container monitoring tool or combination of tools is right for your business, it’s important to identify the metrics you need to monitor, and your proposed continuous monitoring workflow.

4 Tips for Success

1. Don’t Focus Too Much on Individual Containers

The biggest difference between containers and virtual machines in terms of monitoring is the need to shift the focus from individual containers to a pod or an entire cluster. Containerized applications are often built as microservices. As such, individual containers can only account for a small portion of the infrastructure’s performance.

In most cases, you run multiple containers of the same microservice for availability and scalability. Similarly, viewing individual containers can generate misleading information. So, focus on monitoring a specific set of containers as a unit. That being said—sometimes you need to see metrics for a specific container to debug a specific issue.

2. Pay Attention to Network Traffic

Traffic between containers

For containers, network traffic is much more complex than in monolithic applications. It’s important to understand how container network traffic flows, and monitor it accordingly.

For containers, network traffic between containers on the same machine is just as significant as network traffic between different machines. In some cases, much more traffic goes from one container to another than from one machine to another. Therefore, it is essential to monitor traffic between containers, whether they are on the same machine or different machines.

API traffic

Another aspect of container network monitoring is that microservices typically communicate with each other via a REST API. When monitoring HTTP response codes, it’s important to note that a high number of 5xx errors doesn’t necessarily mean your customers are directly impacted, because these errors can occur in container-to-container traffic.

Visualizing the network mesh

Finally, containers can create very complex network meshes, so it’s important to understand which microservices communicate with which microservices. This is less important for traditional infrastructure monitoring, where you could focus monitoring based on the VM name or network segment.

For containers, you need to create a full service map to visualize the traffic between your microservices. This gives you a better understanding of your network traffic and helps identify anomalous, unwanted, or malicious traffic.

3. Reduce Unneeded Alerts

Containerized systems also require a different approach to alerting. For example, in a VM-based deployment, it is common to receive alerts when a VM restarts—this is not a good idea for containers, which are created and destroyed very frequently. Part of the container orchestrator’s job is to move containers to different nodes based on various factors.

This means that container starts and stops are not abnormal, so you usually don’t need to be notified. However, you should receive notifications if a container restarts too many times (in Kubernetes this is known as CrashLoopBackOff).

The same applies to resource usage. Depending on your setup, you might have autocaling mechanisms that go into action when containers use too many resources on a machine. In this case, containers that use more resources on a machine can be handled by autoscaling. You should be alerted when the autoscaling mechanism fails to find a suitable machine for the container, or if resource utilization is extreme, which could indicate an error or cyber attack.

4. Leverage Tools for Real-time Monitoring

Containers are ephemeral by design. This means that the data you have depreciates over time, making real time analysis important. Monitoring tools that provide data visualization can come in useful.

Monitoring tools provide a graphical interface to help uncover important changes and anomalies. There are also advanced monitoring solutions that combine machine learning (ML) with automatic alerting, ensuring accurate, timely and accurate reporting of incidents to all parties involved.

Container Observability with Lumigo

Lumigo is a cloud native observability tool, purpose-built to navigate the complexities of microservices. Through agentless automated tracing, Lumigo stitches together asynchronous requests across the many distributed components that make up a cloud native app. From ECS to third party APIs Lumigo visualizes requests in one complete view, and monitors every service that a request passes through. Leveraging the end-to-end observability that Lumigo provides, as well as the many features that make debugging container apps easy, developers have everything they need to find and fix errors and issues fast:

With Lumigo users can:

See the end-to-end path of a container request and full system map of applications
Monitor and debug third party APIs and managed services (ex. Amazon DynamoDB, Twilio, Stripe)
Integrate alerts with notification platforms like Slack and go from alert to root cause analysis in just a few clicks
Explore application performance to understand system behavior and optimize performance and costs

Get started with a free trial of Lumigo for your microservice applications

See Additional Guides on Key Microservices Topics

Together with our content partners, we have authored in-depth guides on several other topics that can also be useful as you explore the world of microservices.

Debug fast and move on.

Resolve issues 3x faster
Reduce error rate
Speed up development

No code, 5-minute set up

Start debugging free