- Guide Content
CloudWatch Logs: The Basics and a Quick Tutorial
What Is AWS CloudWatch Logs?
CloudWatch Logs is a monitoring and management service provided by Amazon Web Services (AWS) that collects and stores log files from various resources, including EC2 instances, AWS Lambda functions, AWS CloudTrail, and other AWS services. It enables users to view and analyze these logs in near real-time, assisting in debugging applications and understanding system behavior.
This service simplifies log management by aggregating logs across different sources, offering a centralized operational analysis and troubleshooting platform. Users can set alarms and notifications based on specific log data patterns, enabling proactive identification of issues and integration with automated response mechanisms.
This is part of a series of articles about AWS Cloudwatch.
In this article
Key Features of CloudWatch Logs
CloudWatch Logs offers the following features:
Query Log Data
CloudWatch Logs allows for the execution of detailed queries on log data facilitated by CloudWatch Logs Insights, a tool that offers interactive log analytics. This enables the crafting of queries to filter, aggregate, and sort log data, making it easier to uncover specific information or identify trends.
The service supports a variety of query statements, from simple text searches to complex aggregations, accommodating diverse analysis needs. It enables rapid isolation of issues within large volumes of log data.
Detect and Debug Using Live Tail
Live Tail in CloudWatch Logs provides real-time log streaming, enabling the ability to view and analyze log data immediately as it’s generated. This feature is useful for detecting and debugging issues as they occur, offering instant visibility into application behavior and system performance. Live Tail supports various filters and patterns, allowing users to focus on specific log entries or events of interest.
Audit and Mask Sensitive Data
CloudWatch Logs incorporates features for auditing and masking sensitive data within log files, ensuring compliance with privacy laws and regulations. This brings the ability to define rules to automatically identify and redact sensitive information, such as personally identifiable information (PII), before storing the logs. This helps maintain data security and privacy while benefiting from comprehensive log analysis and monitoring.
Amazon CloudWatch Logs Key Concepts
When working with logs in CloudWatch, it’s important to be familiar with these concepts:
Log Class
CloudWatch Logs supports two log classes:
- Application logs: Generated by the applications running on your AWS resources, providing insights into their operations and performance.
- Platform logs: Originate from the AWS services, offering visibility into service-level events and operational health.
Log Events
Log events represent individual records within CloudWatch. Each event consists of a timestamp and a message detailing a single occurrence within the monitored system or application. Events are the basic units of log data, providing the raw information used for analysis and troubleshooting. By examining these events, users can identify problems, track changes, and understand the sequence of actions leading to a specific outcome.
Log Streams
Log streams are sequences of log events from the same source. They aggregate related events, facilitating organized log data analysis. Streams help manage log volume by grouping events by their origin, such as a particular application instance or AWS resource. They make it easier to correlate related events and identify patterns.
Log Groups
Log groups serve as containers for log streams that share the same retention, monitoring, and access control settings. They represent a higher-level organizational unit, categorizing logs by application, service, or environment. Log groups streamline log management by consolidating related streams, offering a unified view of log data. They enforce consistent policies across related logs, simplifying configuration and policy management.
Metric Filters
Metric filters enable the creation of custom metrics from log data for the definition of rules to extract information from log events, transforming it into numerical metrics that can be monitored and alerted on. This allows for detailed analysis and action based on specific log patterns or values.
Retention Settings
CloudWatch Logs offers flexible log retention settings, enabling users to specify how long log data should be retained before deletion. This allows for cost-effective log management, as users can balance the need for historical data access against storage costs. Retention periods can be customized per log group, providing granular control over data lifecycle management.
Common Use Cases of AWS CloudWatch Logs
Here are some popular use cases for logs in CloudWatch:
Troubleshooting Application Issues
By aggregating logs across distributed environments, CloudWatch Logs enables rapid identification and resolution of errors, ensuring minimal impact on user experience. This capability is useful for maintaining high availability and performance in modern applications. Using CloudWatch Logs enables the ability to quickly pinpoint the root cause of issues, reducing downtime and improving service reliability.
Monitoring Performance
Analyzing log data allows you to assess system health, identify performance bottlenecks, and understand usage patterns. This insight enables proactive measures to ensure consistent performance and user satisfaction. It helps organizations adjust resources and configurations in response to real-time data, optimizing performance and resource utilization.
Analytics and Cost Optimization
CloudWatch Logs supports analytics and cost optimization by providing detailed usage and performance data. By analyzing log data, organizations can gain insights into system efficiency, identify underutilized resources, and make informed decisions on scaling and provisioning. This can lead to significant cost savings and improved system efficiency.
Debugging System Issues
CloudWatch Logs offers in-depth visibility into system operations and application behavior. Providing access to detailed log data enables the ability to identify and correct issues at the system level, ensuring stability and reliability. This debugging capability helps maintain the integrity of IT systems, allowing for prompt identification and resolution of software bugs, configuration errors, and other operational problems.
Amazon CloudWatch Logs Challenges and Limitations [QG3]
While CloudWatch Logs is a useful tool, there are some concerns to be aware of:
Complex User Interface
CloudWatch’s user interface can be challenging, particularly when troubleshooting and identifying the root cause of issues. This complexity arises because CloudWatch comprises several tools, requiring users to navigate between them to find specific information. As the volumes of logs increase, filtering and searching through the interface becomes more complicated.
For example, when troubleshooting an error in an AWS Lambda function, users must first identify the error in the CloudWatch dashboard, navigate through numerous log groups named after the erroring function, and manually sift through pages to find the specific log entry that caused the error.
Source: AWS
Costs Are Difficult to Predict
CloudWatch’s pricing structure is complex and unpredictable, posing a significant challenge for users trying to estimate costs. The pricing depends on many factors, including the number of queries, events, metrics, and dashboards used, the setup of alarms and custom events, and the volume and retention period of logs.
This complexity makes it difficult for users to predict their monthly expenses, as costs vary widely based on the activities and resources used. CloudWatch Logs are often used to monitor unpredictable events such as errors, traffic spikes, or security issues, making it more challenging to predict costs.
Usability at Scale
As users accumulate logs at the terabyte scale and wish to retain them for extended periods, CloudWatch’s usability and cost-effectiveness diminish significantly. Managing such large volumes of logs for diagnostic or security purposes becomes costly and impractical with CloudWatch.
While it remains an effective tool for monitoring metrics, users will often turn to more scalable and cost-effective solutions for log management and analysis as log data volumes grow.
Quick Tutorial: Getting Started with CloudWatch Logs
Using the CloudWatch agent, you can collect logs from Amazon EC2 instances and on-site servers to CloudWatch Logs. Let’s see how to install the agent and start collecting log data. The code below was shared in the AWS documentation.
Step 1: Install the CloudWatch Agent
The CloudWatch agent is available for several operating systems. For example, if you are using Amazon Linux 2, you can install it by running this command:
sudo yum install amazon-cloudwatch-agent
Additionally, ensure that the IAM role associated with the instance has the CloudWatchAgentServerPolicy attached.
Step 2: Set Up the CloudWatch Agent Configuration File
Before you use the CloudWatch agent on any servers, you’ll need to create a CloudWatch agent configuration file or possibly multiple files. These JSON files detail the metrics, logs, and traces you want the agent to gather, with the option for custom metrics.
There’s a wizard to help you create this file, which you can run using the command amazon-cloudwatch-agent-config-wizard. It will guide you through the following questions:
- Where will the agent be installed (Amazon EC2 instance or on-premises server)
- Which operating system is the server running (Linux or Windows Server)
- Whether the agent should forward log files to CloudWatch Logs
- What should be the retention period of log files sent to CloudWatch Logs (use -1 to keep logs forever)
- What metrics set to use (default metrics set or custom set)
- Whether to collect custom metrics from applications or services using StatsD or collectd
- Whether you are transitioning from an existing SSM Agent
The wizard can automatically detect which AWS Region to use and what credentials are needed, as long as your AWS credentials and configuration files are ready before you launch the wizard.
Step 3: How to Gather High-Resolution Data using CloudWatch Agent
In the agent configuration, the metrics_collection_interval field determines the frequency, in seconds, at which data is collected. To gather high-resolution data, set this field to a value that is less than 60.
For example, if you want all of your data to be high-resolution and collected every 10 seconds, set the value as follows:
"agent": {
"metrics_collection_interval": 10
}
You can also specify different intervals for different types of data. In the example below, cpu data will be collected every second, while all other data will be collected every minute.
"agent":{
"metrics_collection_interval": 60
},
"metrics":{
"metrics_collected":{
"cpu":{
"resources":[
"*"
],
"measurement":[
"cpu_usage_guest"
],
"totalcpu":false,
"metrics_collection_interval": 1
},
"disk":{
"resources":[
"/",
"/tmp"
],
"measurement":[
"total",
"used"
]
}
}
}
You’ll need to restart the agent for configuration changes to take effect.
Lumigo: Cloud Native Monitoring for AWS
By unifying logs, metrics, and traces into a single interface, Lumigo empowers developers and DevOps teams with comprehensive context for analyzing and resolving issues swiftly. It reduces the time spent on root cause analysis by 80% while dramatically cutting costs. With Lumigo, troubleshooting becomes fast, efficient, and cost-effective, delivering unparalleled visibility across the entire stack. Users can seamlessly search and analyze logs and click directly into the corresponding traces, accelerating resolution times while enjoying significant cost savings.
With Lumigo, users can:
Cut costs, not logs: Gain control over their observability expenses without compromising visibility. Say goodbye to toggling logs on and off in production.By consolidating logs and traces into one platform, Lumigo streamlines data aggregation, allowing you to eliminate duplicates and reduce the volume of required logs. This consolidation ultimately lowers overall costs.
Quickly get the answers you need with powerful SQL syntax: Simplify the search, filtering, aggregation, and visualization of logs using SQL for immediate access to pertinent troubleshooting information. Analyze logs effortlessly with interactive dashboards and intelligent data visualizations while gaining deep insights that provide a quick understanding of any issue.
Reduce troubleshooting time by over 80%: Lumigo automatically enriches traces with complete in-context request and response payloads and correlates them to the relevant logs and metrics. This enables developers to view logs in the context of the associated traces while seamlessly navigating from logs to traces and vice versa. Lumigo brings all your troubleshooting data into a single, correlated dashboard view.