The open source applications described in this article are available from Lumigo’s Github repository.
If you look around the Serverless Application Repository console, you will find a number of applications that can help you ship logs from CloudWatch Logs to external services. One such example is the LogzioCloudWatchShipper application below.
However, after I deploy these serverless applications I still need to go to the CloudWatch Logs console and subscribe my log groups to the relevant Lambda function. Since every function has a matching log group in CloudWatch Logs, you need to do this all the time. This is a chore and one that should be automated away. As I discussed in a previous post, you can automate this process using CloudWatch Events and Lambda. I even shared the code to do that so you can deploy them into your account.
However, as I later discovered, this approach does not scale well because of the regional concurrency limits on Lambda. Instead, you should stream the logs to a Kinesis Data Stream or Firehose Stream first, as I outlined in this post.
Auto-subscribe log groups to an ARN
Today, we are going to make it easy for you to adopt either of these approaches through a Serverless application called auto-subscribe-log-group-to-arn.
To deploy it to your AWS account, head over to the SAR console and search for “auto-subscribe-log-group-to-arn”. Make sure you tick the box “Show apps that create custom IAM roles or resource policies”.
For this application to work, we need to know the Destination ARN, which point to a Lambda function, Kinesis Data Stream or a Firehose Delivery Stream. The application would handle the complexity related to permissions:
- It will add the necessary Lambda permission to allow CloudWatch Logs to invoke the destination Lambda function.
- It will pass along an IAM role to allow CloudWatch Logs to push log events to either Kinesis or Firehose.
Additionally, you can customize the configuration of the subscription filter by overriding the FilterName and FilterPattern parameters.
You can also specify a Prefix. If you only want to capture Lambda function logs, then you should set this to “/aws/lambda”. This allows you to forward logs for different services to different destination ARNs by deploying multiple instances of this application. To do this, you need to give each instance of the application a different “Application name”. This would create a separate CloudFormation stack for each.
Once deployed, this application would create two related functions.
The first function is triggered by the following CloudWatch Event pattern. Whenever a log group is created, the function would run and subscribe it to the destination ARN if it matches the configured prefix.
The second function is triggered by a CloudWatch Schedule and runs every hour. It will iterate through all the log groups and make sure they’re all subscribed to the destination ARN. This saves you from manual efforts to deal with existing log groups when you first deploy the application. It also helps you update all the log groups automatically when you decide to change the destination ARN (by redeploying the application with a different configuration).
Auto-update retention policy
CloudWatch Logs carries a storage cost of $0.03 per GB per month. By default, the retention policy for all log groups is to never expire them in CloudWatch Logs. Which means your storage cost would continue to grow month-on-month.
If you’re shipping the logs elsewhere already, and paying for processing and storage cost with the external service, then it doesn’t make sense for you to keep them in CloudWatch Logs also, considering the cost implications.
Today, we’re announcing a second application, to help you automate the process of updating the retention policy of your logs.
This application works the same way as auto-subscribe-log-group-to-arn. It has two functions, one for updating log groups whenever they are created, another for dealing with existing log groups.
For these functions to work, we just need to know for how many days you’d would like to keep the logs in CloudWatch Logs.
What happens when we publish a new version?
When a new version of these applications is published, the resources that have been deployed in your account remain unchanged. To get the latest version, you have to follow the steps in this post and deploy the latest version(s) of applications again.
Doing so will trigger a CloudFormation update against the existing stacks and update the resources accordingly.
How can you contribute to this project or raise feature requests?
The code for these applications is available on Github. You can contribute towards the project via pull requests and you can raise feature requests via Github issues.
So that’s it! I hope you find these serverless applications useful. We look forward to hearing your feedback. And let us know of any other chores that you’d like us to help you automate away through serverless applications.