How to push logs to ElasticSearch and Cloudwatch from ECS-fargate using FluentBit?

This blog demonstrates how the logs are pushed to CloudWatch and Elasticsearch by using AWS FireLens with the FluentBit plugin.

• Fluent Bit is an open-source Log Processor and Forwarder which allows you to collect any data like metrics and logs from different sources, enrich them with filters, and send them to multiple destinations.
• FireLens is a new log driver for ECS tasks where you can deploy a Fluent Bit( or a Fluentd) sidecar with the task and route logs to it. Using AWS FireLens, we can direct container logs to storage and analytics tools without modifying our deployment scripts. With a few configuration updates on AWS Fargate, select the destination and optionally define filters to instruct FireLens to send container logs to where they are needed.

How to push logs into CW & ES?

With the Fluent Bit plugin for AWS container image, you can route logs to Amazon CloudWatch and multiple destinations like Elasticsearch, Amazon S3, Amazon Kinesis Data Streams, and partner tools by using multiple outputs in the fluent bit image configuration as below.

In the current architecture, I declared the variables inside the image configuration and passed their values from the task definition so that you don’t need to modify the image for each microservice. you just need to modify the task definition.

• For structured logs in both CloudWatch and Elasticsearch, I have used JSON parser and filter in fluent bit image configuration. I have written PARSER config in a separate file and I used that file in the log destinations.conf file.

• In the task definition part, add the FluentBit configuration to route logs to CloudWatch and ElasticSearch.

References :

1. https://aws.amazon.com/blogs/opensource/centralized-container-logging-fluent-bit/
2. https://github.com/aws/aws-for-fluent-bit
3. https://docs.amazonaws.cn/en_us/AmazonECS/latest/userguide/firelens-using-fluentbit.html