Unified Search & Log Aggregation in Microservices on Cloud
A common Search framework for end users, business users, and operation team to perform search and track necessary metrics at high scale
- Provide high performant free text and facet search for end users.
- Enable business analysts and marketing team to gain insights into user access metrics to perform necessary segmentation for targetted marketing.
- Enable operations team to manage and monitor overall microservices health, request-response thik time, multiple service interaction, microservice lineage, error codes, and quickly alert on metrics that are beyond threshold limit.
- Ability to auto scale the infrastructure without manual intervention as the data grows.
- Identifying most appropriate log readers for the services running on docker containers.
- Modeling the log payload that could track necessary request parameters along with instrumentation metrics.
- Provide a mechanism for microservice lineage to track complex request & response cycle that involves multiple service interactions.
- Choosing a common framework to satisfy different user needs for different purposes such as end user searching, marketing team, business analysts team, and operations team.
- Not enough details on the capacity of the data growth.
- Chose AWS ElasticSearch and performed multiple proof of concepts to identify the proper log reader that includes Graylog Extended Log Format, Filebeat, Fluentd, Journald, Syslog, and awslogs.
- awslog reader was chosen based on the ease of use with docker containers and performance.
- Modeled the log format that contains correlation id and other request & response data to track the microservice lineage.
- Lambda was used to interpret any metrics that are out of threshold and sends notification to SQS to alert respective operations team and finally insert the log stream into ElasticSearch
- Archive log data at different intervals based on the nature of the service logging it and upload it to Glacier for future use
- Created multiple dashboards for business, marketing, and operations team using Kibana visualization
THE SOLUTION DIAGRAM: