Results

Unified Search & Log Aggregation in Microservices on Cloud

A common Search framework for end users, business users, and operation team to perform search and track necessary metrics at high scale

BUSINESS GOALS:

  • Provide high performant free text and facet search for end users.
  • Enable business analysts and marketing team to gain insights into user access metrics to perform necessary segmentation for targetted marketing.
  • Enable operations team to manage and monitor overall microservices health, request-response thik time, multiple service interaction, microservice lineage, error codes, and quickly alert on metrics that are beyond threshold limit.
  • Ability to auto scale the infrastructure without manual intervention as the data grows.

CHALLENGES:

  • Identifying most appropriate log readers for the services running on docker containers.
  • Modeling the log payload that could track necessary request parameters along with instrumentation metrics.
  • Provide a mechanism for microservice lineage to track complex request & response cycle that involves multiple service interactions.
  • Choosing a common framework to satisfy different user needs for different purposes such as end user searching, marketing team, business analysts team, and operations team.
  • Not enough details on the capacity of the data growth.

THE SOLUTION:

  • Chose AWS ElasticSearch and performed multiple proof of concepts to identify the proper log reader that includes Graylog Extended Log Format, Filebeat, Fluentd, Journald, Syslog, and awslogs.
  • awslog reader was chosen based on the ease of use with docker containers and performance.
  • Modeled the log format that contains correlation id and other request & response data to track the microservice lineage.
  • Lambda was used to interpret any metrics that are out of threshold and sends notification to SQS to alert respective operations team and finally insert the log stream into ElasticSearch
  • Archive log data at different intervals based on the nature of the service logging it and upload it to Glacier for future use
  • Created multiple dashboards for business, marketing, and operations team using Kibana visualization

THE SOLUTION DIAGRAM:

logging_es_microservice