Results

Highly Scalable Signal Events Persistent Store

Store and serve millions of investment decision making signal events with sub-second response time

BUSINESS GOALS:

  • Migrate legacy system that stored and served millions of investment decision making signal events to a more scalable environment.
  • Enable user base to apply different combinations of date range filters on 10 years’ worth of signal events with very little engineering involvement.
  • Avoid in-memory cache warmup during server start-up and reduce the start-up time from hours to minutes to serve user base as quickly as possible.
  • Move away from in-memory cache to high performance schema-less data storage to create stateless environment that could scale horizontally.
  • Reduce infrastructure cost by retiring high memory intensive servers and replacing them with commodity servers.
  • Generate different analytical reports for internal curators on the signal events based on event type, date ranges, ticker symbols, sectors, and various filtering criteria.

CHALLENGES:

  • Reuse legacy signal event generation and serving process to control engineering development cost.
  • Choose appropriate persistent store that can facilitate schema-less design, retrieve events based on different date ranges, and generate analytical reports for internal curators.
  • Perform data quality checks between new and legacy system on millions of signal events.
  • Meet the SLA of sub-second API payload response.
  • Provide backward compatibility between new and legacy system until rollout is complete.

THE SOLUTION:

  • Treselle’s Engineering team chose MongoDB for persistent store and Spring REST API as a microservice that provides all CRUD functionalities.
  • Designed a hybrid schema model to store variety of signal events in a single collection with appropriate partition keys to accommodate sharding in the future.
  • Provided controls to toggle the events to be served from in-memory and MongoDB during transition and testing times.
  • Performed marshalling and un-marshalling of JSON response between actual API response payload and JSON stored in the MongoDB.
  • Executed multiple SoapUI functional test cases and Gatling performance stress tests to ensure committed SLA was satisfied.
  • Designed different reporting interfaces such as API-based clients, MongoDB Compass, and Metabase for internal curators to consume the analytical reports.
  • Retired high memory intensive servers by turning off in-memory store and enabled servers to be started within 2 minutes by avoiding the cache warmup.

THE SOLUTION DIAGRAM:

 

pqs_se_mongo_case_study