Highly Scalable Signal Events Persistent Store
Store and serve millions of investment decision making signal events with sub-second response time
- Migrate legacy system that stored and served millions of investment decision making signal events to a more scalable environment.
- Enable user base to apply different combinations of date range filters on 10 years’ worth of signal events with very little engineering involvement.
- Avoid in-memory cache warmup during server start-up and reduce the start-up time from hours to minutes to serve user base as quickly as possible.
- Move away from in-memory cache to high performance schema-less data storage to create stateless environment that could scale horizontally.
- Reduce infrastructure cost by retiring high memory intensive servers and replacing them with commodity servers.
- Generate different analytical reports for internal curators on the signal events based on event type, date ranges, ticker symbols, sectors, and various filtering criteria.
- Reuse legacy signal event generation and serving process to control engineering development cost.
- Choose appropriate persistent store that can facilitate schema-less design, retrieve events based on different date ranges, and generate analytical reports for internal curators.
- Perform data quality checks between new and legacy system on millions of signal events.
- Meet the SLA of sub-second API payload response.
- Provide backward compatibility between new and legacy system until rollout is complete.
- Treselle’s Engineering team chose MongoDB for persistent store and Spring REST API as a microservice that provides all CRUD functionalities.
- Designed a hybrid schema model to store variety of signal events in a single collection with appropriate partition keys to accommodate sharding in the future.
- Provided controls to toggle the events to be served from in-memory and MongoDB during transition and testing times.
- Performed marshalling and un-marshalling of JSON response between actual API response payload and JSON stored in the MongoDB.
- Executed multiple SoapUI functional test cases and Gatling performance stress tests to ensure committed SLA was satisfied.
- Designed different reporting interfaces such as API-based clients, MongoDB Compass, and Metabase for internal curators to consume the analytical reports.
- Retired high memory intensive servers by turning off in-memory store and enabled servers to be started within 2 minutes by avoiding the cache warmup.
THE SOLUTION DIAGRAM: