Challenge Overview
Hercules wishes to build a system to serve data queries. The system will be designed to handle heavy load of requests that are mostly read not write.
The system will load information about entities (represented in JSON format, so the structure is very flexible), store that in DB and then allow user to retrieve attributes of those entities. Also, the system will provide operational statistics to the user.
This design provides the Executor class which acts as the main harness of feed reading implementation.
It uses the FeedMonitorService to continuously monitor data feed URLs and ad them as Zookeeper znodes for further processing.
It uses the HeartbeatMonitorService to keep track of znodes which need to be processed again because their earlier processing was not successful.
It uses the DataIngestionWatcher which acts as an observer of the Zookeeper znodes and watches for state changes like znode creation or modifications. It is responsible for reading and persisting the entities to database. It uses many workers to perform these tasks concurrently. The DataIngestionWorker is the worker class.
The design is a fault tolerant design in the sense that if feeds are not able to processed properly, then they are rescheduled and tried again. Any changes made by the unsuccessful processing are rolled back.
Final Submission Guidelines
N/A