Challenge Overview
Recently, we have built a system that will push challenge related information into elasticsearch. It basically consist of three pieces.
1. Trigger then monitoring the tables changes and generate records into project_event_log table, here are the related tables ( project_info, project, project_phase, resource, prize, project_spec, project_studio_specification)
2. A logstash config that read records from project_event_log table and generate kafka events for later processing (conf/challenges-kafka.conf.j2)
3. Another logstash config that read events from kafka queue and update the elasticsearch (conf/challenges-es-feeder.conf.j2)
But current implementation has several problems and failure needs to fix.
For this challenge we'd like you to fix the problems and do more testing to make sure the functionality is working proper for all cases. Please check the forum attachment for detail of the problems.
For problem analsys, you can increase the internval for logstash execution, so we can check
1. records are properly inserted in to project_event_log table
2. events are properly published to the kafka queue
3. the elasticsearch records are properly updated.
Final Submission Guidelines
- Updated submission that fulfills the requirements
Complete set of test files and steps to make sure the application is running properly.
Explanation about the cause of problems, to see if it is a common pattern needs to be addressed.
Verification Steps for fixing of the problems.