Challenge Overview
Overview
Topcoder is using Elasticsearch to power the new generation of the challenge search functionality.
In previous challenges we have built the infrastructure to generate events in Kafka for any changes to project, project_info, project_spec, project_studio_specification, prize, submissions, project_phase, and resource.
The Challenge Events Design spec (see in forum) shows the format of the events generated in Kafka.
The attached docker-compose file and assets will allow you to execute locally all components necessary for the solution, this includes:
- Informix: container, which has all triggers and tables necessary for the events to be generated
- Kafka: where events are generated
- Elasticsearch: where the data is created / updated
- Logstash initial load: performs the initial load of the data from Informix to Elasticsearch
- Logstash kafka: generates kafka events from informix to Kafka.
The plan is to execute the initial load once before deploying the solution, after that we'll need to listen to kafka events in order to update elasticsearch whenever challenges are created, updated, or deleted.
Scope
The task for this challenge is to create a logstash configuration that listens to kafka events and inserts, updates, and deletes challenges in elasticsearch based on the data on those events.
This new logstash configuration can NOT use informix directly, all data should be available in the kafka event, if it is not please let us know, we will fix the logstash kafka configuration.
Note that when the event being generated is related with project, project_info, project_spec, and project_studio_specification table, the kafka event will contain all challenge related data, but when the event is related with prize, submissions, project_phase, and resource there will be only information specific to those tables. This will require looking up the challenge in elasticsearch and updating, inserting, or deleting items in sub objects of the challenge. Refer to the challenge data structure in elasticsearch for more information, Challenge contains a list of phases, prizes, users, etc.
About Environment
We have provided a docker based solution, so you can execute locally and test locally. Please check the doucument in forum.
Final Submission Guidelines
- Updated solution based on the submission in forum
- Updated deployment guide and Verification Documentation