Topcoder Challenge | Topcoder Community

Challenge Overview

Create a Kafka processor to CUD (Create / Update / Delete) data in elastic search based on the events received in Kafka bus

Topcoder is streamlining its APIs and in this project we will be upgrading the existing Project API to V5 standards
Project API powers Topcoder connect, Topcoder Challenge V5 API and supplies data for Direct as well

You could check out Challenge Processor ES for reference. Please note that the codebase shared is just to reference, the requirement for this processor may vary from the reference processor.
Source code of partially upgraded project API can be found here. Please make sure to check `v5-upgrade` branch.
Topcoder Kafka Event bus standards document is provided in the challenge forum which will be accessible after registration.

Create a Kafka Group Consumer and make it listen to 3 configurable topics
- project.notification.create
- project.notification.update
- project.notifcation.delete
Kafka consumer should support connecting to ES hosted in AWS as well.
Please refer this thread to learn more about the structure of Kafka message that will be used by this processor
Every Create / Update / Delete message will have the following structure.
- Create message will have all fields that are needed to create data in ES
- Update message will have fields which need to be updated in ES
- Delete message will just have the id of the data to be deleted in ES

Please check the ES indices creation script here and refer it for the ES indices structure. You could use the same script to create indices in ES. There will be minor update in the script to add another nested attribute in metadata index. It is posted in the forum.
3 ES indices are involved in this processor. One for Project related data, one for Timeline related data and one for metadata.
Based on the `resource` field in the payload, we will choose which ES index to use. You need to maintain a mapping between available resource fields and ES index to use in a constant file
Each index has a different structure and CUD logic in each index could be different, so please make sure to check the ES indices structure and create CUD logic accordingly.
Events received in `project.notifcation.create` topic should be used to create data in ES
Events received in `project.notifcation.update` topic should be used to update data in ES
Events received in `project.notification.delete` topic should be used to delete data in ES

You would need to create E2E tests for this processor. If there is a piece of code which can’t be tested by E2E tests, you could create unit tests for those piece of code and execute it as part of E2E tests itself.
Ideally the code coverage is expected to be 100% for all files. If there is difficulty in achieving it, please discuss the same in forum
We expect that the tests should be robust enough to capture bugs in future development
Use separate ES indices for testing with the suffix `_test`. E.g. if the actual index name is projects, use the index name `projects_test` for testing.
Use Mocha and Chai for testing and Istanbul for code coverage report
Use Sinon for mocking if there is a need to mock any scenario
Use the commands `npm run test` for running E2E tests and `npm run test:cov` for running E2E tests with code coverage report

You may reuse the deployment document from the reference processor but please make sure to update the content according to the requirements of this processor

Please create a brief validation document to verify the new processor created as part of this challenge.

Use Async / Await pattern in the codebase
Please try to reuse the code as much as you can instead of creating redundant code for processing similar messages.
Ensure that there are no lint errors and every test pass.
If you have any doubts, Please feel free to post it in the forum. We will be happy to resolve your queries as soon as possible.

Zip your codebase with validation document and submit the zip file using the submit link in the challenge.
After submitting, please ensure that you are able to download your submission from Online Review