Topcoder Submission Processor - Integrate With ElasticSearch and Kafka

Challenge Objectives

• Create a Nodejs (version 8) based processor that integrates with ElasticSearch and Kafka

 

Project Background

• Topcoder is starting to streamline its APIs and we are in the process of upgrading our existing APIs to version 5.

• One of the APIs is called Submission API. This will be working with a processor app (the one in scope for this contest) to manage data.

• This is an ongoing project which is being enhanced periodically.

 

Technology Stack

• This is a back-end only app

• We are using Nodejs version 8

• ElasticSearch

• Kafka

 

Code Access

• No code will be provided. You are building the processor app from scratch

 

Individual Requirements

Create a Processor app (Major Requirement)

• Write a nodejs app that interfaces with Kafka and ElasticSearch.

• The app will listen to the following topics on kafka:

  • submission.notification.create

  • submission.notification.update

  • submission.notification.delete

• When a message arrives on either of the above topics, the processor will parse the payload and create / update / delete data in Elasticsearch

• So, taking an example, the processor will first receive a message in submission.notification.create to create data in ElasticSearch. The processor will then subsequently receive the update and delete messages for the data, which it will update accordingly in ElasticSearch.

• The data to create / update in ElasticSearch will be present in the message received in Kafka itself.

• The format of the message received will be shared in the contest forum.

Write Unit Tests (Major Requirement)

• Provide unit tests for the processor

 

Deployment Guide and Validation Document

• Submit a README file on how to deploy and test your app. Kindly provide verification steps too.
   

Important Notes

• Kindly use the official Elasticsearch library for interfacing with ElasticSearch.

• Kindly use the no-kafka library for interfacing with Kafka

• For tests, kindly use the mocha library along with chai

• Kindly use the standard library to lint your code

• Certain elasticsearch specific points:

  • The index to use in elasticsearch needs to be a configuration read off an environment variable.

  • There will be 4 document types - submission, review, reviewType and reviewSummation. The document type to use will be present in the message received from kafka

  • Under the three kafka topics, you can get a message related to any of the 4 document types above. For update / delete operations, you can use the id attribute present in the message payload to identify which document to update / delete. Additionally, you can refer to the specification, which defines the payload for each of the 4 resources that correspond to the document types. This should give you an idea about the type of the payload data present in the message.

• There will be no read operations. You will only be creating and updating (including deleting) data in elasticsearch.

Final Submission Guidelines

• Please zip and upload the code base for the app

• Provide a README file containing instructions on how to deploy your app as well how to run the tests.

• Verify that there are no lint errors before you submit