Challenge Overview
Context
Project Context
Our client wants to create an employee management system to determine employees that are no longer working on active projects and to understand their qualifications and expertise for suitability in other projects
Challenge Context
Create a nodejs based processor that listens to topics on kafka and processes the messages that arrive.
Expected Outcome
A nodejs processor that listens for messages sent on multiple kafka topics. When a message arrives, it will process the message based on the topic it has arrived under. It will update the corresponding data in elasticsearch.
Challenge Details
Individual requirements
We are currently building an API for multiple data models. Currently, these APIs are using Amazon QLDB as their database, but soon, we will also integrate Elasticsearch with the API such that, all non read operations will utilize QLDB and all data will be read from Elasticsearch (before falling back to QLDB).
To populate the data into Elasticsearch, the API, after updating the QLDB database, will post the data to Topcoder’s Bus API, which sends the data on a kafka bus. You are not expected to understand the Bus API.
This is where the elasticsearch based processor, that you will be creating as part of this contest, comes into picture. You need to, as part of this contest, create a Nodejs based processor that listens to multiple topics on kafka, and once the messages arrives, it processes these to extract the data that the API passed earlier, and stores / updates the data in Elasticsearch.
We currently have a similar setup for Topcoder’s other APIs. Examples:
-
The submission api and the submission elasticsearch processor. In this case the API interfaces with Dynamodb and posts the data to Kafka, which is then processed by the processor and the data is stored / updated in Elasticsearch. The API itself uses Elasticsearch for read operations (before falling back on Dynamodb).
-
The lookups api and the lookup api processor. Similar to the submission api and submission processor. Different data.
For this contest, you are creating the processor part of the workflow - which will listen on the kafka topics, process the message, and store / update the data in Elasticsearch.
Development Assets
-
No code to share. This is the first contest for the processor
-
We will however be sharing an ERD model diagram and the swagger specification for the API in the contest forum. These will give you information about the type of data that the processor will have to handle and thus help you in defining the index.
Technology Stack
-
Nodejs version 12
-
Kafka version 2.x
-
Elasticsearch
Important Notes
-
Use standard for your lint tool
-
Use the async await pattern
-
You need to provide a Docker based deployment, in addition to local deployment
-
Tests are needed. You need to use mocha and chai for the tests
-
Soft delete is not to be supported for now
Deployment guide and validation document
You need to provide a deployment guide (in markdown) that describes how to deploy the processor locally and through docker. Additionally, you also need to provide a verification document (in markdown) with steps on how to verify that the processor runs as expected.
Scorecard Aid
Judging Criteria
-
A major requirement of this contest is to correctly determine the elasticsearch index. Since we do not have an API yet, you get the benefit of deciding the index to use for elasticsearch, based on the ERD model and / or swagger specification.
-
Another major requirement is to be able to correctly identify the action to be carried out, based on the topic. Certain topics are associated with data insertion, others with data update (including data delete). The processor should correctly carry out the action as intended.
-
Tests will have their own dedicated section in the scorecard
Final Submission Guidelines
Submit a zip file that contains the source code