Challenge Overview
Challenge Objectives
Create a Nodejs (version 8) based processor that integrates with ElasticSearch and Kafka specifically for Member API
Technology Stack
Project Background
Code Access
Submission requirements
1). Create a Processor app (Major Requirement)
- Write a nodejs app that interfaces with Kafka and ElasticSearch.
- The app will listen to these following topic on kafka:
-- member.action.profile.create
-- member.action.profile.update
-- member.action.profile.trait.create
-- member.action.profile.trait.update
-- member.action.profile.photo.create
-- member.action.profile.photo.update
-- member.action.profile.delete
-- member.action.profile.trait.delete
- When a message arrives on either of the above topics, the processor will parse the payload and create / update / delete data in Elasticsearch
- So, taking an example, the processor will first receive a message in member.action.profile.create type to create data in ElasticSearch.
- The processor will then subsequently receive the update and delete messages for the data, which it will update accordingly in ElasticSearch.
- Pay attention each data format are different
- The data to create / update in ElasticSearch will be present in the message received in Kafka itself.
- The format of the message received need use same structure same like submission-processor-es structure
2). Write Unit Tests (Major Requirement)
- Provide unit tests for the processor
Deployment Guide and Validation Document
Submit a README file on how to deploy and test your app. Kindly provide verification steps too.
Important Notes
- Use the official Elasticsearch library for interfacing with ElasticSearch.
- Kindly use the no-kafka library for interfacing with Kafka
- For tests, kindly use the mocha library along with chai
- Kindly use the standard library to lint your code
Some elasticsearch specific points:
-- The index to use in elasticsearch needs to be a configuration read off an environment variable.
- There will be 8 topics. The topics to use will be present in the message received from kafka
- Under that kafka topic, you can get a message related to any of the topic above.
- For update / delete operations, you can use the id attribute present in the message payload to identify which document to update / delete.
- Additionally, you can refer to the Topcoder Member API Swagger, which defines the payload for each of the 8 that correspond to the document types.
- This should give you an idea about the type of the payload data present in the message.
- There will be no read operations. You will only be creating and updating (including deleting) data in elasticsearch.
- Please zip and upload the code base for the app
- Provide a README file containing instructions on how to deploy your app as well how to run the tests.
- Verify that there are no lint errors before you submit
Create a Nodejs (version 8) based processor that integrates with ElasticSearch and Kafka specifically for Member API
Technology Stack
- Back-end only app
- Nodejs version 8
- ElasticSearch
- Kafka
Project Background
- Topcoder is starting to streamline its APIs and we are in the process of upgrading our existing APIs to version 5.
- One of the APIs is called Member API. This will be working with a processor app (the one in scope for this contest) to manage data.
- This is an ongoing project which is being enhanced periodically.
Code Access
- We just completed similar submission-processor-es from another challenge. Basically you can copy same structure but need address the different topics for Member API
- Check submission-processor-es repo here: https://github.com/topcoder-platform/submission-processor-es/tree/develop
Submission requirements
1). Create a Processor app (Major Requirement)
- Write a nodejs app that interfaces with Kafka and ElasticSearch.
- The app will listen to these following topic on kafka:
-- member.action.profile.create
-- member.action.profile.update
-- member.action.profile.trait.create
-- member.action.profile.trait.update
-- member.action.profile.photo.create
-- member.action.profile.photo.update
-- member.action.profile.delete
-- member.action.profile.trait.delete
- When a message arrives on either of the above topics, the processor will parse the payload and create / update / delete data in Elasticsearch
- So, taking an example, the processor will first receive a message in member.action.profile.create type to create data in ElasticSearch.
- The processor will then subsequently receive the update and delete messages for the data, which it will update accordingly in ElasticSearch.
- Pay attention each data format are different
- The data to create / update in ElasticSearch will be present in the message received in Kafka itself.
- The format of the message received need use same structure same like submission-processor-es structure
2). Write Unit Tests (Major Requirement)
- Provide unit tests for the processor
Deployment Guide and Validation Document
Submit a README file on how to deploy and test your app. Kindly provide verification steps too.
Important Notes
- Use the official Elasticsearch library for interfacing with ElasticSearch.
- Kindly use the no-kafka library for interfacing with Kafka
- For tests, kindly use the mocha library along with chai
- Kindly use the standard library to lint your code
Some elasticsearch specific points:
-- The index to use in elasticsearch needs to be a configuration read off an environment variable.
- There will be 8 topics. The topics to use will be present in the message received from kafka
- Under that kafka topic, you can get a message related to any of the topic above.
- For update / delete operations, you can use the id attribute present in the message payload to identify which document to update / delete.
- Additionally, you can refer to the Topcoder Member API Swagger, which defines the payload for each of the 8 that correspond to the document types.
- This should give you an idea about the type of the payload data present in the message.
- There will be no read operations. You will only be creating and updating (including deleting) data in elasticsearch.
Final Submission Guidelines
Submission deliverables- Please zip and upload the code base for the app
- Provide a README file containing instructions on how to deploy your app as well how to run the tests.
- Verify that there are no lint errors before you submit