Challenge API - Kafka Integration Phase 2

Introduction

Welcome to the Phase 2 of the Challenge API-Kafka Integration updates. In this challenge we’re going to take our CRUD events produced by our Challenge API and push them onto our Kafka event bus so that we can consume them downstream by other services. In this challenge we’re going to continue building out all the other events we need. Currently, from Phase 1, we only implemented events for registration/unregistration. These can be used as examples of how to implement the rest of the events required.

 

OBJECTIVE

Phase 1 (Already completed):

We have migrated User registration & Unregistration events to Kafka topic by leveraging Event Bus API. We have used very basic payload structure in this phase (Completed).

Phase 2 (Scope of this challenge):

1) We will modify User registration & Unregistration structure to comply with new json structure which is finalized (Refer Event payload structure below).

2) We have identified more events from Challenge API (Check ChallengeAPI-Kafka_Integration_Phase2_EventsList.docx attached in the forums) and start pushing them to Kafka by leveraging Eventbus API with finalized event structure.
 

You may refer changes that we made into phase 1 at below Pull request.
https://github.com/appirio-tech/ap-challenge-microservice/pull/250

Check the forums for steps to follow to get access to the source code repository.

 

Workflow:

For an overview of the workflow of the overall EventBus Kafka integration, please refer to the following diagram : https://drive.google.com/file/d/1RFScVK_kaJfzlVrLjuFD6Q4Rqn6oy-R3/view

 

Tasks:

1. A common structure / classes to push event data to Eventbus (which in turns send payload to Kafka topic) is ready. We need to leverage them and write generic code to push event data to Kafka topic.

2. Kafka config parameter should be configurable (eventBus URL, Auth token & topic name)

3. Events (services) that we want to push to Kafka topic (same like phase 1 implementation) are listed in the ChallengeAPI-Kafka_Integration_Phase2_EventsList.docx attached in the forums.

4. Call eventBus/events web service with payload (Refer Event payload structure below)

5. Prepare test cases (postman collection) for all the services that you create/modify.

6. Developer documentation needs to be submitted along with final submission which may cover overall informations about classes purpose, structure that created / modified.

   This includes updating/adding the documentation for the classes introduced in phase 1 ( EventBusServiceClient.java, RegistrationMessage.java, TopcoderEventMessage.java, UnregistrationMessage.java and UserChallengePair.java)

7. Unit testing on all updated/newly created code is required.{C}{C}{C}{C}

 

Event payload structure

• Format: JSON

• Structure:

{
"topic": "string",

"originator": "string",

"timestamp": "ISO 8601 timestamp",

"mime-type": "string",
"payload": "object"
}

 

• topic : use “kafka.queue.java.test” for now.

• originator : For testing purpose, use your Topcoder handle as the originator in order to differentiate the events from events sent by other members.

• payload : The payload depends on the event, it is a JSON object which holds all the information sent to the REST endpoint corresponding to the event. Additionaly to any information related to the event.
  
   

• Example:
  {

  "topic": "kafka.queue.java.test",

  "originator": "tc-project-api",

  "timestamp": "2018-02-14T00:00:00",

  "mime-type": "application/json",

  "payload": {

  "topicId": "20668",

  "postId": 71828,

  "postContent": "submission phase Completed",

  "userId": 23200120,

  "projectId": "3600",

  "initiatorUserId": 23200120

  }

  }
  
   

  To check the Topcoder dev Kafka queue, you can use the following tool https://github.com/BadalZ/event-test-util

  As mentioned above, you can differentiate your events by checking the “originator” field of the payload.

  E.g. for handle name - bzcoder event should be like..

  {

  "topic":"notifications.kafka.queue.java.test",

  "originator":"bzcoder",

  "timestamp":"Sat Mar 31 10:09:43 EDT 2018",

  "mime-type":"application/json",

  "payload":"{\"date\":\"Sat Mar 31 10:09:43 EDT 2018\",\"projectId\":30064205,\"phaseId\":902588,\"phaseTypeName\":\"Iterative Review\",\"state\":\"END\",\"operator\":\"22841596\"}"

  }

Final Submission Guidelines

• Updated Code and verification steps.

• You should follow the same structure that we have developed in Phase 1 : https://github.com/appirio-tech/ap-challenge-microservice/pull/250

• Winner will be required to make a pull request against this branch: https://github.com/appirio-tech/ap-challenge-microservice/tree/kafkaEvents