Key Information

Register
Submit
The challenge is finished.

Challenge Overview

Project overview

The client would like to implement some inter-connect routes in Apache Camel to move data between systems, initiated via REST calls.

Technology

  • Java 11
  • Apache Camel 3.4.0
  • SpringBoot 2.3.1
  • Maven 3.8.8
  • Kafka 2.6.0

Flow

The caller (System A) will make an HTTP REST call to the Spring Boot application (your submission).  This call will be routed to Apache Camel.  You should implement Camel endpoints, routing, and processing to:
  • Handle an HTTP REST request that initiates processing to:
  • Make a TCP connect to a configured server
  • Pull data from the TCP socket according to the format of the data (provided in the forum)
  • Parse the data
  • Convert the data to JSON
  • Route the data to a Kafka endpoint in Camel
  • The final message will be put onto a configured Kafka topic
  • A general HTTP response and message will be sent to the HTTP REST caller


Request

The request to me REST API should look like:
  • GET `/cs_08`
  • GET `/cs_07`


TCP data

The data structure for the binary data can be found in an XLSX spreadsheet in the forum.  

Note that we are going to support two mappings for this challenge:
  • CS_07
  • CS_08

The overall flow will be the same - pull data from a TCP socket, convert it to JSON, publish to a Kafka topic.  The only differences will be the format of the input and output.  You’re encouraged to offer up an extensible solution that can be easily tweaked for further mappings in the future.


JSON output to Kafka

The CS_08 JSON message to Kafka will be a raw JSON object, with fields matching column "I" in the spreadsheet in the forum, like:

{
    "INTRADAY_ADD_MOD_DEL_FLAG":0,
    "Professor_CODE_IN_SEGMENT":"ASDF1",
    "Professor_FULL_NAME":"John Doe",
    ...
}

The CS_07 JSON message to Kafka will look similar:

{
    "INTRADAY_ADD_MOD_DEL_FLAG":0,
    "CONTRACT_TOKEN":12345,
    "INSTRUMENT_TYPE":"ASDF123",
    ...
}

Any field marked "<Not using>" can be left out of the JSON message


Apache Camel

Apache camel should be used, and route, endpoint, and processing should all be properly registered with the CamelContext in the SpringBoot app


REST response

The response to either REST call should look like:

Success

{
"status": 200,
"message": "Request is Successful ",
}

With an HTTP status code of 200

Error:

{
"timestamp": "2020-09-27T02:50:24.400+00:00",
"status": 500,
"errorCode”: “ERR_002”
"error": "Internal Server Error",
"message": " Internal Server Error",
}

HTTP status code of 400 and errorCode "ERR_001" for when an error has occurred. Most likely whenever wrong or incomplete TCP data has been received.
HTTP status code of 500 and errorCode "ERR_002" for when an error occurred when publishing to Kafka
HTTP status code of 503 and errorCode "ERR_003" for when Kafka is unavailable

Example TCP errors:
  • TCP connection not established
  • When socket connection is broken
  • Exception when creating socket connection
Example Kafka errors:
  • Unable to connect to Kafka Bootstrap server
  • No such topic found or invalid topic
  • Key not found or invalid key in Kafka
  • NotFoundException : Indicates that an operation attempted to modify or delete a connector or task that is not present on the worker.
  • RetriableException: An exception that indicates the operation can be reattempted.
  • IllegalWorkerStateException: Indicates that a method has been invoked illegally or at an invalid time by a connector or task.
Example processing errors:
  • Invalid input message
  • Data conversion exception like converting data into Json and binary.
  • If any mandatory input or value is null: NullPointerException


Retries

If there is a failure, we want to ensure that the process of pulling, parsing, and publishing is retried.  The number of times to retry should be configurable, defaulting to 3.  If the process failed too many times, we'll report the last error that caused a failure


Logging

Every request and response should be logged in INFO level for successful and failure case.
Develop a Spring AOP library that will log at:
  • The entry and exit of every call invocation.
  • Also, all errors need to be logged in ERROR level.
  • Input request.
  • Exception
  • Start of method notification
  • End of method notification
  • Conditions if applied
The logs to be produced in a format matching the attributes published in the forum.
 

Configuration

Logging details

TCP endpoint configuration:
  • Socket Connection Details
  • Host
  • Port
Kafka Configuration:
  • Kafka details
  • Kafka topic name
  • Bootstrap-server host and port

Links

Extensibility

You will notice that this challenge includes two mappings.  You are encouraged to offer an extensible solution that is easily tweaked / configured to support other mappings.  There will be a number of other integrations similar to this one potentially in the future, so we need to offer the cliennt an easily configurable solution that requires minor changes to deploy to target new data

Unit tests

Unit tests are *not required* for this challenge.

Scoring guidelines

Major requirements
  • REST endpoint, TCP processing, and Kafka publishing work
  • Mapping is easily configurable / extensible
Minor requirements
  • Documentation is complete
Submission

Please submit:
  • Code for the SpringBoot app that meets the requirements above
  • Unit tests that cover the application
  • README.md that describes how to deploy, test, and configure the app
  • Validation.md that describes how reviewers and the client can verify the app is functioning properly


Final Submission Guidelines

Please see above

ELIGIBLE EVENTS:

2021 Topcoder(R) Open

REVIEW STYLE:

Final Review:

Community Review Board

Approval:

User Sign-Off

SHARE:

ID: 30152359