Topcoder - Kafka Stream Implementation - Ideation Challenge

Challenge Objectives

Design a detailed solution of how to introduce Kafka streams in our client’s existing systems. In addition to the integration details, the solution must also satisfy the security and performance constraints in place.

Tech Stack

• Apache Kafka

Detailed Requirements

For this ideation challenge, you need to create a detailed architecture design of how to introduce Kafka streams in our client’s existing systems in order to post and consume events from topics.

 

The points to be covered in the overall architecture design:

• Implement a secure way to post and consume events from topics

  • Common options/patterns/frameworks for implementing ABAC and/or RBAC against topics

  • Authentication strategy:

    • OpenID Connect (1st preference)

    • oAuth2 (2nd preference)

    • Other alternatives, in case none of the above works

• Trade-offs for using individual schema v/s multiple schemas per topics

• Performance impact for using individual schema v/s shared schema

• Security impact for using individual schema v/s shared schema

• 3 options for `avro & json based schema registries` and their pros and cons

  • Schema enforcement must

    • Reject malformed messages

    • Return intelligent error messages to submitter

    • Not persist any errored messages to any topics

• Define topic structure strategy and rationale

 

The points to be considered while designing the architecture:

• All the different applications, producing messages, will be using native Kafka libs to post the messages

• Assume an already running Kafka cluster with monitoring in place and basic administration knowledge already understood.

• We will start with a few (6) and eventually have 100s of producers.

• There will be multiple applications with multiple coding languages

• The volume will be 1000s of messages per day

• Kafka instance running in Kubernetes and will be behind a load balancer (not directly accessible)

 

Judging criteria

All submissions will be reviewed and evaluated subjectively by the challenge copilot and the client and there will be no appeals phase.

 

The following metrics will be considered during the review:

• Completeness of the proposed solution

• How well have you documented the proposed solution

• The feasibility of the proposed solution

What to Submit

The deliverable would be the system design and architecture documents along with the preferable tech stack details.

 

Please submit your proposed solution in both DOCX and PDF formats.

Final Submission Guidelines

Please read above