Challenge Overview

Welcome to the Pioneer Data Ingestion Tool API Implementation Challenge

In previous challenges we have defined the high level architecture for the event streaming platform and created api specification for data ingestion tool. In this challenge we will implement the defined API. Later we will use this api to build a frontend app for the end users

Background

We are building a scalable event streaming platform that will be used to provide customers with real time notifications on events that occurred in the system. Specific use case that we’re targeting is in the financial sector, but the platform will be designed as a generic event streaming solution. Scalability is a major concern as the solution would be used to process millions of events daily.

Event streaming platform will consist of three parts:

1. Producer - that ingests source data into Kafka cluster

2. Aggregation and filtering of the source data

3. Delivery of the events to end users

 

Most of the source data is generated in real time (ex Bob sent $5 to Alice) and some data is generated during the night work process (ex balance for Bob’s account is $10). Regardless of how the data is generated, it is available in Kafka topics and will be used by our Producer to send event notifications. 

 

See the project architecture document for more information (posted in challenge forums). You should read and understand the existing architecture before reading the challenge requirements below.

 

Task Details

Your task in this challenge is to implement the data ingestion tool based on the api specification provided in the forums. All api endpoints are in scope. A lot of the endpoints will call external apis (Kafka) - those details are described in api specification.

Technology stack for the api is Spring Boot, JPA/Hibernate , Postgres.

 

Create a docker file for the application and docker-compose script to start all the services (api, db, kafka). Unit tests are not in scope. Create a Postman collection for verifying api endpoints and create some test data to demonstrate the api usage.

 

Submission Guidelines

 

These should be the contents of your submission:

1. Codebase of the ingestion tool api

2. Readme with configuration and deployment details

3. Postman collection for verifying the api endpoints.

Final Submission Guidelines

See above