Challenge Overview
We have a client that is developing their own IoT devices management platform. Naturally they get huge amouts of data from the devices and they need scalable clustered infrastructure to process all that streaming data.
In this challenge we'll create a sample Scala program to process data from Kafka queue using Akka. The actual data processing is not all that important in this challenge, but the accent is on using as much as Akka's built-in features as possible, instead of writing custom code to support the clustering. Here are the requirements:
- Create an Akka based cluster program in Scala, with a minimum of 3 nodes, to ingest data from Kafka. The program should be an implementation of a simple Actor model or non Actor model (your choice) to ingest data and process (simply log) the data. We recommend you choose an appropriate Actor model, but in either case add a simple description (one-two sentences) on what model you chose and why. You will need to use the Akka’s built in clustering capability to manage the distributed nodes.
- Set up your own Kafka service with at least a few partitions and write a short script to push simple strings to simulate the data from the devices.
- Develop a simple algorithm to shard the data from the multiple Kafka partitions to the number of Akka nodes. For example, if there are 4 partitions in Kafka topic, and 3 nodes in the Akka cluster, the assignment can be 2 (Kafka partitions), 1, and 1 on each node. The cluster can simply log the ingested data to the log file to simulate the process.
- Demonstrate the capability that when a node is killed and/or added, the Kafka partition pipelines will be re-balanced to the remaining nodes accordingly. Show the results in the log file.
You can use https://github.com/akka/reactive-kafka for stream processing, and you can also suggest other libraries if necessary.
Useful links
There are many available resources/articles highlighting useful patterns in Akka. Make sure to check out this this and this
Useful links
There are many available resources/articles highlighting useful patterns in Akka. Make sure to check out this this and this
Final Submission Guidelines
Submit the complete source codeWrite a Readme.md document detailing build,configuration and deployment steps
Submit a short video demonstrating the required features.