Proactive Device Monitoring - Scaling Infrastructure

Overview

 

Ready for something a little different?  We're looking for folks experienced with delivering production level cloud applications. Know how to take your POC application from 1 user to a million? We want you!

 

The client is looking to take a MVP product created by topcoder to the next level.  The application consists of a monitoring agent that collects data (e.g. performance metrics/statistics data) from Windows based laptop/desktop computers and periodically connects to a backend API which stores the data in HBase. Administrators can use this data to make decisions regarding new laptops, additional memory, software patches and upgrades, etc.

 

Note - this challenge is primarily about scaling the existing application, not about re-designing it. The focus is on getting it scaled up to a large number of users.

 

The current architecture is running in "proof of concept" mode and contains the following pieces:

 

• A node / passenger based service which accepts data from the agents and then logs and stores that information in HBase, either directly or via openTSDB (for time-sensitive data points)

• An agent which runs on the end user's Windows computer, collects data, and forwards that data to the API endpoint for processing

 

RIght now the API server, HBase database, and openTSDB are all running on a single Amazon EC2 Linux instance.

 

The POC is currently serving about 40 end user systems. The eventual goal is substantially larger - 100,000 end user systems. At present, the agent collects information from the end user system once every 10 minutes. Each end user system (agent) submits, on average 50 MB of data per day.

 

Challenge Deliverables

 

This challenge is primarily about scaling. The client would like to roll this out in phases targeting increasing numbers of end user systems. Your submission needs to provide a clear plan for scaling from the current 40 clients to each of the following volumes:

 

• "Tier Zero" POC - 40 clients

• "Tier 1" 10k clients

• "Tier 2" 20k clients

• "Tier 3" 50k clients

• "Tier 4" 100k clients

 

We realize there will be a good deal of estimating going on here and you'll be asked to draw from personal experience, best practices, and targeted research. Please don't be afraid to use the forums for a deeper discussion!

 

The primary deliverable for this challenge is a detailed infrastructure plan for scaling this application. Please keep deployment and hosting costs in mind throughout this process and aim to keep them as reasonable as possible. Your submission should including the following Sections:

 

1. The systems and hosting architecture for the "scaled up" version of this service (using the target "Tiers" above).

   1. What needs to be split out into separate servers?

   2. How many of each?

   3. What are the likely bottlenecks and how do we overcome them?

   4. Will the basic software needs to be changed when moving from single-server to multi-server (and potential load-balancing)?

   5. Recommendations for hosting - is Amazon the right place for this? What other alternatives should be considered?

   6. How do we achieve resiliency (five 9s uptime) for data capture and storage?

2. Estimated Year over Year costs for storage and hosting.

   1. Please use publically provided hosting / service / storage costs for these estimations.

   2. Please estimate costs using your recommended service providers.

   3. Include estimates for each of the Tiers above.

3. Security of the Solution

   1. Security of data as it transits from the end user system to the database

   2. What do the hosting / service providers that you've proposed do to secure the customer's data?

4. Plan for moving from one tier to the next.

5. Proposed plans for load testing.

6. Gap analysis - what areas does your proposed solution not cover? What caveats are there to your solution?

7. What are we missing, not thinking of? What other pitfalls or architectural considerations should we be thinking about?

 

Challenge Scoring

 

This challenge will be scored, by the client, with a custom scorecard. There will not be community reviewers. We will share detailed scores with you after the challenge closes. Each area of your submission will be scored considering the following criteria:

 

1. Clarity and presentation - how well is it presented and how easy is it to understand?

2. Completeness - have you addressed all the requirements and points?

3. Feasibility of your solution (where applicable) - how practical is it to implement?

4. The quality of your research and the details provided.

 

Scores for each section in the challenge deliverable will be weighted as follows:

 

• Section 1 (Systems and hosting architecture): 30%

• Section 2 (YoY cost estimates at each tier): 20%

• Sections 3 - 7: 10% each

 

Your submission needs to cover all sections (1 - 7) in order to win this challenge - please keep that in mind.

Your submission should in the form of a Word or PDF document including a detailed infrastructure recommendation based on the requirements above, along with diagrams, spreadsheets, and other supporting information and documents.

Final Submission Guidelines

Deliverable as described above.