Challenge Overview
Competition Task Overview
The purpose of this challenge is to create a service that will visite a site via a provided proxy, grap the content from a certain page and return the data in JSON format.
More specifically, this challenge must cover the following:
1. The service should be built as REST service, let's call it the AccountService.
2. Programming language should be node.js
3. In this version, we are only scraping one site, but the goal is to be able to scrape multiple sites, so we want the module that does the actual scraping to be pluggable so that we can easily add more modules to scrape multiple sites at the same time. This will likely also require some configuration parameters so the AccountService can load multiple scraping modules at runtime.
4. The AccountService must monitor all errors and report the errors to an error service, the error service will also be REST based so you can just make the endpoint of the error service configurable in this challenge. Account service will post an array of error objects to the error service, each error object will have an title and a details field. Typical errors include errors in proxy, bank sites changed the layout and AccountService can no longer scrape the data, etc...
5. Please list out blockers / risks / issues you faced when building this, we want to know them all so we will have a better idea about how to handle them when we go with other bank sites.
6. The winner will be provided with the real error service to test compatibility and fix any issues found.
7. Provide REST API doc in postman format
8. Provide documentation or simple architecture doc that shows how next challenges will need to create new plugin for different bank sites.
9. Make sure code is well documented.
Technology Overview
Node.js
REST
Web Scraping
JSON
AWS
Documentation Provided
Register to see details about the proxy.
Final Submission Guidelines
Submission Deliverables
Below is an overview of the deliverables:
1. Full code that covers all requirements
2. A complete and detailed deployment document explaining how to deploy the application including configuration information.
3. REST API doc in postman format
4. Documentation or simple architecture doc that shows how next challenges will need to create new plugin for different bank sites.
Final Submission
For each member, the final submission should be uploaded via the challenge detail page on topcoder.com.