Web Scraper Project 04 - Multiple Accounts Support

CHALLENGE OBJECTIVES

• Add settings UI to an existing web scraper project to add multiple accounts
• Update the scraper to scrape purchase history for all accounts
• Update the scraper to save purchase history for all accounts

 
PROJECT BACKGROUND

• The purpose of this project is to implement a web scraper that will extract the purchase history list from specified E-commerce (EC) Site.
• The tool will be used by our client who owns representative EC site in Japan.
• This is more of a PoC than a formal tool.

 
WORKFLOW

1. At first, the operator will register users by manually inserting some user records into the database, and then the operator will retrieve the hashed user_id from database.
2. For each user_id, the operator will send an email to the corresponding user asking him / her to visit the web app (described in detail below) to register his / her account, the url will be something like http://www.somewebsite.com/users/<user_id>, only the user_id here is a variable, the rest will be static.
3. User will complete account registration on this page.
4. If user registration is successful, the API will save the user’s auth token to some specific EC site in the database, and scraper will use that auth token to scrape the user’s purchase history and product details.
5. If user registration fails or token expires / fails login for some reason, the API will also record that in the database. Operator will check the database to see which user failed to register and will manually send email to these users to try again.

 
GENERAL REQUIREMENTS
After this challenge, we’ll have 3 code modules:

• The web app which must be done using Vue.js
• The backend REST API which must be done using Java 8 + SpringBoot
• The scraper that’s a standalone command line application built using Java 8
• You can skip unit tests in this challenge, and to run the base code correctly use ./gradlew clean build -x test

 
We still want to maintain all modules in the same git repo, so please make sure your code structure properly separates them yet makes it possible to have them in the same repo.
 
Also the API and scraper would share quite a bit of common code, so it makes sense to have them in the same project but be built with different gradle tasks
 
DETAILED REQUIREMENTS
We are providing summary info about each part of the app below, please register to download the full spec with screenshots to understand the full details of each module.
 
The Web App
The web app will consist of a couple simple web pages:

1. EC Site Settings List: this page has the following elements
   1. The title: hardcoded text
   2. The Description: hardcoded text
   3. EC Site Name: hardcoded text
   4. EC Site Usage: a checkbox, the value of this will need to be saved in the backend
   5. Status: this is a label that will show the login status (success / failure / etc…)
   6. Try to Login: this is a button that will invoke the next page / dialog
2. EC Site Login: this page has the following elements
   1. Title: Simple text
   2. Email Input
   3. Password Input
   4. Code Description: hardcoded text
   5. Code: code input
   6. CAPTCHA Image
   7. Login button

 
When user hits the web app, the web app needs to make sure the user_id field is valid (i.e. existings in the database) via an API, and if not, the web app will show an error page and won’t allow the users to do anything else.
 
If the user_id is valid, it will retrieve user’s previous EC site settings from database and populate them on the UI.
 
Server Side API
The following API endpoints must be implemented to support the Web app:

1. GET /users/:user_id/ec_sites
   1. Get the list of EC sites info for the user from database
2. GET /users/:user_id/ec_sites/:ec_site_id
   1. Get the specific EC site info from database
3. PUT /users/:user_id/ec_sites/:ec_site_id
   1. Update the specific EC site’s information in database for the user.
4. GET /users/:user_id/login_init
   1. Initialize login process, check full spec for details
5. POST /users/:user_id/login
   1. Try to login to EC site using the provided info, check full spec for details. We may not be able to do this via our own API, but instead have to use the EC site’s own login page and then parse the token / cookies from the response.

 
Scraper
The scraper will need to be updated to do the following:

1. Get purchase history for all EC site accounts in the database (right now there’s always just 1 account)
2. Update the EC site authentication implementation: please check full spec for details

 
OPERATING SYSTEMS & BROWSERS

• Linux / Mac OS / Windows
• Must work on on Chrome / Firefox / Safari (Mac only)
• IE support is NOT mandatory

 
DEVELOPMENT ASSETS

• Base code
• Detailed requirements spec with screenshot

 
TECHNOLOGY STACK
The following technology stack will be used as part of this challenge:

• JDK 8
• Gradle 3.5
• Spring Boot 1.5.7
• MySQL
• Vue.js

Final Submission Guidelines

FINAL DELIVERABLES

• All original source code.
• A detailed readme in markdown format explaining how to build, configure and deploy your code.
• A detailed verification document in markdown format showing how to verify that your submission works properly (note: video based verification is NOT acceptable).