Website Scraper Challenge

Challenge Objectives
The purpose of this challenge is to implement a web scraper that will extract the purchase history list from E-commerce (EC) Site. The tool will be used by our client who owns representative EC site in Japan.
 
Technology Stack

• JDK 8
• Gradle 3.5
• Spring Boot 1.5.7
• Libraries such as scraping, DI, unit test may may be freely used.

Data Extraction

1. Data fields: order number, order date, product name, unit price, product quantity, product distributor, total amount of money, delivery status
2. Data format: Json

 
Challenge Requirements
The following features need to implemented in this challenge:

1. EC Site authentication module
   1. Input:
      1. EC site authentication page URL
      2. EC site of account information: email and password, such account information should be easily configurable
   2. Output: signed in EC site initial page of HTML data
   3. Process:
      1. Get the sign-in page (email address input)
      2. Based on the result of the above, set the email address, to get the sign-in page (password input)
      3. Based on the result of the above, set a password, to get the Signed in EC site initial page
   4. Remarks: we don’t have to take MFA into account
2. Purchase history list analysis module
   1. Input:
      1. HTML data of the purchase history list page of each EC site
      2. The purchase history data extracted last time
   2. Output:
      1. Purchase history list information
      2. Information extracted from HTML data: order number, order date, product name, unit price, product quantity, product distributor, the total amount of money, delivery status
   3. Process:
      1. Get the HTML data of the purchase history list page from the EC site.
      2. Read the purchase history list data that was previously gotten.
      3. Comparing the data above, extract the new data and ship it to Json.
   4. Remarks:
      1. Information to be obtained is the difference data of the previous acquired data
      2. The pagination should also be taken into account
3. Note: the application must be built using Spring Boot
4. The following requirements are not in scope of this challenge but you should consider your code code design to accommodate for them in the near future:
   1. Get detailed purchase history
   2. Get product details
   3. Support multiple EC sites
   4. Support multiple accounts

Final Submission Guidelines

• Build script to execute the program
• Detailed readme in markdown format covering the following:
  • Build instructions
  • How to configure the program
  • How to execute the program and verify results