Challenge Overview
Challenge Overview
Topcoder wants to create a generic framework to import and scrape skills from a variety of websites. This will allow Topcoder members to pull in their information from other online sites and display it on their Topcoder profile. Some data will be scraped, some data will be imported via APIs.
A member will be able to add one to many sites to their profile to grab data. These sites have to be based on existing integrations. Once the member connects the sites, we will update their stats on a daily basis through a job that will fetch skills for every connected account.
Something New
We are looking for the best Topcoder community members to be involved in a private group to develop this framework. The perfect candidates should be proficient in NodeJS, ReactJS, Mongo/Dynamo, Swagger, and available to take on this project in next several months. You would actively participate in the work that gets posted to the private group. This will be in the form of direct tasks or private challenges. In return, you will receive a fixed monthly payment ($1,100 minimum per month) if you meet our minimal criteria, and if you contribute more in any given month, you will receive higher payments that scale with your productivity.
This challenge is the first challenge of this series, and it is the only one challenge open for the public community. The top 3 members of this challenge will be recruited into the private group for the further development.
Challenge Goals
The goal of this challenge is to build a basic importer to import skills from one of the following websites.
You can freely choose one of the websites above and implement a basic importer.
The importer should address the following requirements.
(1) Import skills from the external websites
The framework should support the import of skills by the following ways.
(a) Import by Site APIs: The framework can pull data via APIs provided by the external sites.
(b) Import by Data scrappers: If the external site doesn’t provide APIs to pull data, the framework can scrape data from the sites. In this way, the scraper will obey all robots.txt and will act as a good web citizen and not violate terms or conditions.
(2) Authentication of external website accounts
Some information might be private and only be retrievable after account authentication, the framework should allow a member to connect their account through the following ways.
(a) oAuth - if the external site provides oAuth API, then oAuth is the preferable way for the framework to connect the account, since there is no need to store the member’s confidentials directly.
(b) Others - if the external site didn’t provide oAuth API, then we need to find another way to authenticate. We should avoid storing member’s confidentials in our framework directly, if possible.
(3) Skill data
We focus on the raw skill data for now, so the skill data can be
-
Likes on Behance
-
Stars on GitHub
-
Votes on StackOverflow
-
...
The importer should be implemented in NodeJS, the database should be MongoDB (if necessary).
Final Submission Guidelines
- A zip file containing your PoC importer source code
- A README.md as a deployment guide, and you should provide a description which website you chose to import, and what kind of skill data you imported.
- A Verification.md for providing verification steps. You should provide the detailed verification steps here, to verify the imported data are correct.