Challenge Overview
A previous challenge has implemented a set of REST APIs for handling video assets, including storing them and managing them (create, retrieve, update, delete). This challenge will add new functionality for scraping specific sites for live video feeds and exposing those in the existing REST API.
Existing API
The existing Node application and deployment details are in Gitlab, and the URL to the repository can be found in the forum.
Live TV Category
This challenge will implement a new category to the data structure - "Live TV". This will be returned as a normal category and normal video information in the /videos GET response, but it will be configured and scraped differently in the admin pages. Live TV will be a live video stream for certain sites, and this initial challenge will target two:
* CBS News (http://www.cbsnews.com/)
* News Max TV (www.newsmaxtv.com)
Admin configuration
In the admin pages, we will add a new "Live TV" tab. In this tab we will configure live TV scrapers as new functionality. The point of these scrapers will be to update the live TV playback URL on a regular basis (configured as a scheduled job in Heroku, like the existing scrapers). These scrapers will look at the configured URL and will pull out the HLS / M3U8 link for playback.
The admin configuration will include:
* Scraper name
* Parent category
* Provider
* URL of the site
* Image link
* Description
* Current URL of the playback stream
These are similar to videos, but we want to store these separately. The admin pages must support CRUD operations for live TV scrapers.
Running a scraper
The goal of the scraper is to load the configured URL of a news site, like "http://cbsnews.com" and find the live playback HLS stream. Note that both of the required sites (CBS News and News Max) will *not* return an M3U8 stream for regular user agents.
User Agent
Your best bet to get M3U8's is to identify an iPhone user-agent so that an HLS stream is provided. If you identify to the server as a standard browser, it will try to give you an F4M, which requires Flash for playback. That's not what we want.
The scraper will run and will update the registered Live TV record in the database with the current playback URL. These playback URLs are dynamic, so we need to periodically scrape the page and find the playback URL to enter into the database. This URL will be returned to the client UI for playback.
Examples:
In debugging the sites, these are the example M3U8 streams I found. Note that these should play back natively in Safari, but they may time out and stop working at any time, which is why we are building scrapers to update the playback URLs.
* http://ooyalahd2-f.akamaihd.net/i/newsmax02_delivery@119568/master.m3u8
* http://cbsnews-origin.mdialog.com/cbsnews/live/303cff1f-cb0e-4036-935e-254347b02dbe/manifest.m3u8
Heroku scheduler
These new live TV scrapers should be runnable similar to the feedScraper.js, but should be a different script! In the past we have wanted to stick with just the feedScraper.js, but we are going to have a second script for live TV scraping. This script should work similar to the feed scraper that just accepts a live TV scraper name for running.
Returning the videos
The live TV links should be returned as videos in the "/videos" feed. The should return with the current playback URL for the stream, underneath the category "Live TV". The provider, image URL, and description given should be returned in as fields for the videos. The format should be no different - the UI will treat these as any other videos.
Admin
These live TV videos should *not* show up on the Videos tab in the Admin pages. You can filter these out in JS by looking at the Live TV sub-category name.
Heroku deploy
Make sure the Heroku deployment information is up-to-date and that you keep the package.json up to date as well. Don't expect the deployment to be anything other than "npm install" / "npm start" locally and "git push heroku master" for Heroku deployment.
Existing API
The existing Node application and deployment details are in Gitlab, and the URL to the repository can be found in the forum.
Live TV Category
This challenge will implement a new category to the data structure - "Live TV". This will be returned as a normal category and normal video information in the /videos GET response, but it will be configured and scraped differently in the admin pages. Live TV will be a live video stream for certain sites, and this initial challenge will target two:
* CBS News (http://www.cbsnews.com/)
* News Max TV (www.newsmaxtv.com)
Admin configuration
In the admin pages, we will add a new "Live TV" tab. In this tab we will configure live TV scrapers as new functionality. The point of these scrapers will be to update the live TV playback URL on a regular basis (configured as a scheduled job in Heroku, like the existing scrapers). These scrapers will look at the configured URL and will pull out the HLS / M3U8 link for playback.
The admin configuration will include:
* Scraper name
* Parent category
* Provider
* URL of the site
* Image link
* Description
* Current URL of the playback stream
These are similar to videos, but we want to store these separately. The admin pages must support CRUD operations for live TV scrapers.
Running a scraper
The goal of the scraper is to load the configured URL of a news site, like "http://cbsnews.com" and find the live playback HLS stream. Note that both of the required sites (CBS News and News Max) will *not* return an M3U8 stream for regular user agents.
User Agent
Your best bet to get M3U8's is to identify an iPhone user-agent so that an HLS stream is provided. If you identify to the server as a standard browser, it will try to give you an F4M, which requires Flash for playback. That's not what we want.
The scraper will run and will update the registered Live TV record in the database with the current playback URL. These playback URLs are dynamic, so we need to periodically scrape the page and find the playback URL to enter into the database. This URL will be returned to the client UI for playback.
Examples:
In debugging the sites, these are the example M3U8 streams I found. Note that these should play back natively in Safari, but they may time out and stop working at any time, which is why we are building scrapers to update the playback URLs.
* http://ooyalahd2-f.akamaihd.net/i/newsmax02_delivery@119568/master.m3u8
* http://cbsnews-origin.mdialog.com/cbsnews/live/303cff1f-cb0e-4036-935e-254347b02dbe/manifest.m3u8
Heroku scheduler
These new live TV scrapers should be runnable similar to the feedScraper.js, but should be a different script! In the past we have wanted to stick with just the feedScraper.js, but we are going to have a second script for live TV scraping. This script should work similar to the feed scraper that just accepts a live TV scraper name for running.
Returning the videos
The live TV links should be returned as videos in the "/videos" feed. The should return with the current playback URL for the stream, underneath the category "Live TV". The provider, image URL, and description given should be returned in as fields for the videos. The format should be no different - the UI will treat these as any other videos.
Admin
These live TV videos should *not* show up on the Videos tab in the Admin pages. You can filter these out in JS by looking at the Live TV sub-category name.
Heroku deploy
Make sure the Heroku deployment information is up-to-date and that you keep the package.json up to date as well. Don't expect the deployment to be anything other than "npm install" / "npm start" locally and "git push heroku master" for Heroku deployment.
Submission format
Your submission should be provided as a Git patch file against commit hash 43e0514545f1ed1cc859ee96ef9194837eff47f3. MAKE SURE TO TEST YOUR PATCH FILE!