Challenge Overview
A previous challenge has implemented a set of REST APIs for handling video assets, including storing them and managing them (create, retrieve, update, delete). This challenge will add HLS validation to the scrapers that read HLS / M3U8 streams:
* NewsMax TV
* CBS News
* Fox News
It will also update the BBC News scraper to pull MPEG DASH / MPD streams instead of HLS.
Existing API
The existing Node application and deployment details are in Gitlab, and the URL to the repository can be found in the forum.
HLS Validator
Some of the streams don't currently fit with the HLS standard, particularly the CBS News Live TV stream - it doesn't seem to match the M3U8 / HLS spec exactly. We need to add some basic validation to the scrapers that parse out HLS streams to ensure the streams we add to the data store are valid.
Implementation
When we are scraping HLS streams for Fox News, News Max TV, or CBS News, we want to validate the HLS streams. The implementation is at your discretion, but here are some possibile implementation tips:
* https://github.com/erikwt/hlsvalidator
* https://www.npmjs.com/package/m3u8parse
* https://developer.apple.com/library/ios/technotes/tn2235/_index.html
Configurable
The validation should be able to be turned on or off via a configuration flag that can be set in the environment and using "heroku config:set", named "VALIDATE_HLS_STREAMS". The configuration value will either be true or false.
BBC and MPEG/DASH
The client has asked if we can update the BBC scraper to parse out MPEG/DASH streams instead of HLS. These streams have an "mpd" extension, like:
http://vod-dash-ww-live.edgesuite.net/usp/auth/vod/piff_abr_full_sd/b6998b-p04721k8/vf_p04721k8_968b9322-7a5c-4d9a-8585-f7f3c45302f4.ism/pc_hd_abr_v2_dash_master.mpd?__gda__=1473241500_48f91b7587c1544692bbd0ef56d814e9���
This challenge needs to ensure that the BBC scraper parses out the MPD / DASH stream URLs when available. It can default back to HLS when no DASH videos are found.
* NewsMax TV
* CBS News
* Fox News
It will also update the BBC News scraper to pull MPEG DASH / MPD streams instead of HLS.
Existing API
The existing Node application and deployment details are in Gitlab, and the URL to the repository can be found in the forum.
HLS Validator
Some of the streams don't currently fit with the HLS standard, particularly the CBS News Live TV stream - it doesn't seem to match the M3U8 / HLS spec exactly. We need to add some basic validation to the scrapers that parse out HLS streams to ensure the streams we add to the data store are valid.
Implementation
When we are scraping HLS streams for Fox News, News Max TV, or CBS News, we want to validate the HLS streams. The implementation is at your discretion, but here are some possibile implementation tips:
* https://github.com/erikwt/hlsvalidator
* https://www.npmjs.com/package/m3u8parse
* https://developer.apple.com/library/ios/technotes/tn2235/_index.html
Configurable
The validation should be able to be turned on or off via a configuration flag that can be set in the environment and using "heroku config:set", named "VALIDATE_HLS_STREAMS". The configuration value will either be true or false.
BBC and MPEG/DASH
The client has asked if we can update the BBC scraper to parse out MPEG/DASH streams instead of HLS. These streams have an "mpd" extension, like:
http://vod-dash-ww-live.edgesuite.net/usp/auth/vod/piff_abr_full_sd/b6998b-p04721k8/vf_p04721k8_968b9322-7a5c-4d9a-8585-f7f3c45302f4.ism/pc_hd_abr_v2_dash_master.mpd?__gda__=1473241500_48f91b7587c1544692bbd0ef56d814e9���
This challenge needs to ensure that the BBC scraper parses out the MPD / DASH stream URLs when available. It can default back to HLS when no DASH videos are found.
README
Make sure the README is updated with verification information about the new features and configuration information so they can be easily added.
Unit tests
As with the other scrapers, unit tests are required for these new chanages.
Heroku deploy
Make sure the Heroku deployment information is up-to-date and that you keep the package.json up to date as well. Don't expect the deployment to be anything other than "npm install" / "npm start" locally and "git push heroku master" for Heroku deployment.
Make sure the README is updated with verification information about the new features and configuration information so they can be easily added.
Unit tests
As with the other scrapers, unit tests are required for these new chanages.
Heroku deploy
Make sure the Heroku deployment information is up-to-date and that you keep the package.json up to date as well. Don't expect the deployment to be anything other than "npm install" / "npm start" locally and "git push heroku master" for Heroku deployment.
Submission format
Your submission should be provided as a Git patch file against the commit hash mentioned in the forum. MAKE SURE TO TEST YOUR PATCH FILE!