Challenge Overview
A previous challenge has implemented a set of REST APIs for handling video assets, including storing them and managing them (create, retrieve, update, delete). This challenge will implement validation of all videos scraped from the scrapers.
Existing API
The existing Node application and deployment details are in Gitlab, and the URL to the repository can be found in the forum.
Existing API
The existing Node application and deployment details are in Gitlab, and the URL to the repository can be found in the forum.
Video validation
This challenge focuses on video validation. We already have basic validation of M3U8 / HLS videos, and we are going to expand this as follows:
Video URL validation
For every single video URL scraped, we will make a request to the URL to ensure two things:
1. The response is a 200 HTTP response (after any redirects)
2. The "content-length" on the response header is greater than a configurable value (defaulted to 500kb)
3. The "content-type" on the response header is a media type (video/mp4, or some other media type, not text or HTML)
This will ensure that the URL is valid (doesn't return a 404 or 500 error) and that the content coming back is a video file.
For each video in the database, we will add a new table column that indicates whether or not the video validated successfully. We can call this boolean flag "video_valid" or "validated"
Failing validation
Failed videos will still be added to the database, but we'll set the boolean flag mentioned above to false. The REST APIs will *not* return failed videos in requests for videos, which will ensure that the UI will not show any videos that failed validation. You will need to update the REST APIs to ensure they don't return videos that fail.
Re-run validation
Videos may fail validation at the time of scraping, but they may work again later. This challenge will add and document a Node script that can re-run validation for all videos, or optionally for videos from a specific provider. The validation will do the same checks for all videos, including ones that passed validation when scraped and those that failed, and will update the new table column appropriately. It's expected that the REST API will then immediately start to return any videos that pass validation with the re-run, even if they failed validation at the time of scraping.
Configuration
There needs to be new environmental config values for validation:
1. VALIDATE_VIDEOS will be a boolean flag to turn validation on or off for all scrapers. If this value is *not* set in the environment, we will default to true.
2. VALIDATE_CONTENT_LENGTH will be a value (in kilobytes) that videos must meet in the "content-length" check. We will default to 500 if this value is not set in the environment.
README
This challenge focuses on video validation. We already have basic validation of M3U8 / HLS videos, and we are going to expand this as follows:
Video URL validation
For every single video URL scraped, we will make a request to the URL to ensure two things:
1. The response is a 200 HTTP response (after any redirects)
2. The "content-length" on the response header is greater than a configurable value (defaulted to 500kb)
3. The "content-type" on the response header is a media type (video/mp4, or some other media type, not text or HTML)
This will ensure that the URL is valid (doesn't return a 404 or 500 error) and that the content coming back is a video file.
For each video in the database, we will add a new table column that indicates whether or not the video validated successfully. We can call this boolean flag "video_valid" or "validated"
Failing validation
Failed videos will still be added to the database, but we'll set the boolean flag mentioned above to false. The REST APIs will *not* return failed videos in requests for videos, which will ensure that the UI will not show any videos that failed validation. You will need to update the REST APIs to ensure they don't return videos that fail.
Re-run validation
Videos may fail validation at the time of scraping, but they may work again later. This challenge will add and document a Node script that can re-run validation for all videos, or optionally for videos from a specific provider. The validation will do the same checks for all videos, including ones that passed validation when scraped and those that failed, and will update the new table column appropriately. It's expected that the REST API will then immediately start to return any videos that pass validation with the re-run, even if they failed validation at the time of scraping.
Configuration
There needs to be new environmental config values for validation:
1. VALIDATE_VIDEOS will be a boolean flag to turn validation on or off for all scrapers. If this value is *not* set in the environment, we will default to true.
2. VALIDATE_CONTENT_LENGTH will be a value (in kilobytes) that videos must meet in the "content-length" check. We will default to 500 if this value is not set in the environment.
README
Make sure the README is updated with verification information about the new features and configuration information so they can be easily added.
Unit tests
Unit tests are required for this challenge.
Heroku deploy
Make sure the Heroku deployment information is up-to-date and that you keep the package.json up to date as well. Don't expect the deployment to be anything other than "npm install" / "npm start" locally and "git push heroku master" for Heroku deployment.
Submission format
Your submission should be provided as a Git patch file against the commit hash mentioned in the forum. MAKE SURE TO TEST YOUR PATCH FILE!