Hercules C++ Spark App - HTML5 Video support through GStreamer

Challenge Objectives

• C++ application running on MacOS X, offering Javascript APIs to developers
• More details can be found here: http://www.sparkui.org/index.html
• We will target Mac OS X for this challenge.  Submitters and reviewers will need access to OS X

Basic requirements for this challenge:

Build out the Javascript API to support the HTML5 video element, using the GStreamer libraries that have been recently added.  

NOTE: AAMP will NOT be used for this challenge.  This will be GStreamer only

Project Background

"Spark" is the  name for the pxScene project we've done work for in the past.  The links below offer a lot more details on these particular technologies. pxCore is a C++ graphics library that is used on a wide variety of embedded devices that the client distributes.  Spark is a wrapper application around pxCore that exposes the drawing APIs to Javascript through Node and V8.

The client would like to start supporting MSE / EME in Spark through GStreamer.  This is the long-term goal of this project

• MSE: https://en.wikipedia.org/wiki/Media_Source_Extensions
• ESE: https://en.wikipedia.org/wiki/Encrypted_Media_Extensions

The current pxScene doesn't support the video element, so this challenge will build that out, using playback from Gstreamer, rendering content to pxScene.  Future challenges will use this new element to start to implement MSE and ESE APIs.

Technology Stack

• Spark: https://github.com/pxscene/pxCore
• Build instructions:  https://github.com/pxscene/pxCore/blob/master/examples/pxScene2d/README.md
• Debug instructions using VSCode: https://github.com/pxscene/pxCore/blob/master/examples/pxScene2d/VSCODE_DEBUG.md

Note:  It's my assumption that we are to use the system GStreamer libs (either installed via Brew on macOS or apt-get on Ubuntu).  I'll post an update in the forum if that's not the case.

Code access

• The main Spark code is available in Github
• We will target this branch, which has the Gstreamer libraries added that the client wants to use: https://github.com/vinodkd80/pxCore/commits/_video
  • You are free to target any commit hash at 2424966 or after.

Individual requirements

This challenge needs to support the HTML5 video element API in pxScene, allowing us to playback unencrypted video files (https://www.w3schools.com/html/html5_video.asp).  Note that this won't support HLS or Dash - that will be coming in a future challenge.  For this challenge, we can expect that the video playback will be of local videos that we enter "file:///" URLs into in the browser.js or by calling `./Spark <local file path>` to play a file directly.

You should implement the video playback using the GStreamer libraries added as part of the _video branch.  That should be enough for basic video playback.  Feel free to document if there are specific formats that won't work with the libraries added.  I'm assuming .avi and .mp4 files will work alright.

The frames will be retrieved from GStreamer and then rendered using pxScene.  The goal is to have the new video element be a pxScene object that we can rotate, manipulate, and layer, just like any other pxScene object.

Integration

Javascript playback

What I would like to see is the ability to enter an mp4 or avi link in the browser.js navigation bar in Spark and have playback start.  

Events

Note that all relevant video element events should be raised. (https://www.w3schools.com/tags/ref_av_dom.asp)

Out of scope

• Text tracks are out of scope for now.  We'll probably add that later, but it's not needed right now.  
• We don't need to display controls on the video element - it's assumed those will be implemented in pxScene Javascript
• If there are other ancilliary features you think could be removed from scope, feel free to raise them in the forum.

Validation

We will create a new videoPlayer.js file, similar to the browser.js that can be loaded in Spark. This will allow a user to load up a Dash URL and control playback via the UI.

The user should be able to:

• Enter a file location in a text box and initiate playback
• See a progress bar that moves as playback happens.
• Seek to a specific playback time by clicking the progress bar.
• See the current playback time and the total duration of the stream
• Pause / resume playback

The videoPlayer.js should use the HTML 5 video API implemented above to perform the required functionality.

pxVideo object

Note that the AAMP integration in the _video branch already implements an object called pxVideo.  We'll need to make sure that our implementation doesn't overlap with the AAMP implementation.

Sample files

Any file here is fine for testing:  https://peach.blender.org/download/
Others you may find are also OK, just make sure they are licensed appropriately, preferably Creative Commons.

Deployment guide and validation document

Make sure to require two separate documents for validation.

A README.md that covers:

Updated build instructions for building the new Spark app with AAMP integration

A Validation.md that covers:

How to validate basic video playback and how to use the new videoPlayer.js file

What To Submit

• Patch file targeting a recent commit hash of pxCore (specifically the _video branch here: https://github.com/vinodkd80/pxCore/commits/_video)
• Updated codebase (optional, but definitely recommended in case there are problems with your patch file)
• README.md
• Validation.md

 

Final Submission Guidelines

Please see above