Challenge Overview
The Hercules Fog Recorder is an application that runs in an embedded environment and caches large numbers of small files for later, or simultaneous, retrieval. The main files captures are streams of video data, cut into small files in HLS (HTTP Live Streaming) format.
Problem:
The application works well in most cases, but the client has a specific device that has a hard-drive that is formatted with an XFS file system tuned to larger files, instead of the small HLS fragment files we will be downloading.
The XFS file system is formatted like this, as an example:
mkfs.xfs -f -b size=4096 -d agcount=4 -s size=4096 -l size=2m -r extsize=4m,rtdev=/dev/sda1 /dev/sda2
The problem here is the real-time extsize for the XFS formatting. This is fine for large files (>=4MB), but isn't good for small files, since they will essentially take up a full "block" of real-time storage, which is 4MB, and potentially up to 32MB on certain devices. Our HLS fragments tend to be in the 300-600KB size, so there's a large discrepancy.
Solution:
The client has asked for suggestions, and this challenge will implement some custom code that could alleviate the issue by caching in memory and then writing to disk with metadata that allows us to serve up the files, while efficiently using the space for each block.
The client has asked for an abstracted file system. So for example, manage a directory map in RAM. This can be cached to persistent storage periodically (using the 'normal' file partition).
Each file will map to a block, offset, and length. Blocks will fill, with each file appended in place until a 4MB block would be surpassed.
Blocks will need to be destroyed only when every file contained is logically deleted.
Simplifications:
Given that this will be used only for media files, logically a given recording will continuously append. Culling or completed recordings will always occur from the other end. So no need to ever recycle space from a "removed file" in the middle of a block, or move files around inside a block. We will append until the recording is complete and then the entire recording can either be played back or deleted.
Similarly, we won't ever have files that need to span multiple blocks, since the biggest fragment is still going to be far smaller than 4MB. And we don't really care about wasting space at the end of a given block.
We can use directories/subdirectories to manager chunks on disk.
Files will be fully written (either to RAM or disk) before they are requested for reading, but note that a current block may be being read and written to at the same time, because it is valid for a user to play back a recording as it is being recorded.
Buffering
Writes need to be buffered. The client software currently uses a 512KB RAM space to buffer before writing to disk, and we can do the same. If a file is requested for reading while still in the RAM buffer, then it should be read from RAM buffer.
Challenge submission
Your submission should be a set of C++ files that can be dropped into an existing project and used. We want the code to be free from as many 3rd party libraries as possible, since the embedded system has limited libraries available. If you think a 3rd party library could be very useful, please mention it in the forum, and the copilot will investigate if it's available, but chances are it likely won't be an option.
Compiler
We are targeting Linux for submissions, and g++ 4.5.1. This is a rather old compiler, so please make sure that you test on it explicitly.
Tests
Please include a simple test application that wraps the C++ files and does a good job of showing how the code works and can easily be used to validate the submission for both writing files and then reading them back out again using the stored metadata. Targeting "make" for your test application should be alright, just make sure to follow the compiler version restriction above.
Submission details
No video is required, but your README.md needs to be very thorough in details and validation information so that reviewers can easily build and validate your submission.
Problem:
The application works well in most cases, but the client has a specific device that has a hard-drive that is formatted with an XFS file system tuned to larger files, instead of the small HLS fragment files we will be downloading.
The XFS file system is formatted like this, as an example:
mkfs.xfs -f -b size=4096 -d agcount=4 -s size=4096 -l size=2m -r extsize=4m,rtdev=/dev/sda1 /dev/sda2
The problem here is the real-time extsize for the XFS formatting. This is fine for large files (>=4MB), but isn't good for small files, since they will essentially take up a full "block" of real-time storage, which is 4MB, and potentially up to 32MB on certain devices. Our HLS fragments tend to be in the 300-600KB size, so there's a large discrepancy.
Solution:
The client has asked for suggestions, and this challenge will implement some custom code that could alleviate the issue by caching in memory and then writing to disk with metadata that allows us to serve up the files, while efficiently using the space for each block.
The client has asked for an abstracted file system. So for example, manage a directory map in RAM. This can be cached to persistent storage periodically (using the 'normal' file partition).
Each file will map to a block, offset, and length. Blocks will fill, with each file appended in place until a 4MB block would be surpassed.
Blocks will need to be destroyed only when every file contained is logically deleted.
Simplifications:
Given that this will be used only for media files, logically a given recording will continuously append. Culling or completed recordings will always occur from the other end. So no need to ever recycle space from a "removed file" in the middle of a block, or move files around inside a block. We will append until the recording is complete and then the entire recording can either be played back or deleted.
Similarly, we won't ever have files that need to span multiple blocks, since the biggest fragment is still going to be far smaller than 4MB. And we don't really care about wasting space at the end of a given block.
We can use directories/subdirectories to manager chunks on disk.
Files will be fully written (either to RAM or disk) before they are requested for reading, but note that a current block may be being read and written to at the same time, because it is valid for a user to play back a recording as it is being recorded.
Buffering
Writes need to be buffered. The client software currently uses a 512KB RAM space to buffer before writing to disk, and we can do the same. If a file is requested for reading while still in the RAM buffer, then it should be read from RAM buffer.
Challenge submission
Your submission should be a set of C++ files that can be dropped into an existing project and used. We want the code to be free from as many 3rd party libraries as possible, since the embedded system has limited libraries available. If you think a 3rd party library could be very useful, please mention it in the forum, and the copilot will investigate if it's available, but chances are it likely won't be an option.
Compiler
We are targeting Linux for submissions, and g++ 4.5.1. This is a rather old compiler, so please make sure that you test on it explicitly.
Tests
Please include a simple test application that wraps the C++ files and does a good job of showing how the code works and can easily be used to validate the submission for both writing files and then reading them back out again using the stored metadata. Targeting "make" for your test application should be alright, just make sure to follow the compiler version restriction above.
Submission details
No video is required, but your README.md needs to be very thorough in details and validation information so that reviewers can easily build and validate your submission.