Challenge Overview
-
The task of the challenge is pretty straightforward. Build an image classifier. The classifier should return the probability of the True case, where the True case is: “the image contains the ‘target device’”. In other words, the classifier should return the probability of the input image containing the image of a ‘target device’.
-
The returned value for each input image should be a float in the range [0, 1].
-
The number of training images is not that large, so participants are encouraged to use data augmentation techniques if required, and possibly generate similar images with altered lighting, brightness, skewness, rotation etc to ensure that your trained model generalizes well.
-
The participants are free to use any technique of their choice, be it machine learning, deep learning or even basic rule-based computer vision techniques.
-
ROC area under curve score (ROC AUC) will be used to evaluate the quantitative performance on the submissions against a holdout test dataset. For more information about this metric, please refer to its implementation in scikit-learn. The overall submission will be scored based on other criteria as well described in the 'Submission Evaluation' section below.
-
The path of training images, testing images and configurations should not be hard coded into the code. They should be either passed via the command line arguments, or should be configurable via a dedicated JSON based config.json file.
-
The tool should be a command line tool, with separate commands to run the train, validate and test steps.
-
The Code should be implemented in Python 3.7
Submission Evaluation
-
The submission will be evaluated subjectively, but broadly the rating will be determined by the following:
-
70% - model performance in terms of ROC AUC score.
-
20% - Code quality and issues related to the code
-
10% - Documentation and Ease of deployment, following solely the deployment guide.
-
Data access & description
-
The dataset can be found in the Forum.
-
Inside the dataset folder, the ‘our_target_device_examples’ folder contains images of the device that we are looking to detect, while ‘not_our_target_device_examples’ folder contains counter-example images of a device that is not our target device. For counter examples, you are free to use any additional images you want to us that does not contain our target device image.
Final Submission Guidelines
What to Submit
-
The code
-
A PDF/Word/Markup format based report detailing the techniques and algorithms used in the implementation.
-
A README.md file detailing the deployment instructions, along with example and steps to verify the various capabilities of the Code.
-
Optional - screenshots or video to make it easier to quickly verify and test the code.
Technology Stack
-
Python