Problem Statement Classification - Problem Statement Classification

Problem Statement

Background

The Laboratory for Innovation Science at Harvard (LISH) is interested in understanding the solution development strategies the participants use in TopCoder Marathon Challenges, with a specific interest in identifying differences in behavior, if any, across the various types of challenges. To do so, LISH needs to obtain accurate classifications of existing challenges along several dimensions, including: type of competition, challenge objective, submission format, submission complexity, and so on. Each dimension of interest is listed below and detailed under the heading Classifications of Marathons.

We are interested in studying problem-solving as a dynamic process. As a first step to achieve this goal, we need to be able to distinguish between the types of marathons.

Task

You will be provided with a personalized csv file, containing 231 links to marathon problem statements, and an Excel spreadsheet template to fill in the labels. Your task is to provide classifications along several categories for each problem statement. Each category will require either a numerical answer or a choice from a given set of answers. Classification categories and labeling details are fully specified below.

You are free to classify as many problems statements as you would like, strictly following the order in the CSV file. To be eligible for payment, at least 50 sequential problem statements starting from the beginning need to be completely labeled. The 'How to submit' section contains instructions on how to obtain your input CSV file.

This challenge is posted in the marathon track because we believe that regular marathoners are good candidates to create high quality classifications. However, the challenge is open to everyone. Also note that this is not a usual marathon contest in the sense that you may be able to solve it without writing a single line of code, by creating manual annotations. However, feel free and encouraged to automate as much of the labeling process as you like.

Deliverables

You are required to create the following two files:

1. Filled-in CSV table, converted from the filled-in Excel spreadsheet: Each row of this file should contain your labels for each classification category for a given problem statement (you must complete a minimum of 50 rows).
2. Methodological document: This is a free-style document, outlining tools and approaches, used to streamline labeling, weak or questionable spots of the applied classification, and corresponding choices made by the labeler.

How to submit

Submitting solutions serves a dual purpose in this challenge.

• If you make an example submission (using the "Test Examples" button on the online submission interface) then the response will contain a link to your input CSV file, that is your personalized list of 231 problem statements you will need to annotate. In this case no scoring is done and no input is required from you. You should make an example submission so that you can start your work.
• If you make a full submission (using the "Submit" button on the submission interface) then the scorer system downloads and evaluates your annotated CSV file. For this to work you need to upload it to the internet where it is available for the scorer system.

For both cases described above you must implement a class named ProblemClassifier, which implements a single function: getAnswerURL(). Your function will return a String, that in case of full submissions corresponds to the URL of your CSV submission file. (In case of example submissions your return value is ignored.)

You may upload your files to a cloud hosting service such as Dropbox or Google Drive, which can provide a direct link to the file.

• To create a direct sharing link in Dropbox, right click on the uploaded file and select share. You should be able to copy a link to this specific file which ends with the tag "?dl=0". This URL will point directly to your file if you change this tag to "?dl=1". You can then use this link in your getAnswerURL() function.
• If you use Google Drive to share the link, then please use the following format: "https://drive.google.com/uc?export=download&id=" + id
• You can use any other way to share your result file, but make sure the link you provide opens the filestream directly, and is available for anyone with the link (not only the file owner), to allow the automated tester to download and evaluate it.

An example of the code you have to submit, using Java:

public class ProblemClassifier {
  public String getAnswerURL() {
    //Replace the returned String with your submission file's URL
    return "https://drive.google.com/uc?export=download&id=XYZ";
  }
}

Submitting your methodological document is not required during the online phase of the contest. If your CSV file is selected as a candidate for winning a prize, then you will need to send your method description in email after the online submission phase is over.

Evaluation

The submitted labels will be automatically screened and selectively verified by a set of judges. A quality of the labels Q is going to be estimated as the weighted rate of correct answers, where the weights are based on how frequently certain labels occur in the set of correct answers. (As an example, getting a '2-Different' answer right in the 'system_vs_provisional' question (see later) scores more than getting a '1-Similar' answer right in the same question, because most of the contests belong to this latter category. The weights used in calculating Q are not disclosed.

Any malformed or inaccessible file, or one that violates any of the constraints listed in the "Output file" section will receive a zero score.

The expected correct answers are created by two expert annotators. In the cases their chosen labels differ both labels will be accepted as correct.

Q must be higher than 0.8 in order for the labels to be accepted. This threshold is established by comparing annotations of a non-expert labeler to the expected correct answers.

Note that the scores shown on the leaderboard during the online submission phase will NOT reflect the quality of your solution. It will show either 0 if there are formatting errors in your output, or the number of entries in your CSV file that are syntactically correct, regardless of the quality of your annotations.

Prizes

If the table passes each threshold, i.e., Q > 0.8 and at least 50 problems statements have been fully labeled, then the labeler receives a payment $5 per PS (equals to $1155 if all 231 PS are labeled). The total challenge budget for this prize pool is $7000, thus if more than 1400 labels are received, they will be ordered in the order of Q, and for the top 1400 the payment will be offered. We reserve the right to distribute the prizes differently if a large number of high-quality labels is received - to support a fair payment for high-quality work.

Additionally, the submissions will be ranked based on integrative quality parameter:

S = Q * Ln(PS_labeled / 50)

The best labelers will share a bonus $2000 of the prize money as follows:

  1st                   $1000
  2nd                    $500
  3rd                    $300
  4th                    $200

A methodological document has to be present and of good quality in order to be eligible for the bonus prize money.

Early participation incentive

The first 10 labelers who submit 50 qualified (Q>0.8) labels by the end of the first week will additionally receive $50 bonus prize.

Best automated approach special prize

If you developed a software tool to help your annotation work then send it along with your methodological document at the end of the contest. Please document clearly what it does and how to run it. The best of such tools (as evaluated by the contest organizers) will receive a special prize of $500. There are no objective criteria for 'best', but the more categories your tool handles and the better quality of labels it creates, the better the tool is. If we don't receive a tool that deserves this prize then this budget of $500 is added to the main prize of $7000 described above.

Classification of Marathons

Each classification category that you will need to provide numerical labels for is described below. The category abbreviation that you will include in your deliverable CSV file as a column header is shown within the hard brackets "[...]".

Note that the labels listed here may be slightly different from the ones provided in the Excel template, e.g. for readibility these have spaces like '1 - Yes', but there are none in the template, like '1-Yes'. In your deliverable always use the labels found in the template.

• Is it a Machine Learning competition? [machine_learning]
  
  1 - Yes (Task is to develop a model that is to be optimized to the data)
  
  2 - No (Task is to develop an algorithm, performing a computational task)
  
  
• Is the goal to improve an existing solution? [improve]
  
  1 - Yes (State of the art solution or code from a previous similar contest is provided and allowed to be used as a starting point)
  
  2 - No (Solutions are to be developed from scratch)
  
  
• What is the target of the challenge? [target]
  
  1 - Quality (Quality Maximization is the primary target. Make quality of the process outcome as high as possible)
  
  2 - Resource or Both (Resource Optimization is the most or equally important. Make the process to consume as little resource (e.g. time, memory, disk space, computational steps) as possible, while maintaining a certain quality of outcome.)
  
  
• What is the primary type of data the solution is required to work on? [data_type]
  
  0 - No data (No pre-existing data is provided to this challenge. Data, if used, is randomly generated from a small number of seeds.)
  
  1 - Regular (Regular, spreadsheet like data. A list of independent records with numerical or categorical content.)
  
  2 - Image (encodes an image)
  
  3 - Signal (data records form a time series)
  
  4 - Text (string of characters)
  
  5 - Other (e.g. unstructured data)
  
  
• Is a search for external data part of the challenge? [data_external]
  
  1 - Yes (A search for new datasets is explicitly encouraged in the problem statement)
  
  2 - No (The data provided to the challenge is considered to be sufficient for the task)
  
  
• Data Volume, estimated: [data_volume]
  
  1 - 1Mb and below
  
  2 - 1Mb to 1Gb
  
  3 - 1Gb to 10Gb
  
  4 - 10Gb to 100Gb
  
  5 - 100Gb and above
  
  6 - Description of the data and state of the data links do not allow for educated guess
  
  
• What is the primary type of task the solution is required to perform over data? [task_type]
  
  1 - Exploration (searching for descriptive patterns in the dataset)
  
  2 - Strategy (maximizing expectation of an outcome of a sequential Action-Responce process (e.g., a game) based on a limited information on each step)
  
  3 - Optimization (using complete information about the system to find an implementation of a process that maximizes some measure of quality)
  
  4 - Inference (reconstructing missing parts of the data)
  
  5 - Classification (assigning test cases to predetermined classes)
  
  6 - Segmentation (clustering cases or outlining regions)
  
  7 - Prediction (making guesses about the future based on time-series data. Note that the word 'prediction' appearing in the text doesn't necessarily mean that the this is the best category.)
  
  8 - Detection (finding predetermined objects or patterns in the data)
  
  9 - Other
• Submission type during the contest (code vs data) [submission_type]
  
  1 - Code (The submission is a code, compiled and run on TC platform.)
  
  2 - List of Answers (The submission is a list of answers, directly compared to ground truth. The submission can be in a form of a short piece of code, returning e.g. link to publicly hosted data or a list of hardcoded answers)
  
  
• Engagement barrier (how hard is to start the competition) on the scale of 1-5 [engagement_barrier]
  
  "It is easy to produce a first meaningful submission" - Do you agree?
  
  1 - Strongly Agree (A problem has an obvious trivial solution that returns a positive score)
  
  2 - Agree
  
  3 - Undecided
  
  4 - Disagree
  
  5 - Strongly Disagree (Requires a lot of preparation work and self-education. The minimal positive-scored solution requires a lot of coding, data-cleaning, mastering existing code, or surpassing a high quality threshold)
  
  Note: the Excel template contains different labels, please use those.
  
  
• Scoring system type (Absolute vs relative) [scoring_type]
  
  1 - Absolute (The score shows absolute performance of the solution on the test.)
  
  2 - Relative (The score shows performance relative to the performance of the other solutions.)
  
  
• Complexity of the scoring function on the scale of 1-5 [scoring_complexity]
  
  "The solution evaluation system is easy to comprehend and navigate; I don't expect surprises or exploits" - Do you agree?
  
  1 - Strongly Agree
  
  2 - Agree
  
  3 - Undecided
  
  4 - Disagree
  
  5 - Strongly Disagree
  
  Note: the Excel template contains different labels, please use those.
  
  
• Testing capabilities during the contest [provisional_testing]
  
  1 - Shared (An offline scorer tool, sufficient for tracking individual progress, is either provided or can be implemented with a trivial effort)
  
  2 - TC preferred (An offline testing is feasible, but online testing is more reliable)
  
  3 - TC only (Offline testing capability is limited, predominantly online testing is expected)
  
  
• Testing after the contest versus testing during the contest: [system_vs_provisional]
  
  1 - Similar (System test is structurally and statistically similar to the provisional test. The test-cases in both tests are generated under the same conditions without differentiation between the tests.)
  
  2 - Different (System test is structurally or qualitatively different from the provisional test. Systematic bias is expected.)
  
  
• Total prize pool (including milestones and bonuses) [prize_pool]
  
  Input the sum of all prizes. This additionally includes all milestones and bonuses, if applicable.
  
  (All the awards here and further are filled in only if they are explicitly mentioned either in Problem Statement or on Registration Page (accessible from the Problem Statement from the link: "Register & Rules")).
  
  If the prizes are disclosed, input the full amounts as a plain number (ten thousand dollars -> "10000", quotes are for clarity only).
  
  If there are no prizes, input "0".
  
  If the prizes are mentioned, but their amounts are not disclosed in the two sources above, input "-1".)
  
  
• Total milestone and bonus amount [prize_extras]
  
  Input the sum of all milestones and bonuses, if applicable.
  
  
• Number of prizes [prize_number]
  
  Input the number or prize positions offered (without bonuses or milestones).
  
  
• First place prize [prize_first]
  
  Input the amount of the first prize. This does not include milestones or bonuses.
  
  
• Non-monetary awards [prize_nonmonetary]
  
  Were there any explicitly mentioned prizes or rewards that did not have a monetary value ($)?
  
  (Invitation to recognition event, T-Shirts, TopCoder open points, etc...)
  
  1 - Yes
  
  2 - No
  
  
• Prize eligibility threshold? [prize_threshold]
  
  1 - Yes (There is a minimal quality threshold that the solutions has to exceed)
  
  2 - No (A trivial submission can be eligible for a prize)
  
  

Definition

Examples