Register
Submit a solution
Status: ‌Cancelled client request

Challenge Overview

Topcoder is working with a group of researchers organized by the University of Chicago that are competing to understand a series of simulated environments.  We’re asking you to make predictions about this environment on two different dates. The primary Predict Test starts at simulation date 2022-07-15 at 0.00 am.  This time will be denoted as either “P-Time” or “P-Day” in the following but we have a few questions that request predictions at “F-Time” or “F-Day” which is 2022-07-14 at 0.00 am .  We are looking for simulation models that can predict the future states of our Urban World for 30 days after P-Day in two scenarios:
 

  • Scenario 1 (the “No Change” Scenario): In the No Change Scenario, nothing happens. The simulation is continued beyond P-Day without any exogenous events. The purpose of Scenario 1 is to test the basic ground truth understanding.

  • Scenario 2 (the “Close Recreational Sites” Scenario): In this scenario, one third of the recreational sites in our world are closed permanently (20 out of 60) at P-Time meaning permanently flagged as closed and can no longer be visited by individuals. The purpose of Scenario 2 is to test the understanding of how individuals will react to a situation where their choices (of choosing recreational sites) become limited. 

  • Scenario 3 (F-Day Predictions)

     

    Task Details

     

    There are 4 questions to answer scenarios 1 and 2, as follows, and 3 questions for scenario 3. The full dataset can be downloaded from the forum.

     

    Predict Test Question 1            

    For Scenario 1 and Scenario 2 and for each of the following five recreational sites {2629, 5298, 5299, 10648, 7976}, predict the number of individuals visiting the site per day for 30 days starting on P-Day.

     

    Subquestion 1a) Point Estimate

  • Predict the total number of visits (check-ins) per day in the next 30 days for each of the five recreational sites.

    • Note that this question does not ask for UNIQUE individuals checking in at the site on that day. Thus, if an individual visits the same recreational site multiple times on the same day, then each of these visits counts.

    • For example, if individuals [A, B, A, B, C, A] visited a recreational site on a day, then the ground truth number of visits would be 6, even though only three distinct individuals visited the site on this day.

  • Deliverable : Five time series (one per site) of length 30 days per scenario. Each value corresponds to the predicted number of visitors on one day intervals.

    • To avoid confusion during evaluation, please use the following data format: (Scenario, SiteID, Date, Prediction), where Scenario in {Scenario 1, Scenario 2} specifies if the prediction is for Scenario 1 (“No Change Scenario”) or Scenario 2 (“Close Sites”), SiteID is one of the five sites for which you do the prediction, Date is one of the 30 subsequent days starting on P-Day, and Prediction is a real value (or integer) of the number of visitors that you predict for the corresponding recreational site on the corresponding day.

    • For example, consider the following lines:

      • (Scenario 1, Site_1, 2022-07-15, 42)

      • (Scenario 1, Site_1, 2022-07-16, 55)

      • …

      • (Scenario 2, Site_5, 2022-08-14, 33.75)

    • Denotes that for Scenario 1, and for Site_1, you predict a total number of 42 visits on P-Day, while in Scenario 2 for Site_5, you predict at total number of 33.75 visits on the last day to be predicted.

    • Clearly, the ground truth number of visitors is an integer, but you can use real values for your prediction.

    • Your answer should have a total of 300 (= 2 Scenarios * 5 Sites * 30 Days) of predicted values


      Subquestion 1b) Confidence Intervals

  • Provide 50%, 90% and 95% confidence intervals for your predictions in Question 1a

  • Deliverables : Five times six time-series of length 30 days for each scenario. That is, for each of the five recreational sites, and for each of the 30 days, both a lower and an upper bound of the confidence interval for each level of confidence.

    • These will be compared against empirical ground truth confidence intervals obtained from the TA1A simulation.

    • Please use the following format for your answer: (Scenario , SiteID, Date, LoC, LB/UB?, Predicted Bound) where Scenario in {Scenario 1, Scenario 2} specifies if the prediction is for Scenario 1 (“No Change Scenario”) or Scenario 2 (“Close Sites”), SiteID is one of the five sites for which you do the prediction, Date is one of the 30 subsequent days starting on P-Day, Level of confidence LoC in {50%, 90%, 95%} is the level of confidence of your confidence interval, LB/UB? In {LB, UB} specifies if this is the lower or the upper bound of your confidence interval, and Predicted Bound is your predicted value for this bound

    • For example, the lines:

      • (Scenario 1, Site_1, 2022-07-15, 95%, LB, 21.0)

      • (Scenario 1, Site_1, 2022-07-15, 90%, LB, 27.3)

      • (Scenario 1, Site_1, 2022-07-15, 50%, LB, 35.0)

      • (Scenario 1, Site_1, 2022-07-15, 50%, UB, 48.3)

      • (Scenario 1, Site_1, 2022-07-15, 90%, UB, 57.751)

      • (Scenario 1, Site_1, 2022-07-15, 95%, UB, 78.0)

    • Provide your bounds for Scenario 1, Site_1 on the first day (P-Day). These lines imply that in Scenario 1, in Site_1, on the first day, your 95% confidence interval for the number of visitors is [21.0,78.0], your 90% confidence interval is [27.3,57.751] and your 50% confidence interval is [35.0,48.3]. Note that these numbers were chosen completely arbitrarily as an example only.

    • Your answer should have a total of 1800 (= 2 Scenarios * 5 Sites * 30 Days * 3 Levels of Confidence * 2 Bounds) of predicted values.                                                   


      Predict Test Question 2
       

  • For each scenario and each of the same five recreational sites specified in Question 1 ( { 2629, 5298, 5299, 10648, 7976 } ), predict the daily number of meetings at the sites each day for 30 days (group level question). This question is identical to Question 1, except that instead of predicting the number of visitors, we are asking you to predict the number of meetings per day. Note that this question does NOT ask for confidence intervals.

  • For each scenario, predict the number of meetings per day in the next 30 days for each of the five recreational sites.    

  • Deliverable : Five time series of length 30 for each scenario. Each time series should correspond to one of the five recreational sites, and each value corresponds to the predicted number of meetings on one day.

    • To avoid confusion during evaluation, please use the following data format: (Scenario, SiteID, Date, Prediction). This is analogous to Question 1, except that Prediction now corresponds to the predicted number of meetings in the specified Scenario at the specified SideID on the specified Date.

    • Your answer should have a total of 300 (= 2 Scenarios * 5 Sites * 30 Days) of predicted value.                        


      Predict Test Question 3

  • For the following set of 100 individuals (specified by their UID), predict the unique set of recreational sites that the individual will visit at least once in the next 30 days for each scenario. This question does NOT ask for confidence intervals.

    UID-44202c1fe5, UID-ea6ed87213, UID-74b389b575, UID-e44f3d6f45, UID-3e89d59d0a, UID-8fd9900365,
UID-84511fed53, UID-3aba08cb07, UID-cecdcd8dbe, UID-68dacdaf66, UID-bc4db336c6, UID-b0ec2e17a8,
UID-f1a7d6b081, UID-f0bcc2674e, UID-8422761fca, UID-4dd808026c, UID-37787f24a6, UID-55a0e3191c,
UID-46fbc1a37e, UID-c85b9e7030, UID-4e6166502f, UID-cebeaf3333, UID-2c43b5962d, UID-3b2b08a4de,
UID-e3b8d3943c, UID-588dacd236, UID-99a18e760f, UID-71b065c6f0, UID-486714bf64, UID-bb6d097bcd,
UID-5970e2dfc7, UID-172b3f5b91, UID-104a92cba6, UID-b3c58c5215, UID-6d3d3bd1c1, UID-3ec7b78739,
UID-99d88940af, UID-b2cb37b9d0, UID-4183642d06, UID-32fbad793c, UID-e7beb23c35, UID-e8c46277b9,
UID-237230a73a, UID-c1b82f70b5, UID-bdf04a76f8, UID-b9a57efadc, UID-19060f45e2, UID-bc93117f6d,
UID-22f37b9787, UID-6505b897ac, UID-ebea5f9528, UID-a033f0b10e, UID-375c19ec0b, UID-4cfd50e2f3,
UID-651d44b6d4, UID-88b1652d30, UID-9a10de20c7, UID-04d4335ea9, UID-ca37bc0c84, UID-b9fe10f51b,
UID-e0bbc5c01d, UID-e02e12e2db, UID-403ca6f15f, UID-a0a216d80a, UID-25f4e748e0, UID-bef5394139,
UID-f38250617c, UID-8b2b6de3ea, UID-1520ea6ffb, UID-fe261453ef, UID-a2c6273c2b, UID-52510a59c1,
UID-0b17f4bcac, UID-4cb178a090, UID-225c1d5acc, UID-f020d237a8, UID-a7b22d1254, UID-543918c143,
UID-c5f235af54, UID-65b77b84bb, UID-3f8b5c2abb, UID-711d5d70f2, UID-adf6b2f0a5, UID-311dc1060d,
UID-d39d716c3d, UID-4c79b4833d, UID-5580f363c1, UID-31bbaa942c, UID-7d503bef0c, UID-bade547a09,
UID-5c06a1c617, UID-6d593356cb, UID-fa0a9f7446, UID-b0a541b22e, UID-14479193d9, UID-acfb4a9f75,
UID-cea3801a2f, UID-9e580333f5, UID-2c9f265e4e, UID-193fe89563

  • For each of the 100 specified individuals, predict which recreational sites these individuals will visit (regardless how often) for the 30 day period

  • Deliverable:100 sets of recreational site IDs. Each set may have up to 60 (Scenario 1) / 40 (Scenario 2) SiteIDs.

    • To avoid confusion during evaluation, please use the following data format: (Scenario, UID, SiteID), implying that in the specified scenario, the specified individual (UID) visited recreational site SiteID.                

    • For example, the following data:

      • (Scenario 1, UID-42, SiteID-5)

      • (Scenario 1, UID-42, SiteID-26)

      • (Scenario 1, UID-42, SiteID-48)

      • ...

    • means that you predict that person UID-42 will visit the following three sites in the 30 days after P-Day in Scenario 1: {SideID-5, SiteID-26, SiteID-48}

    • A list of all recreational sites (their siteIDs) is provided in the initial data package, including a list of recreational sites closed in Scenario 2.

      Predict Test Question 4

  •  Predict the average number (among all individuals) of recreational site visits per day

    Subquestion 4a) Point Estimates

  • For each scenario, and for each of the 30 days after P-Day, predict the average number of visited recreational sites among all individuals.

    • For example, if on a Saturday, a total of 12,000 site visits took place by 5000    individuals, then the answer for this day would be 2.4

    • Deliverable : One time-series of length 30 per scenario, corresponding to the predicted average number of recreational site visits.

    • Please use the following data format: (Scenario, Date, Prediction), implying that in the specified scenario, on the specified date, you predict an average number of Prediction recreational site visits.

    • For example, a line such as:

      • (Scenario 2, 2022-07-15, 1.265)

    • Denotes that for Scenario 2, on the first day, you predict an average of 1.265 recreational site visits per individual.    

    • Your answer should have a total of 60 (= 2 Scenarios * 30 Days) of predicted values.                        

  • Subquestion 4b) Confidence Intervals:

    Provide 50%, 90% and 95% confidence intervals for your prediction in Question 4a.

  • Deliverable : For each scenario and for each level of confidence (50%, 90%, 95%): Two time series of length 30. That is, 30 lower bounds and 30 upper bounds (one lower bound and one upper bound per day.

  • Please use the following data format: (Scenario, Date, LoC, LB/UB?, Prediction), to denote that for the specified scenario (Scenario), on the specified date (Date), at the specified level of confidence (LoC), the lower or upper bound (LB/UB?) is has the predicted value.

  • For example, the following line:

    • (Scenario 1, 2022-07-15, 90%, LB, 1.165)

  • Denote that for Scenario 1, on date 2022-07-15, the lower bound of your 90% confidence interval is 1.165.

  • Your answer should have a total of 360 (= 2 Scenarios * 30 Days * 3 Levels of Confidence * 2 Bounds) of predicted values.

    Predict Test Question 5            

  • For Scenario 3 and for each of the following five recreational sites {2629, 5298, 5299, 10648, 7976}, predict the number of individuals visiting the site per day on F-Day.

Predict Test Question 6    

  • For Scenario 3 and each of the same five recreational sites specified in Question 5 above ( { 2629, 5298, 5299, 10648, 7976 } ), predict the daily number of meetings at the sites on F-Day(group level question). This question is identical to Question 5, except that instead of predicting the number of visitors, we are asking you to predict the number of meetings per day. Note that this question does NOT ask for confidence intervals. 

Predict Test Question 7    

  • For the following set of 100 individuals (specified by their UID, same as Predict Test Question 3), predict the number of recreational site visits on F-Day.


 

Goal of This Challenge: 

You are asked to build models to make predictions to the corresponding questions. Your solution will be judged based on the novelty as well as the performance on the given data.

 A few of the given questions will be objectively scored according to answers that are known to be correct, and others will be scored subjectively.

For all predictions, please provide clear model training and usage to create the predictions.  The reviewers should be able to easily expand your code to use a new / expanded data set. Do not leave anything to be assumed here, no matter how trivial.  This will be part of the review at the end of the challenge, so the more information you provide, and the better your documentation is, the better your chances of winning will be.

The dataset to use for this challenge is in two parts.  The first part is from the Explain Phase of this challenge. It samples the time period from 08/01/2019-10/31/2019.  This initial data set contains the map of the world you'll be analyzing.  The second part is more recent and involves granular data from 06/15/2022-07/13/2022 and less granular data from 04/16/2022-07/13/2022 about certain recreational sites.  This data set also includes responses to the research requests that have been answered by the simulation team.  You can learn more about the Phase 2 Research Requests by reviewing this document.



Important Note:


Each University of Chicago Team has the ability to request additional information from the virtual world simulation teams beyond what is initially provided through a “Research Request” process.  Data files or folders that are denoted with an “RR” are the output of this process. In the Code Document forum you’ll find a link to a Research Request document which provides the original request submitted by the University of Chicago researchers that can provide some context.  The requests have to include a plausible collection methodology (e.g. surveys or instruments that can collect data). There may be additional data that is provided over the course of this challenge submission period. You are encouraged to include this input into your analysis.



Final Submission Guidelines

Submission

The final submission must include the following items.

  • A Jupyter notebook detailing:

    • How the data is prepared and cleaned, from the tsv files

    • How each model is created, trained, and validated

    • How individual predictions are created

    • How we can plug in new data into the model for validation purposes.

      • This will be part of the review, so please ensure the reviewer can easily put in the held-back data for scoring purposes.

    • Answers / Exploration of the counterfactual prediction questions detailed above.  This should be well documented and clearly described.

 

Judging Criteria

Winners will be determined based on the following aspects:

  • Model Effectiveness (40%)

    • The research teams will be comparing your work with theirs in answering the “Unscored” Questions above.  Your submission will receive a subjective evaluation from this team.

  • Model Accuracy (30%)

    • Are your predictions on the held back review data (“Scored” Questions)  accurate?

  • Model Feasibility (20%)

    • How easy is it to deploy your model?

    • Is your model’s training time-consuming?

    • How well your model/approach can be applied to other problems?

  • Clarity of the Report (10%)

    • Do you explain your proposed method clearly?

Review style

Final Review

Community Review Board

Approval

User Sign-Off

ID: 30104412