Reclamation: Streamflow Forecast Rodeo Marathon Match

OVERVIEW

This challenge seeks to improve the skill of short-term streamflow forecasts (10 days). Solvers will develop and implement their methods for locations across the western United States, attempting to outperform state of practice streamflow forecasts.

There will be two parts of the series: A “warm-up” Marathon match beginning on August 28, 2020 followed by 12 monthly competitions occurring over a year-long period beginning September 23, 2020, where solvers provide real-time daily streamflow forecasts. Know more about the year-long series here.
This is the warm-up Marathon match challenge.
 

INTRODUCTION

Streamflow forecasts are integral to water management. With higher skill forecasts, water managers can likely better operate facilities for high flows, to mitigate drought impacts, and to achieve other improved outcomes (e.g. hydropower generation). 

The growing capability of Artificial Intelligence (AI) / Machine Learning (ML) and High-Performance Computing (HPC) has begun to be applied toward generating streamflow forecasts. This has the potential to complement traditional physically based streamflow forecast methods via hybrid or purely data-driven approaches. This challenge aims to spur innovation in the field of streamflow forecasting, with emphasis on assessing the potential application of ML/AI for streamflow forecasting. 

PRIZE STRUCTURE

The total sum of all prizes is $34,000.

TASK OVERVIEW

Your task is to predict a 10-day streamflow forecast in 6 hours intervals for the specific locations. For each location, you must return the streamflow forecast in cubic feet per second  at 06:00, 12:00, 18:00 and 00:00, from the 1st to the 10th day (i. e., 40 values per location; the first value should be for 06:00 on the 1st day, the last value should be at 00:00 on the 11th day - which is the end of the 10th day). All times are in UTC (also called GMT). 

The quality of your algorithm will be judged by how closely the predicted streamflow matches the actual, measured values. See Scoring for details.

We will evaluate the solutions both on live and past data. 

INPUT DATA

There is no official data set. We have prepared a list of data sources which we expect may be useful for the task. It is located here. You are also allowed to use any other data, provided it is freely available. You may also use data that may not be freely available provided: (1) they can be made available to Topcoder for scoring/validation and (2) they could be procured by the client for future use.

When predicting the past data, you are allowed to use only data from the time period prior to the target date.

GROUND TRUTH DATA

In this challenge, the task is to provide the forecast for the following locations:
 

We will use the following source to generate ground truth for scoring:

https://nwis.waterservices.usgs.gov/ - all locations except for ESSC2

https://dwr.state.co.us/ - location ESSC2

These websites offer various services which you may find useful for generating your predictions. It contains real-time and historic streamflow values for locations across the United States. 

See, e. g., this info about the sites, generated by one of the services available on the first above mentioned website.

We provide the tool to download ground truth values for the locations which will be used in this challenge.

OUTPUT FILE

In this contest you must submit a docker container which may be used to generate the forecast. The output must be created as a single CSV file. The file must be formatted as follows:

DateTime,LocationID,ForecastTime,VendorID,Value,Units

 where

• DatetTime is date and time for which we predict the streamflow, in YYYY-MM-DDThh format, e. g., 2020-09-20T18,

• LocationID is the location identifier,

• ForecastTime is date and time of the beginning of the 10 day period for which the prediction is being made, in the same format as the first column,

• VendorID is TC+ followed by your Topcoder handle, e. g., if your handle is Alice, you should set this to TC+Alice,  

• Value is the streamflow forecast for the given location, date and time, in cubic feet per second,

• Units is CFS. 

Your solution file should contain the above header line (optional), plus exactly 400 lines (mandatory): 40 lines for each of the 10 target locations.

Sample lines:

DateTime,LocationID,ForecastTime,VendorID,Value,Units
2020-09-20T06,NFSW4,2020-09-20T00,TC+your_TC_handle,8710,CFS
2020-09-20T12,NFSW4,2020-09-20T00,TC+your_TC_handle,9120,CFS
2020-09-20T18,NFSW4,2020-09-20T00,TC+your_TC_handle,9270,CFS
2020-09-21T00,NFSW4,2020-09-20T00,TC+your_TC_handle,9470,CFS
2020-09-21T06,NFSW4,2020-09-20T00,TC+your_TC_handle,9790,CFS
. . . 

SUBMISSION FORMAT

This match uses the "submit code" submission style. The required format of the submission package is specified in a submission template document. This current document gives only requirements that are either additional or override the requirements listed in the template.

Sample submission 1 package is available here.

You must not submit more often than 3 times a day. The submission platform does not enforce this limitation, it is your responsibility to be compliant to this limitation. Not observing this rule may lead to disqualification.

An exception from the above rule: if your submission scores 0, then you may make a new submission after a delay of 1 hour.

The train.sh script has no input arguments. Note that train.sh will NOT be used during the provisional testing phase NOR during the final testing phase. We shall use it only in the validation phase and only for the submission winning prizes. You are not obliged to provide train.sh in your submission. We will ask the winners to provide it with the documentation (see the “Final Prizes” section at the end of this document).

The test.sh script has this signature (which is different from the one in the template!):

test.sh <target-dates> <output-file>

<target-dates> is the comma-separated list (without spaces) of dates of the first days of the 10-day periods for which the forecast is being calculated, in the YYYY-MM-DD format.

<output-file> is a file where your code must save its forecast in the format specified in the "Output file" section. 

A sample call:
./test.sh 2020-01-20,2020-02-10,2020-03-01 /workdir/forecast.csv

This should generate the file with forecasts for these three periods:

•  from 2020-01-20T06:00 to 2020-01-30T00:00,

•  from 2020-02-10T06:00 to 2020-02-20T00:00,

•  from 2020-03-01T06:00 to 2020-03-11T00:00.

CODE REQUIREMENTS AND SUGGESTIONS 

• The allowed time limit for the test.sh script is 15 minutes × (number of target dates) during the provisional/final testing and 2 hours during the validation phase.

• The allowed time limit for the train.sh script is 7 days.

• Hardware specification. Your docker image will be built and your test.sh script will be run on a Linux AWS instance with this configuration: m4.xlarge, 500 GB volume size. In case you place on a prize winning position, the validation of your train.sh script (and test.sh afterwards) will be done on m4.10xlarge or p3.2xlarge Linux AWS instance, based on your preference. Please see here for the details of these instance types.

• We understand that the instance type (m4.xlarge) and the allowed running time (15 minutes) may be very limiting. Also, dockerizing your solution may be a difficult task which may be worth doing only in case you win a prize. Therefore, if you are able to calculate the forecasts on your own system, but fail to set it up for our system, you can consider doing the following:

  • Calculate the forecast on your system and upload it to a location where it can be downloaded (Dropbox, Google Drive, ...).

  • Set up your test.sh script so that it downloads the precalculated forecast.

Sample submission 2, which uses this approach, is available here. If you do it this way, you must make sure that your precalculated forecast is available at the time when we call your test.sh script. See “Provisional and final testing” section below for more details on this. Also note that in case you win a prize, you will have to provide the dockerized version of your solution with an updated test.sh script which actually calculates the forecast. 

PROVISIONAL AND FINAL TESTING

During provisional testing your code will be executed using 6 target prediction dates in the range from 2019-01-01 to 2019-12-31. 

During final testing your code will be executed in two different ways:

• Live forecast testing - with target dates following right after the submission phase. For each target day, we will execute the code always one day before the target date. On each day, we will start executing the solutions at 15:00 UTC and finish before 23:00 UTC. We will use at least 5 different target dates for final tests, all between 2020-09-29 and 2020-10-12.

• Past forecast testing - with target dates from the past. We will use at least 20 different target dates for final tests, all between 2019-01-01 and 2020-08-31.

For each target date, in your test.sh you are allowed to use only data known prior to the target date. If you use any precalculated models (built by your train.sh), in case of past forecast testing you are allowed to use only data known prior to 2019-01-01 when building these models. This limitation does not apply for live forecast testing - you are allowed to use different models for live and past forecast testing. 

During provisional and final tests, we will only run your test.sh script. In case you place on a prize winning position after the final tests, we will run your train.sh followed by your test.sh with new target dates, to verify that your system can provide the forecast for any date in the future and any date in the past after 2019-01-01. We will also verify whether your solution complies with the rules stated in the previous paragraph.

SCORING

During scoring, your solution CSV files (generated by your docker container) will be matched against expected ground truth data using the following algorithm.

If your test.sh script doesn't generate the expected output file (e.g. because it terminates with an error), you will receive a score of 0.

If your solution is invalid (e.g. if the tester tool can't successfully parse its content), you will receive a score of 0.

It may happen that ground truth is not known for a specific location and time (e. g., it may be impossible to measure the streamflow when the stream partially freezes). If at least 10% of the values out of the 40 values (that is, at least 4 values) for a given location and target date are unknown, the location is excluded from scoring at this target date. Note that only values with timestamps 00:00, 06:00, 12:00 and 18:00 are relevant for the scoring purposes; observed values with different timestamps do not affect the score.

Otherwise your score for a single target location and a single target date is calculated as:

NSE_W = (NSE_1_10(actual, predicted)+NSE_6_10(actual, predicted))/2,

where actual is the true observed value (streamflow) at each location and time, predicted is your algorithm's output at the corresponding location and time, and NSE_1_10 and NSE_6_10 is Nash–Sutcliffe model efficiency coefficient calculated over all the 10 days and last 5 days of the prediction range, respectively. 

Then NSE_W_Avg_Past and NSE_W_Avg_Live is calculated as the average of NSE_W values over all locations and all target dates from the past forecast and live forecast testing, respectively.

For the provisional score, NSE_W_Avg_Overall is equal to NSE_W_Avg_Past. For the final score, we set NSE_W_Avg_Overall = (NSE_W_Avg_Past + NSE_W_Avg_Live)/2.

Finally, for display purposes your score is mapped to the [0...100] range (100 being the best), as:

score = 100 / (2 - NSE_W_Avg_Overall)

The scoring script is available here.

GENERAL NOTES

• This match is not rated. 

• Teaming is allowed. Topcoder members are permitted to form teams for this competition. If you want to compete as a team, please complete a teaming form. After forming a team, Topcoder members of the same team are permitted to collaborate with other members of their team. To form a team, a Topcoder member may recruit other Topcoder members, and register the team by completing this Topcoder Teaming Form. Each team must declare a Captain. All participants in a team must be registered Topcoder members in good standing. All participants in a team must individually register for this Competition and accept its Terms and Conditions prior to joining the team. Team Captains must apportion prize distribution percentages for each teammate on the Teaming Form. The sum of all prize portions must equal 100%. The minimum permitted size of a team is 1 member, with no upper limit. However, our teaming form only allows up to 10 members in a team. If you have more than 10 members in your team, please email us directly at support@topcoder.com and we will register your team. Only team Captains may submit a solution to the Competition. Notwithstanding Topcoder rules and conditions to the contrary, solutions submitted by any Topcoder member who is a member of a team on this challenge but is not the Captain of the team are not permitted, are ineligible for award, may be deleted, and may be grounds for dismissal of the entire team from the challenge. The deadline for forming teams is September 11th, 2020. Topcoder will prepare a Teaming Agreement for each team that has completed the Topcoder Teaming Form, and distribute it to each member of the team. Teaming Agreements must be electronically signed by each team member to be considered valid. All Teaming Agreements are void, unless electronically signed by all team members by September 18th. Any Teaming Agreement received after this period is void. Teaming Agreements may not be changed in any way after signature.

• Relinquish - Topcoder is allowing registered competitors or teams to "relinquish". Relinquishing means the member or team will compete, and we will score their solution, but they will not be eligible for a prize. Once a competitor or team relinquishes, we post their name to a forum thread labeled "Relinquished Competitors".

• Use the match forum to ask general questions or report problems, but please do not post comments and questions that reveal information about the problem itself or possible solution techniques.

• In this match you may use any programming language and libraries, including commercial solutions, provided Topcoder is able to run it free of any charge. You may also use open source languages and libraries, with the restrictions listed in the next section below. If your solution requires licenses, you must have these licenses and be able to legally install them in a testing VM (see “Requirements to Win a Prize” section). Submissions will be deleted/destroyed after they are confirmed. Topcoder will not purchase licenses to run your code. Prior to submission, please make absolutely sure your submission can be run by Topcoder free of cost, and with all necessary licenses pre-installed in your solution. Topcoder is not required to contact submitters for additional instructions if the code does not run. If we are unable to run your solution due to license problems, including any requirement to download a license, your submission might be rejected. Be sure to contact us right away if you have concerns about this requirement.     

• You may use open source languages and libraries provided they are equally free for your use, use by another competitor, or use by the client. If your solution includes licensed elements (software, data, programming language, etc) make sure that all such elements are covered by licenses that explicitly allow commercial use.

• If your solution includes licensed software (e.g. commercial software, open source software, etc), you must include the full license agreements with your submission. Include your licenses in a folder labeled “Licenses”. Within the same folder, include a text file labeled “README” that explains the purpose of each licensed software package as it is used in your solution.     

FINAL PRIZES

In order to receive a final prize, you must do all the following:

• Achieve a score in the top 15 according to the final test results. See the "Scoring" section above.

• Once the final scores are posted and winners are announced, the prize winner candidates have 7 days to submit:

  • Report outlining their final algorithm explaining the logic behind and steps to its approach, including the documentation on how to run the code. You will receive a template that helps you create your final report.

  • Updated dockerized version of your algorithm with working train.sh and test.sh scripts, along with any assets/materials necessary to deploy, use and train it. The code in your container should produce the same results as in your submissions.

• If you place in a prize winning rank but fail to do any of the above, then you will not receive a prize, and it will be awarded to the contestant with the next best performance who did all of the above.