Prize

1st: $2000

2nd: $1500

3rd: $750

4th: $500

5th: $250

 

Challenge Overview

Over the last few months, a series of challenges have been run to generate a series of high quality financial forecasts for a consumer brand telecommunications provider, ‘Sandesh’.  These challenges have been known as ‘CFO Forecasting’ in various formats.  As a result, a high quality, high accuracy series of algorithmic forecasts have been produced for a number of financial target variables.

 

This challenge is being initiated to improve the forecasts for a subscription based Entertainment over IP service and a group of pay television sports channels offered by Sandesh.

 

As we all know, the COVID-19 pandemic has changed the world significantly and this has impacted consumer brands and their products, meaning that different variables have been affected at different levels.  Some variables don’t appear to have been impacted; some are slightly disrupted, and others are strongly affected.

 

We have categorised these variables into 3 types based on the level of Covid-19 disruption:

1. No Disruption – No obvious impact due to Covid-19.  

2. Minor Disruption – Some disruption due to Covid-19, however the trajectory / trend has recovered to pre Covid-19 levels

3. Major Disruption – Disrupted due to Covid-19, however the trajectory / trend has not recovered and a lasting impact is expected.

 

Please refer to this blog on ‘Forecasting Time Series in COVID-19 Days’ for more details:

https://blogs.sap.com/2020/05/26/forecasting-time-series-in-covid-19-days/

 

Business Context

 

In this challenge, we need to extend the forecast for variables with minor disruptions (mentioned in the section: "variables with minor disruptions for Sandesh Entertainment over IP" and "variables with minor disruptions for Sandesh Sport") from September 2020 to March 2022. So the whole focus will be variables with minor disruptions for the following products:

 

1. Sandesh Entertainment over IP

2. Sandesh Sport

 

Task Details

We will present a set of variables and include:

1. Their privatized values until Aug 2020, and 

2. Related models.

 

The task for each variable (minor impacted only as mentioned in "variables with minor disruptions for Sandesh Entertainment over IP" and "variables with minor disruptions for Sandesh Sport") will be to review the historic data provided and create a 19th month forecast starting from September 2020 and ending in March 2022 (19 months).

 

As we all know, the Covid-19 pandemic has caused major disruptions to businesses of all types, and with this in mind, the community is asked to consider any type of impact caused by the pandemic.  In the UK the major disruption started in March 2020 and some areas have been recovering to some extent since about July 2020.  The community is requested to consider the impact during and the recovery after the Covid-19 period and create a credible 19 month forecast from September 2020 to March 2022.

 

The variables with minor disruptions for Sandesh Entertainment over IP are:

• Gross Adds – the number of new subscribers joining the brand during a month

• Leavers - the number of subscribers who terminated service during that month.

 

The variables with minor disruptions for Sandesh Sport are:

• Gross Adds – the number of new residential customers joining the brand during a month

• Leavers - the number of residential customers who terminated service during that month.

• Revenue (Total) – the total revenue generated by customers of all types of this product per month.

• Residential Revenue – the revenue generated by residential customers only of this product per month.

 

We will also share the baseline model for each variable. Some necessary documentation will be provided for you to run these models. These models produced best MAPE for the variables mentioned above. You can tweak the current model or remodel it or use completely different algorithm to create best results for the above mentioned 6 variables.

 

The table below shows the MAPE achieved on real data with the baseline models:

The dataset and baseline models will be shared in the forum. Note that, in this challenge, there is no threshold or target.

 

Your submission will be judged on three criteria:

1. Business credibility of your plots of 19-month predictions from Sept 2020

2. A White paper

3. Minimizing error (MAPE)

 

The full evaluation criteria will be outlined in the Quantitative Scoring section below.

Business Insight

Sandesh Sport holds exclusive live television rights to a number of popular football competitions which drive volumes and revenues due to high demand from subscribers and other users.

 

As Sandesh has exclusive rights to these competitions, the dates that these football events occur may contribute to seasonality in the data.  We have provided data sets to represent the broadcast dates for two football competitions and one data set for when price changes occurred for these products.

 

Note that price changes are communicated to customers a minimum of 30 days in advance of a price change, and the price change is typically seen by customers in their bill a month after the price change takes effect.  

 

The data sets that we have provided should be used as exogenous variables for prediction in your model.   The exogenous variables to be used are:

1. Top National League (Football)

2. Top Regional Winners League (Football)

3. Price changes (for both products)

Financial Year modeling

Sandesh reports its financial year from April - March.  This may contribute to seasonality based on financial year, and quarters (Jun, Sep, Dec, and Mar), rather than calendar year.

 

Anonymised and Privatised data set

 

‘Z-score’ is used to privatise the real data.

 

For all the variables, following is the formula used to privatise the data:

            zi = (xi – μ) / σ

 

where zi = z-score of the ith value for the given variable

            xi  = actual value

            μ = mean of the given variable

            σ = standard deviation for the given variable

Quantitative Scoring

Quantitative Scoring will consist of two parts:

 

MAPE on Prediction Window

 

The MAPE (Mean Absolute Percentage Error) of the predictions on the privatised data set should be provided over a period of 5 months from April 2020 to August 2020. 

 

Given two values, one ground truth value (gt) and one predicted value (pred), we define the relative error as:

 

    MAPE(gt, pred) = |gt - pred| / |gt|

 

Stress testing/Robustness testing methodology:

 

Once model building is done, robustness of the model is to be calculated. For this we need to do evaluation on a rolling forecasting from origin. See the below image for understanding. Forecast window length is 18 months.

 

 

1. Every horizontal line represents one iteration. 

2. Blue windows are the training period and Orange windows are the forecasting period.

3. One separate directory to be created with the name “robust” to store the forecast for all the iterations.

4. Forecast to be saved in a .csv file with the name submission affixed with the iteration number.

5. Separate python function/module to be created, which will call the model to generate the forecast for different periods as explained in the above image. This is required for code modularity. 

6. The function/module for the robustness, should have below input parameters.

1. Start date of the training window of the iteration 1. (Start date of the series)

2. End Date of the training window of the iteration 1. (August 2020)

3. Forecast period i.e number of months to forecasts. (18 months)

4. Number of iteration. (12 iterations to be done)

5. For subsequent iteration Train/Forecast start and end month should be automatically calculated based on the input given in step A and B as shown in the above image.

7. While running this module the model parameter will be fixed. For an example if it’s an ARIMA model then p,d,q values should be the same throughout all the iteration. If it’s a LSTM then hyper param such as epochs, number of LSTM units, look back/look forward etc should be the same as your final model built for date range mentioned in point 8. All the iterations should run on the same parameter configuration. 

Note: For rolling window accuracy refer https://otexts.com/fpp2/accuracy.html

Final Submission Guidelines

Submission

Your submission should include 

• A codebase. It should be able to take an input spreadsheet as well as the start time period of the COVID (or other events that potentially have impact), and then output the forecast for 19 months (starting from September 2020).

• Plots of your 19-month predictions from Sept 2020. We will judge these plots visually and consider whether they are credible for the business over the long term.  When plotting the curves, please include the data points from Aug 2017 to Aug 2020 too.

• Stress testing results. The predicted files for all stress testing periods. The robustness score should be reported in your whitepaper too.

• A whitepaper. It is a text, .doc, PPT or PDF document that includes the following sections and descriptions:

  • Overview: describe your approach in “layman's terms”

  • Methods: describe what you did to come up with this approach, eg literature search, experimental testing, etc

  • Materials: did your approach use a specific technology?  Any libraries?  List all tools and libraries you used

  • Discussion: Explain what you attempted, considered or reviewed that worked, and especially those that didn’t work or that you rejected.  For any that didn’t work, or were rejected, briefly include your explanation for the reasons (e.g. such-and-such needs more data than we have).  If you are pointing to somebody else’s work (eg you’re citing a well known implementation or literature), describe in detail how that work relates to this work, and what would have to be modified

  • Data:  What other data should one consider?  Is it in the public domain?  Is it derived?  Is it necessary in order to achieve the aims?  Also, what about the data described/provided - is it enough?

  • Assumptions and Risks: what are the main risks of this approach, and what are the assumptions you/the model is/are making?  What are the pitfalls of the data set and approach?

  • Results: Did you implement your approach?  How’d it perform?  If you’re not providing an implementation, use this section to explain the EXPECTED results.

  • Other: Discuss any other issues or attributes that don’t fit neatly above that you’d also like to include

Judging Criteria

You will be judged on the quality of your ideas, the quality of your description of the ideas and its effective demonstration on sample data along with success measures. The winner will be chosen by the most logical and convincing reasoning as to how and why the idea presented will meet the objective. Note that, this contest will be judged subjectively by the client and Topcoder. However, the judging criteria will largely be the basis for the judgement.

 

1. Effectiveness (40%)

   1. Is your algorithm effective?

      1. A stress test (i.e., robustness evaluation) is required. This will serve as a reference during our evaluation. 

      2. We will also evaluate your plots visually and see if it makes sense or not.

   2. Is there any interesting insights from the data analysis?

2. Feasibility (40%)

   1. Is your algorithm efficient, scalable to large volumes of data? 

   2. Is your implementation general enough? There should be no hardcoded hyper-parameters for your model --- they should be all dataset-adaptive.

3. Clarity (20%)

   1. Please make sure your report is easy to read.

   2. Figures, charts, and tables are welcome.

Submission Guideline

Only your last submission will be evaluated. We strongly recommend you to include your great solution and details as much as possible in a single submission.