Key Information

Register
Submit
The challenge is finished.

Challenge Overview

Challenge Objective

The objective of this challenge is to improve on the accuracy of four previous time series forecast models generated in a previous challenge.  The new models need to, at least, have an awareness of a secondary variable (Price Increase) to be described later at the least, but may need further enhancements.  The previous models have been provided as a foundation for the challenge.

  • The accuracy of the winning model must improve on the MAPE error on the privatised data set compared to the foundation model.

  • The winning submission must model the impact of Price increase and, as a result, be able to predict ARPU and Revenue based on the introduction of a subsequent price increase - see July 2019 actuals.

  • The winning model must also reduce the maximum Absolute Percentage Error (APE) compared with the foundation model.

  • The model should be tailored to a 12-mth forecast horizon but must be extendable beyond this time period.

  • The accuracy of a prediction will be evaluated using MAPE error on the privatised data set over a period of 6 months.

Challenge Details

The two target variables are: 

Average Revenue per User (ARPU) is the average revenue generated by a single customer in the full month in question.  

Revenue is the monthly revenue generated by these products, and is therefore a function of the ARPU and the volume of customers.

For the two Broadband products are: 

• Tortoise (legacy product, declining product, available everywhere)

• Rabbit (biggest product, reaching maturity, available in most of the country)

The relationship between key financial variables

  • Revenue = Average Volume of Customers over the period * Average Revenue per existing Customer over the same period.

ARPU performance demonstrates time series features ...

This chart of ARPU for the two products reflects the relative performance of the two products as shown here - x and y axis intersect at £0 ARPU.
Charts include actual performance through to Oct 19, and predictions from the foundation models from Apr 19 to Oct 19.
Data (anonymised and privatised) is provided to Mar 19 only. Predictions for Apr 19 - Oct 19 will be used to evaluate the submissions.

Key aspects to consider …

  1. An outlier (accounting anomaly) in Jan 2019.  This ‘one-time’ drop in ARPU for Tortoise and, to a lesser extent, Rabbit is an accounting anomaly and should be treated, and smoothed.  The proposed treatment should be built into the model, adequately explained and be reproducible when transfered to the real data set.

  2. Data sets for both Tortoise and Rabbit upto and including March 2017 are unreliable.  Data point prior to April 2017 should be excluded; the approach built into the models; and reproducible when transfered to the real data set.

  3. Price Increases are impacting as a ‘Level Shift Outlier’ in the forecasting of ARPU.

In Jan ‘18 and Jul ‘19, a price increase was imposed on the existing customer base. This price increase introduced a ‘level shift’ situation into both products and is clearly visible in the actuals charting above.

The winning model will need to consider the impact of price increase, and take this into account in the future so as to accurately forecast from Apr ‘19 as part of the evaluation of submissions.  Details of the timing and relative magnitude of the price increases are documented below - both historic and future planned increases.

Revenue performance demonstrates similar time series features …

 

This chart of Revenue for the two products reflects the relative performance of the two products as shown here - x and y axis intersect at £0.
Charts include actual performance through to Oct 19, and predictions from the foundation models from Apr 19 to Oct 19.
Data (anonymised and privatised) is provided to Mar 19 only. Predictions for Apr 19 - Oct 19 will be used to evaluate the submissions.

Key aspects to consider for Revenue, as with ARPU…

  1. An outlier (accounting anomaly) in Jan 2019.  This ‘one-time’ drop in Revenue for Tortoise and, to a significantly lesser extent, Rabbitt is an accounting anomaly and should be treated, and smoothed.  The proposed treatment should be built into the model, adequately explained and be reproducible when transfered to the real data set.

  2. Data sets for both Tortoise and Rabbit upto and including March 2017 are unreliable.  Data points prior to April 2017 should be excluded; the approach built into the models; and reproducible when transfered to the real data set.

  3. Price Increases are impacting as a ‘Level Shift Outlier’ in the forecasting of Revenue.

In Jan ‘18 and Jul ‘19, a price increase was imposed on the existing customer base. As with ARPU, this price increase introduced a ‘level shift’ situation into both products and is clearly visible in the actuals charting above.  

The winning model will need to consider the impact of price increase, and take this into account in the future so as to accurately forecast from Apr ‘19 as part of the evaluation of submissions.  Details of the timing and relative magnitude of the price increases are documented below - both historic and future planned increases.  

Price Increases for Tortoise and Rabbit products ...

As context, the following price increases have been applied to the customer base since 2016, and need to be built appropriately into the ARPU and Revenue models.  


 

Financial year modeling

Financial year for Sandesh is April to March (instead of January to December), hence Q1 is April, May and June.

Anonymised and Privatised data set

‘Z-score’ is used to Privatised the real data.

For all the variables, following is the formula used to privatise the data:

            zi = (xi – μ) / σ

where zi = z-score of the ith value for the given variable

            xi  = actual value

            μ = mean of the given variable

            σ = standard deviation for the given variable

Modeling Insight derived from previous challenges

A  SARIMA (univariate model) and LSTM model (multi-variate) have proven to be the best algorithms for predicting the target variables in this data set.  This model is included as the foundation for refinement.

The MAPE error rates of this foundation model on the privatised data set provided have been calculated and have been used to set the threshold targets for this challenge.  The winning model will need to maintain or improve on this accuracy while modeling the price increase impact. The predictions need to reflect the smooth, gradual drift in ARPU.

Quantitative Scoring

Given two values, one ground truth value (gt) and one predicted value (pred), we define the relative percentage error as:

    APE(gt, pred) = |gt - pred| / gt

We will average APEs over months in the test set, and then obtain MAPE. This MAPE will be the major optimization objective. The smaller, the better.

In addition to the overall performance measured by MAPE, we will also dive deep into each month and check the effectiveness on a monthly basis. We will take the maximum APE (MaxAPE) over months in the test set. This serves as a secondary optimization objective. Again, the smaller, the better. It is to some extent similar to variance, considering that we have 6 months in the test set.

Roughly, MAPE takes a role of 70% and MaxAPE has the remaining 30%.

Judging Criteria

Your solution will be evaluated in a hybrid of quantitative and qualitative way. 

  • Effectiveness (80%)

    • We will evaluate your forecasts by comparing it to the ground truth data. Please check the “Quantitative Scoring” section for details.

    • The smaller MAPE & MaxAPE, the better. 

    • The model must achieve better performance than the provided baseline models including improving the coefficient of variation.

  • Clarity (10%)

    • The model is clearly described, with reasonable justifications about the choice.

  • Reproducibility (10%)

    • The results must be reproducible. We understand that there might be some randomness for ML models, but please try your best to keep the results the same or at least similar across different runs.



Final Submission Guidelines

Your submission must include the following items

  • The filled test data. We will evaluate the results quantitatively (See below)

  • A report about your model, including data analysis, model details, local cross validation results, and variable importance. 

  • A deployment instructions about how to install required libs and how to run
     

Expected in Submission

1. Working Python code which works on the different sets of data in the same format

2. Report with clear explanation of all the steps taken to solve the challenge (refer section “Challenge Details”) and on how to run the code

3. No hardcoding (e.g., column names, possible values of each column, ...) in the code is allowed. We will run the code on some different datasets

4. All models in one code with clear inline comments 

5. Flexibility to extend the code to forecast for additional months

REVIEW STYLE:

Final Review:

Community Review Board

Approval:

User Sign-Off

SHARE:

ID: 30117191