CFO Forecast Challenge - Sandesh Brand 2 - Broadband ARPU and Revenue Refinement

Challenge Objective

The objective of this challenge is to improve on the accuracy of four previous time series forecast models generated in a previous challenge.  The new models need to, at least, have an awareness of a secondary variable (Price Increase) to be described later at the least, but may need further enhancements.  The previous models have been provided as a foundation for the challenge.

• The accuracy of the winning model must improve on the MAPE error on the privatised data set compared to the foundation model.

• The winning submission must model the impact of Price increase and, as a result, be able to predict ARPU and Revenue based on the introduction of a subsequent price increase - see July 2019 actuals.

• The winning model must also reduce the maximum Absolute Percentage Error (APE) compared with the foundation model.

• The model should be tailored to a 12-mth forecast horizon but must be extendable beyond this time period.

• The accuracy of a prediction will be evaluated using MAPE error on the privatised data set over a period of 6 months.

Challenge Details

The two target variables are: 

Average Revenue per User (ARPU) is the average revenue generated by a single customer in the full month in question.  

Revenue is the monthly revenue generated by these products, and is therefore a function of the ARPU and the volume of customers.

For the two Broadband products are: 

• Tortoise (legacy product, declining product, available everywhere)

• Rabbit (biggest product, reaching maturity, available in most of the country)

The relationship between key financial variables

• Revenue = Average Volume of Customers over the period * Average Revenue per existing Customer over the same period.

ARPU performance demonstrates time series features ...

This chart of ARPU for the two products reflects the relative performance of the two products as shown here - x and y axis intersect at £0 ARPU.
Charts include actual performance through to Oct 19, and predictions from the foundation models from Apr 19 to Oct 19.
Data (anonymised and privatised) is provided to Mar 19 only. Predictions for Apr 19 - Oct 19 will be used to evaluate the submissions.

Key aspects to consider …

1. An outlier (accounting anomaly) in Jan 2019.  This ‘one-time’ drop in ARPU for Tortoise and, to a lesser extent, Rabbit is an accounting anomaly and should be treated, and smoothed.  The proposed treatment should be built into the model, adequately explained and be reproducible when transfered to the real data set.

2. Data sets for both Tortoise and Rabbit upto and including March 2017 are unreliable.  Data point prior to April 2017 should be excluded; the approach built into the models; and reproducible when transfered to the real data set.

3. Price Increases are impacting as a ‘Level Shift Outlier’ in the forecasting of ARPU.

In Jan ‘18 and Jul ‘19, a price increase was imposed on the existing customer base. This price increase introduced a ‘level shift’ situation into both products and is clearly visible in the actuals charting above.

The winning model will need to consider the impact of price increase, and take this into account in the future so as to accurately forecast from Apr ‘19 as part of the evaluation of submissions.  Details of the timing and relative magnitude of the price increases are documented below - both historic and future planned increases.

Revenue performance demonstrates similar time series features …

 

This chart of Revenue for the two products reflects the relative performance of the two products as shown here - x and y axis intersect at £0.
Charts include actual performance through to Oct 19, and predictions from the foundation models from Apr 19 to Oct 19.
Data (anonymised and privatised) is provided to Mar 19 only. Predictions for Apr 19 - Oct 19 will be used to evaluate the submissions.

Key aspects to consider for Revenue, as with ARPU…

1. An outlier (accounting anomaly) in Jan 2019.  This ‘one-time’ drop in Revenue for Tortoise and, to a significantly lesser extent, Rabbitt is an accounting anomaly and should be treated, and smoothed.  The proposed treatment should be built into the model, adequately explained and be reproducible when transfered to the real data set.

2. Data sets for both Tortoise and Rabbit upto and including March 2017 are unreliable.  Data points prior to April 2017 should be excluded; the approach built into the models; and reproducible when transfered to the real data set.

3. Price Increases are impacting as a ‘Level Shift Outlier’ in the forecasting of Revenue.

In Jan ‘18 and Jul ‘19, a price increase was imposed on the existing customer base. As with ARPU, this price increase introduced a ‘level shift’ situation into both products and is clearly visible in the actuals charting above.  

The winning model will need to consider the impact of price increase, and take this into account in the future so as to accurately forecast from Apr ‘19 as part of the evaluation of submissions.  Details of the timing and relative magnitude of the price increases are documented below - both historic and future planned increases.  

Price Increases for Tortoise and Rabbit products ...

As context, the following price increases have been applied to the customer base since 2016, and need to be built appropriately into the ARPU and Revenue models.  

Final Submission Guidelines

Your submission must include the following items

• The filled test data. We will evaluate the results quantitatively (See below)

• A report about your model, including data analysis, model details, local cross validation results, and variable importance. 

• A deployment instructions about how to install required libs and how to run
   

Expected in Submission

1. Working Python code which works on the different sets of data in the same format

2. Report with clear explanation of all the steps taken to solve the challenge (refer section “Challenge Details”) and on how to run the code

3. No hardcoding (e.g., column names, possible values of each column, ...) in the code is allowed. We will run the code on some different datasets

4. All models in one code with clear inline comments 

5. Flexibility to extend the code to forecast for additional months