Register
Submit a solution
The challenge is finished.

Challenge Overview

Challenge Overview

 

The Green Button Standard is a secure way of transmitting energy usage information developed by the US Department of Energy and National Institute of Standards and Technology.

The goal of this challenge is to develop a Python script to allow third-party applications to retrieve electricity consumption information by downloading the Green Button XML formatted data, validate it, parse it, and generate hourly electricity consumption data (including cost and start time) for every hour of the day and store it in MySQL data table.

This follows the path known as Green Button Connect in the Green Button Documentation.

Challenge Requirements

General Notes

  • This version will only handle UsagePoints of service category (kind) = 0 (Electricity), if the downloaded data is not of type Electricity, the service will then return an error to the caller stating that we only support Electricity data in this version.

    • For more information about UsagePoints kinds, refer to this xsd file that has rich documentation about the entities in GreenButton.

  • Refer to this python library as you will find useful for how we should do parsing.

Python script Requirements

  • It is a command line script.

  • Be compatible with Python version 2.7.3 running on Ubuntu 12.04.2 LTS

  • MySQL database - latest stable version will be used.

  • The script will read following configuration :

  • The execution of the scription :

    • It firsts read records from ‘sm_gbUsers’ table

    • for each returned record :

      • If case is ‘api’

        • construct the API url by using resourceURI configuration and sm_gbUser.user_id (user_id will be used to fill the {subscriptionId} path parameter in resourceURI)

        • invoke the API to download the XML data, use ‘sm_gbUser.access_token’ to set ‘Authorization’ header

      • if case is ‘datafile’

        • Read the xml file from ‘sm_gbUsers.path’

      • validate the XML data (use this xsd file)

        • If not valid, log the information and continue to next record

      • parse the interval readings and convert it into hourly data :

        • Parse interval reading duration :

          • <espi:intervalblock> → <espi:IntervalReading> → <espi:timePeriod> →    <espi:duration>

          • The duration is in seconds

        • if the time duration represents an a hour (3600 seconds) then group all interval readings into a collection where key is the interval reading timestamp and value is an object with cost/value.

        • if the time duration is more or less than 3600 seconds then you need to implement an algorithm that calculate average hourly data for the interval reading.

          • This part of requirement is tricky, but for the algorithm you write you can assume :

            • That the Data Custodian standard supports any duration  from 1 second to 2^32-1.

            • IntervalBlock has a start and duration that should bracket all IntervalReadings contained. You can expect that they (IntervalReadings) will not be overlapping but might be discontiguous (in case readings were missed in communications, for example from meter to utility). So you need also to take discontiguous data into account when calculating average hourly data.

        • Group the parsed data into collection of key/value pairs, where key is the interval reading hour start time (in seconds) and value is an object holding two attributes : cost and value.

      • For each entry in the key/value collection insert new record into ‘sm_hourlyDataDummy’ table :

        • ‘ID’ is auto increment column

        • ‘user_id’ should be taken from ‘sm_gbUsers.ID’ field

        • ‘start_time’ is read from collection item key - convert it to db datetime.

        • ‘SiteID’, ‘GHI’, ‘clearness’, and ‘electricity’ should be set to null

        • ‘GBD_consumption’ is read from collection item value object (object#value)

        • ‘GBD_cost’ is read from collection item value object (object#cost)

Abstracting and Design Patters

Please make sure to create helper/interface to include the common code/functionality.

Also please make sure your design is flexible, use facade design pattern and adapter design pattern (and any other proper design pattern) to enable future extensibility of the solution.

GreenButton Data Model

You can find details about Greenbutton model definition in this page.

Testing and Sample Data

For ‘api’ case :

You need to pass OAuth 2.0 access token when communicating with the API.

Sandbox has OAuth server turned-off so you will be using a predefined-tokens listed in this page for testing purpose.

You will use Sandbox http://energyos.github.io/OpenESPI-GreenButton-API-Documentation/API/

Sometimes sandbox goes down for unknown reasons, you can use this XML to simulate call to the API

(https://drive.google.com/open?id=0By6n2kEXozMwVzhhc2U3S3o0NE5xeGQyRGRleTFoQnNwTnlZ&authuser=0)

For ‘datafile’ case :

You can simply download a sample XML file from API or google drive and store locally. There should be no difference between XML structure between API or local file.

You can get sample data from here http://services.greenbuttondata.org/sample-data.html

MySQL Table Structure

`sm_gbUsers` (
 `ID` : integer, auto-increment, non-null, primary key

 `case` : ENUM(‘api’, ‘datafile’) not null

 ‘path’ : string, nullable, represents path to xml file

 ‘access_token’ : string, nullable, represents the retail customer access token

 ‘user_id’ : integer, nullable, represents the subscription id
)

`sm_hourlyDataDummy` (
 `ID` : integer, auto-increment, not null, primary key

 `user_id` : reference ‘sm_gbUsers.ID’, not null, foreign key

 `siteID` int(11) NULL, always null in this challenge
 `GHI` float(10,6) NULL, always null in this challenge
 `clearness` float(10,6) NULL, always null in this challenge
 `electricity` float NULL, always null in this challenge

 ‘start_time’ : date time, not null
 `GBD_consumption` float(10,6) NOT NULL
 `GBD_cost` float(10,6) NOT NULL
)

Writing Tests

All of your code should be test covered.

As mentioned above, you can use the Sandbox or host the sample data to test your solution.

If you see any problem with sandbox stopped behaving normally, post in challenge forums, and we will reset it (Sandbox).

Please note that your tests must cover both Data Usage that are hourly and non-hourly. You can use data file to test that, and also you can use API in sandbox.

If sandbox does not contain hourly or non-houly interval readings you can use the POST endpoints to create test data by uploading the sample xml files.

Coding Standard

Follow python coding best practices : PEP 8 for the main text, and PEP 257 for docstring conventions

Documentation

Your solution must be well documented.

Readme

Provide a detailed readme file using Markdown language with following information :

  • Overview

  • Setup Prerequisites

  • Usage Example

  • Reference to GreenButton API Documentation

  • Reference this Python Documentation

  • Any details about any limitations of your solution.

Please note, we're judging this competition not just on the code, but also on the quality of the documentation, test coverage, and ease of use.

References

 



Final Submission Guidelines

Deliverable

  • All source code files and scripts that addresses the challenge requirement.

  • Detailed readme file as clarified above.

 

ELIGIBLE EVENTS:

2015 topcoder Open

Review style

Final Review

Community Review Board

Approval

User Sign-Off

ID: 30048666