EPA Android App and PM - Back End Module Architecture

1. Project Overview

The EPA is a U.S. federal government agency devoted to safeguarding the environment. One of the EPA's great concerns is the proliferation of cyanobacterial harmful blooms (cyanoHABs) in the nation's lakes. The following resources provide information on what cyanoHABs are and how they threaten the environment.

• EPA: Cyanobacterial Harmful Blooms (CyanoHABs)
• University of North Carolina: Cyanobacteria [video]

The TopCoder project on cyanoHABs aims to develop an algorithm that will be deployed in an Android app with mapping and data visualization capabilities. The app will inform local and federal policy makers about locations where bloom events are likely to occur, allowing them to concentrate their efforts in those areas.

2. Contest Overview

Welcome to second architecture in the series of four EPA Android App Architecture Contests. Each contest will run after the other and will be using the output of previous contest to build further. We recently concluded the first contest which designed the data management backbone for the system. This contest will use that architecture and will design the complete backbone of the system.

In this contest, we are looking for you to design the module architecture for backend module system of EPA Android App. We want you to use all the available information architecture in form of system architecture and application requirements specification to develop this module architecture. We will also provide access to the admin website front-end that has been already developed and which will interact with this module. We will provide the front-end App prototype which will have maximum interfacing with this module and which will send direct requests to this module. Finally, we will provide recently designed data management architecture which will be working side-by-side with this module to form the complete backbone of the system.

 

Following is the description for each of functionality of data management module:

EPA Android App Data Management Module: This module will be responsible for doing all the back-end operations of the app. It will house prediction algorithm(s) (to be developed separately in marathon match) and statistical analytic engine (to be built as part of this module) in addition to mundane backend tasks of message passing and information transfer. Hence, this module will be computationally intensive and will require great performance consciousness. Please note that performance is the key for this module. So, please make sure to include all possible optimizations so as to make this a very efficient module.

Following are key components for backend:

1.) Prediction algorithm: This algorithm will be developed as a part of marathon match. Currently, you need to make the backend as flexible as possible so as to accommodate the algorithm. The match will launch in a couple of days so we will be able to share with you all the structures that will be used in the algorithm on forums. But there is one thing to note: All the data that will come to the back-end from data management module for this algorithm will be in csv format. The header information for all the data will also be very similar. Finally, the output of the algorithm will also be in csv format. The header information of the output format is currently deemed to be:

Lat (center of prediction area) , Lon (center of prediction area), Date of Estimated value (current day + 7 / 14 / 28), predicted value

This is our current assumption to bring versatility in the output. This is subject to change, so please make sure to consider this as flexible enough.

 

Please read carefully: It may be that we want to use multiple algorithms and assign each place to those algorithm that we think of works best in that specific case. In such case the server backend will also have to collect results of different algorithms and pack them into one output stream for the App. This is why we believe the above header format for output will be helpful in such cases. But please make sure to include the flexibility for more algorithms.

2.) Statistical Engine: We want to house a small statistical engine that will calculate basic statistical values from the provided data. It will also use the same data as provided to the algorithm. The data will come from common store where data management module will have put the processed data. The list of statistics and metrics that need to be calculated after analyzing the data will be provided in forums.

3.) Interaction with Data Management Module: The data management module (architecture provided) will process all the data and put it in common store (as defined in data management module arch). The Back-end will read this data. The data will be of two types: 1.) csv files which will go to algorithm. 2.) jpg/png image files which backend will directly push to front-end as required for display. Please capture any other interaction that with data management module as described in its architecture.

4.) Interaction with Admin Website: The admin will configure certain parameters for the automatic run of the prediction algorithm. The backend will have to use those parameters. Those parameters will be sent to back-end by admin website. Whenever an algorithm run is completed, the back-end will send a mail to the admins. The mail address will be configured by admin website.

5.) Interaction with Front-End: Please capture all interactions with front-end as seen in ARS and SDS. Please consider that you will be sending both image (png) and data to the front-end.

6.) Logging: Please capture all logging scenarios as described in SDS. Please ask in forums if there is any doubt.

7.) Extra Tools: Please make the architecture flexible to add extra components later. For example, we may be required to add an online tester for the algorithm at later stage. No extra work is required currently but just keep it an open architecture to that end as to be easily extensible.

3. Technology Overview

1. The Android App front-end has been developed for Android 4.1-4.4 support using android developer tools. It also uses Google Maps Android API v2, AChartEngine 1.1.0 and ProgressWheel.

2. The Data Standardization component (to be embedded in data management module) is developed in Java.

3. The Admin Website is a CMS developed using Django framework 1.6.1, Python 2.7x and PIL 1.1.7.

4. We are open to use of any technology for the backend as far as it does not limit any specific OS requirements.

5. We plan to host the backend  module along with the data management module and admin website on a single (same) server - ex. Amazon EC2 Server.

6. Open source software resources are welcome, but they must have third party support services available. Please ask in forum when in doubt.

7.  You are allowed to use any open source DB, data storage mechanisms for storing data at various stages.

8. Please note that the developers are not allowed to use any component from TC Catalog for this contest.

 

4. Process Flow and Storage Considerations

- We will use two main storage modules: Common Storage and Back-End Storage.

- The data management module will store all processed data into common store. Please capture the sync issues that may occur if back-end tries to read when DMM is still processing and writing data to common storage. We cannot afford dirty reads here.

- Back-end will read the data, work on it through the algorithm and stats analyses and then store its results into the back-end storage which can either be database or file storage.

Please Note: Both the process flow and storage details are just high-level suggestion from our side. Architects are welcome and encouraged to suggest and propose any changes and updates as deemed necessary in this process flow. Also, architects are allowed to use any media for data storage like DB or files, etc. Please open a discussion in forum to clarify/confirm your ideas when in doubt.

5. Resources Provided

The following resources have been provided in the forums. You will be able to access it after registration:

1.) EPA Android Mobile Front-End Prototype

2.) EPA Admin Website Prototype.

3.) System Design Specification

4.) System Architecture TCUML

5.) Application Requirements Specification and Use Case TCUML

6.) Data Management Module Architecture

Please Note: In case of discrepancies, the System Architecture and ARS takes precedence over all other resources as they are the most update one. Please follow them and if there is still confusion, please ask in forums.