Quartz Energy - XRD API Implementation Challenge Part 2 - Optimization

Challenge Objectives

 

• Mineral densities optimization - your choice of the optimization algorithm - will be compared to the baseline using the evolutionary algorithm available in Excel

• Implement the optimization in Python - objective calculations are implemented already

 

Project Background

• Our client has developed a specific method of analyzing geological data from soil samples which involves converting Weight percent mineralogy data to volume percent and integrating the calculation with porosity and organic carbon data.

• All the calculations are currently in an Excel workbook - our goal is to create an api to store and process the data, i.e. move the calculations from Excel into the codebase.

• In a previous challenges we have designed and built an API specification and database models along with all the crud and calculation endpoints

• In this challenge we’ll implement optimization algorithm for determining optimal mineral densities

• Follow-up challenges will focus on testing (manual and automated)

 

Technology Stack

 

• Python 3

• Flask

• Flask-Restplus

• MongoDB

• SciPy / DEAP

 

Code access

 

Project repository is available at https://gitlab.com/quartz-energy/xrd-services/xrd-services  - see challenge forums to get access. Repository contains only the API specification and implementation.

 

In the challenge forums you’ll find these documents:
 

• Sample Excel files with input and output data

• Data overview presentation

• Calculations specification document

 

We recommend reading these files before moving on to the next section of the challenge spec.

 

The main goal is to find mineral densities that produce lowest average Delta Grain Density with the only constraints being the bounds (lower, upper) for each mineral. This optimization is implemented in Excel (see sheet “Six”), using evolutionary solver algorithm for optimization. Algorithm inputs will be:

1. Objective function (average delta grain density)

2. Input variables that will be optimized by the algorithm (mineral densities) - initial values available in the “mineralDensities” field

3. Constraints - mineral density bounds

 

Individual requirements

 

1. Implement the objective function
   Create the objective(fitness) function that calculates average delta grain density or average bulk density. Delta grain density is available in step6 results. Bulk density can be obtained as difference of step 6 values - apparent grain density and GRI Dry Bulk Density. Mineral densities affect calculations in steps2-6, so to calculate the average delta grain density (or bulk density), we must re-calculate all steps (2-6) for the current mineral densities

2. Implement the mineral densities optimization solver algorithm
   You can use any algorithm of your choosing to optimize the mineral densities. Good starting points are SciPy differential_evolution, or DEAP framework in case you decide on implementing evolutionary optimization. For any other libraries/frameworks ask for approval in the challenge forums. You can also use non-evolutionary based solvers as long as the final average delta grain density is smaller.

Our goal here is to get similar or better results than Excel so your submission will be judged on the final “average delta grain density” found by your algorithm. You should try to find a good combination of maximum iterations, minimum convergence rate, mutation rate, population size and any other parameters of the optimization algorithm. General expectation is that the algorithm performance (time) should not be too long (ex <20 seconds), but this is not a fixed, hard requirement (running time of 5+ minutes would not be acceptable though).

Once the algorithm finds the optimal solution, save the mineral density values to database - update InputData.mineralDensities field

3. Add API endpoint to call the optimization algorithm

Add a GET /optimize/:id?objective=X endpoint that will just call the optimization algorithm to optimize for delta grain or bulk density and return the final mineral densities and average density.

 

NOTE: reviewers will have additional test files available to them to verify algorithm outputs.

 

General requirements

 

• Unit tests are required for all endpoints.

• All endpoints should be annotated with Swagger annotations and swagger UI should be served by the API.

• All configuration parameters should be extracted to a common settings module. Sensitive configuration parameters should be set from environment variables (DB url, credentials, etc).

• Please make sure your code is well-documented. Use the following style guides Google Python Style Guide, Python Style Guide, and Docstring Conventions. Code linter is required. Please make sure it is well-engineered but not over-engineered (YAGNI and KISS) solution. We're looking for well-structured and tested code. Well-structured code follows good design principles like the SOLID principles and well-tested code has comprehensive unit tests.

• The code should be implemented using Python3 only.

 

What To Submit

 

• All source code

• Deployment guide

• Postman collection containing sample calls for all endpoints (success/failure)

• Verification guide - how to set up the environment, start the api, and verification screenshots

• Unit tests coverage report

 

Scoring Methodology

 

Contest Specification Requirements(60% weightage)

 

• Have all major specification requirements been met?

Score: 0-9

Major requirements are:

• all endpoints are implemented correctly and return the correct data and codes, etc,

• all calculations are working correctly, and optimization finds good mineral densities (compared to Excel and other submissions)

• Unit tests - minimum coverage 80%

        Have all major specification requirements been met?

Score: 0-3

Minor requirements are:

 

• Postman collection

• Swagger annotations

 

Best Practices & Comments(30% weightage)

 

• Does the submission follow standard best practices?

Score: 0-3

This section includes the PEP-8 code style, linter, patterns usage, code comments

Please make sure your code is well-documented. Use the following style guides Google Python Style Guide, Python Style Guide, and Docstring Conventions. Code linter is required. Please make sure it is well-engineered but not over-engineered (YAGNI and KISS) solution. We're looking for well-structured and tested code. Well-structured code follows good design principles like the SOLID principles and well-tested code has comprehensive unit tests.

 

Deployment and verification guide (10% weightage)

 

• Does the deployment guide contain everything needed to successfully configure and deploy the api?

Score: 0-3

Final Submission Guidelines

• All source code

• Deployment guide

• Postman collection containing sample calls for all endpoints (success/failure)

• Verification guide - how to set up the environment, start the api, and verification screenshots

• Unit tests coverage report