Omega Microservices / API Challenge - Part 1

Welcome to the Omega microservices challenge. This challenge is first in a series of challenges where we will build Application for a world leading point-of-sale manufacturer.

 

Project Overview

The goal of this app is to allow customers of the POS equipment to manage and provision their fleet of hardware. We will begin this series by building out our apps API - A series of microservices built in Golang connecting to Cassandra database. We will be providing Dockerfiles so you can start with a turn-key Cassandra 3 node cluster. We will be creating 13 microservices in total and already have two completed so you will have some good examples.

 

 

Challenge Overview

Simply put, this challenge requires you to create an API in Golang with Cassandra as the database. The API contains 13 "microservices", two of which we have already implemented to give you an example. 

• Since our services require a minimum of three Cassandra nodes we have provided a Dockerfile to build the image and a docker-compose to run the containers and link them. You are however free to use your own Cassandra database.
• Of the 13 microservices, we have already implemeted two. For these two resources, we have created a utility that will create the schema in Cassandra and load a set of sample data. The tool is called "dgestool" and as part of these challenges you will create a cql file and a csv that corresponds to your new service.
• Enhance the dgestool - Part of the challenge will require adding support to dgestool for detecting and populating the new microservice’s cassandra keyspace table(s).
• Create the microservice - using the two provided microservices as an example you will create another microservice based on the supplied dataguard.yaml swagger file.
• Swagger file - We have provided a swagger yaml file that defines all the services. You may use swagger UI or the online swagger tool to inspect the yaml file if you like but everything you need should be included in the yaml file.
• Sample data file and schema - While you are creating the services you should periodically create the cql file and a sample csv file and use the dgestool to import it.
• Improvements to the process - Unlike a classic code challenge this series requires more setup details (Cassandra) so we feel that docker is a good approach to simplify these steps but we are always looking for ways to improve this so we welcome your feedback.

 

Challenge Application Setup Details

We wanted to make it easier for you to focus on building the microservice. Hence, we have provided (as part of the source code) a Docker image which contains Cassandra - our database for this application. Thus, you don't have to install it, configure it and go through all the time consuming actions - we have already done that for you through a Docker Image. This is optional. If you wish to use your own Cassandra database that you wish to set up on your own / have already set up, feel free to use it.

 

Follow the instruction below to set the application up.

Cassandra Docker Cluster: Start by downloading the source code which will be in the forum as a zip file or (optionally) upon request we will add you to the private gitlab repo. Next you will need to install Docker and docker-compose (formerly known as fig). If you are not working natively on Linux you may need to increase the boot2docker ram to 4G in order to run the containers. Once you have docker and docker-compose installed you should be able to run "$ docker-compose up" from the cassandra-docker directory. If everything is working you should see three containers running with "$ docker ps" in another terminal.

Load data: Next you need to build the dgestool and sample services for importing the schema and data and testing connectivity. To do this set your GOPATH to services/src/dges. You will need to have mercurial installed so you can get the following dependencies.

• RUN go get code.google.com/p/snappy-go/snappy
• RUN go get speter.net/go/exp/math/dec/inf
• RUN go get github.com/gocql/gocql
• Build dgestool and the reference services.

Create the microservice: Using the existing two microservices (customers and enterprisegroups) and the dataguard.yaml file you should be able to create a new microservice from the details in the yaml file. You may choose any one defined in the datagaurd.yaml file, however Locations and Users are the logical next step. If you choose to do more than 1 service a discretionary bonus of up to $1000 may be award.

A tl;dr version of the resources that will be built is as follows:

• Customers (already completed)
  • Enterprise Groups (already completed)
    • Locations
      • Endpoint Services 
      • Terminals 
        • Peripherals
          • Endpoints
            • Manifests
  • Statistics
  • Users / Groups / Roles (consider one service or propose multiple if it makes sense - please explain your decision here)
• Coming soon
  • Events
  • Alerts
  • Manifest Templates

Note: The indentation implies a resource dependency on the previous service. For example, enterprise groups belong to a specific customer and locations are part of an enterprise group.

Sample data file and schema: While you are developing the microservice you should also create the scema and supporting cql file and sample csv file which should contain at least 100 records. You should use the dgestool to import it and be able to test against it. Respect hierarchy and tie with ids from existing sample data. For example, reuse the customers service sample data id as well as existing enterprise groups ids. Keyspaces and table(s) design must maximize scalability at the expense of normalization every time.

 

Challenge Details

To give you a push in the right direction, we have already implemented two resources for the microservice - customer and enterprisegroups. The entire list of resources has been described in detail in the dataguard.yaml file provided along with the source code.

As part of this challenge, you need to 

1. Choose 1 of the several available microservices from the list defined in the dataguard.yaml file and implement it following the patterns from existing customers and enterprisegroups.
2. Update the dgestool to support detection, mapping & import of the new service’s sample data csv.
3. Provide a cql file and csv file (containing 100 records) to be used with the provided dgestool to import the schema and load the data.
4. Provide unit tests that exercise your solution. Use the Go test framework for all unit tests and provide coverage report.
5. Provide suggestion to simplify the environment setup or improve the code structure (Optional - discretionary bonus)
6. E2E tests for testing routes by hitting the REST API and validating the results. (Optional - discretionary bonus) 

 

Additional Notes:

I have provided a Dockerfile for a go-utility-client which you may use to see the sample services running. Look at the Dockerfile (Note this is a different Dockerfile from the cassandra image) in /services. The instructions are in the comments of the readme. In addition this, the Dockerfile has all the dependencies to run the microservices, but they should also build fine on a mac or windows. The use of this docker is completely optional. You should have no trouble developing the service on any os that supports GO. When you run the two services and want to test a GET be sure to set your Accepts and Content-Type headers to application/json otherwise it will return a 404.

In addition to this, certain recommendations that you can follow:

Cassandra recomedations

• Never use a secondary index – find a way not to need one.
• Use the right partition key for your table(s). Time series data for example should have a composite partition key with a time bucket for the partition key.
• Use the right clustering keys for your table(s).
• Watch out for unbounded row growth

Recommended Cassandra reads/references:

• Cassandra High Availability, by Robbie Stickland, Packt publishing
• Cassandra Data Modeling and Analysis, by C.Y. Kan, Packt publishing
• Cassandra Design Patterns, by Sanjay Sharma, Packt publishing

 

Finally as you can tell, the first time setup of these challenges is elaborate and requires knowledge in multiple areas.

We are looking for ways to reduce the barriers to entry. Bonus considerations will be made for submissions that provide a simpler setup to run the challenges.

 

Final Submission Guidelines

Submit your source code in zip file to TopCoder Online Review. Your submission must include the following:

• Detailed setup instructions
• Documentation of unit tests
• Unit test coverage report
• Updated docker image which includes your microservice (Optional)