Topcoder Project Service - Improve local setup

Create a docker-compose file for easy running of Topcoder Connect with Topcoder Project Service locally.

Project Background

Topcode Connect is a client-facing application of Topcoder. Topcoder Connect uses several microservices like Topcoder Project Service and Topcoder Notifications Service. And these microservices by themselves depend on other services. As the quantity of microservices grows it becomes harder and time-consuming to set up all the services locally during development. The main aim of the current challenge is to make the local setup of Topcoder Connect with Topcoder Project Service simple and avoid setup issues. 

On the forum, there would be a guide with 6 steps on how to setup these services manually. We should automate this process to a simpler process, so steps 2, 3, 4 are performed automatically and only 3 steps left:

1. Run one docker-compose with all dependent microservices.

2. Run Project Service.

3. Run Connect.

Technology Stack

• Docker

• Bash

• Apache Kafka

Code access

- Connect App repo https://github.com/appirio-tech/connect-app v5-upgrade branch, commit d0de553ffd20c6c02673abab106fa8abc7c3d9fd or later.

- Project Service repo https://github.com/topcoder-platform/tc-project-service v5-upgrade branch, commit 17ca11c7e9aff4b86ac25cc4b6b7f5b91710c622 or later.

- Demo users are provided on the forum.

- Guide and config to deploy Project Service locally are provided on the forum.

Individual requirements

We already have a docker-compose file which setups some services which are used by Projects Service only. We would like to keep it as it is in case we need to run the Project Service individually without other services. 

Create another docker-compose file inside “local/full” folder which would setup all the required services listed below. It’s preferable that the new docker-compose file “extends” the old one, instead of repeating the services from existent docker-compose.

The list of services which the full docker-compose should run:

• all the services inside existent docker-compose (preferable by reusing it somehow)

• tc-bus-api branch “dev” (reuse Dockerfile if possible)

• project-processor-es branch “develop” (reuse Dockerfile if possible)

• tc-notifications branch “v5-upgrade” (reuse Dockerfile if possible)

• Apache Kafka with Zookeeper
  - example docker-compose is provided on the forum but you may use another approach
  - add commands to 1) add topics, 2) list added topics, 3) watch topics and 4) send a message to a topic to the README (an example is also provided on the forum)
  - we should have a config file where we can list Kafka topics which would be created automatically when we run docker-compose. For now, all the topics inside BUS_API_EVENT and CONNECT_NOTIFICATION_EVENT should be put to the config file and automatically created.

• any other services which our Topcoder microservices require, like additional databases and so on

Notes

There are few things that we have to take into account when converting the manual local setup process into the docker-compose file.

1. To run Topcoder Services inside docker-compose we should somehow pass to them some config variables, especially  AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_URL, AUTH0_AUDIENCE, AUTH0_PROXY_SERVER_URL which cannot be included inside the repository. It’s preferable, that “docker-compose up” command gets these values automatically from the environment variables. So we can define them any way we want, for example by “export AUTH0_CLIENT_ID=...”.

2. To bypass token validation locally, during manual setup we have to update ”/node_modules/tc-core-library-js/lib/auth/verifier.js” manually and add this to line 23: “callback(undefined, decodedToken.payload); return;” in “tc-bus-api”, “tc-notifications” services.
   We should somehow do it automatically when we run them using docker-compose. Perhaps it could be a bash script which would be run inside docker-compose for these services before starting them.

Any additional suggestions on how we can improve the local setup process are highly welcome. You may share them on the forum or privately using the “Contact Manager” link in OnlineReview.

General requirements

• DON’T include the values for environment variables AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_URL, AUTH0_AUDIENCE, AUTH0_PROXY_SERVER_URL inside the code or config files which would be stored inside the repository. These variables should be always configurable manually by setting environment variables.

• The setup process must work on macOS and Linux operating systems. Supporting Windows would be nice but is not required.

• As now all these services would be run using a single docker-compose we need a command to see a log and error output of each of these services individually. Add this command(s) to the README.

• Update Project Service README as per the new local setup process. 

Verification

The local setup guide provided on the forum contains sections with a description of how to populate demo data and how to validate that local deployment works good. If during development you found any issues with provided guides or it doesn’t verify everything, please provide an updated verification guide that could be used during the review.

Final Submission Guidelines

• Git patches for all repositories with changes.

• A brief guide that would describe how the solution works.

• Updated verification guide if necessary.

• The winner would be required to raise a PR to each changed repository.