Topcoder Project Service - Project Templates Refactoring

Refactor project templates by extracting form, pricing and phases data into 3 separate models.

Project Background

Topcoder Project Serice is a main backend service of the Topcode Connect. Topcode Connect is client facing application of Topcoder. Customers use Topcoder Connect to input requirements of their projects, then managers and copilots take it from there.

Currently, we are storing questions of form and price config in `scope` field of the project template entity which we want to break into two separate models. Questions related configuration would be refactored to a new model called `Form` and price/duration estimate related config would be moved to another new model called `PriceConfig`.

Further, we are storing a project plan config as `phases` JSON in project template which we would extract into another new model called `PlanConfig`. And after all these refactoring, we can update project templates to be a composite of all these models. This would allow us to use the same set of questions with multiple project templates and similarly, we can use the same price config for multiple project templates.

Technology Stack

• Node.js

• PostgreSQL

• ElastiSearch

Code access

The work for this challenge has to be done in one repository:
- Project Service repo https://github.com/topcoder-platform/tc-project-service dev branch
YAY: we improved the local deployment process of the Project Service and made README cleaner. Now the local deployment on the Linux/Mac OS systems should be flawless and straightforward. Please, enjoy. And if you have any issue during local deployment let us on the forum and we will try to improve it.

Individual requirements

The new models which we are going to introduce will have versions which in turn will have revisions.
Each object can have several versions. Each version of such an object can have several revisions.
- version - is something for people. When we want to make some new version of some form, we could manually click a button to create a new version. We will be aware of what exact version we are working at the moment and what version is being used inside templates.

- revision - is kind of history of changes or “Ctrl + Z” of changes we saved to the DB. Sometimes we accidentally save broken data into DB and finding issues could be quite hard in big JSONs. So every time we save a particular object instead of updating it we would create a copy with updated data and increased revision number. This would happen without user notice and we don’t need to know what is the number of revision we are seeing at the monent. Though a user could be able to press some button to come to the previous revision.

In this challenge, we will handle changes in Project Service, and later update the Connect App to work with the new model.

1. Create new models

• Table form together with a corresponding model Form
  - id: serial NOT NULL DEFAULT 1 PRIMARY KEY
  - key: character varying(45) NOT NULL
  - version bigint NOT NULL DEFAULT 1
  - revision bigint NOT NULL DEFAULT 1
  - scope: json NOT NULL
  - createdAt, updatedAt, deletedAt, createdBy, updatedBy, deletedBy
  - UNIQUE(key, version, revision)

• Table price_config together with a corresponding model PriceConfig
  - id: serial NOT NULL DEFAULT 1 PRIMARY KEY
  - key: character varying(45) NOT NULL
  - version bigint NOT NULL DEFAULT 1
  - revision bigint NOT NULL DEFAULT 1
  - config: json NOT NULL
  - createdAt, updatedAt, deletedAt, createdBy, updatedBy, deletedBy
  - UNIQUE(key, version, revision)

• Table plan_config together with a corresponding model PlanConfig
  - id: serial NOT NULL DEFAULT 1 PRIMARY KEY
  - key: character varying(45) NOT NULL
  - version bigint NOT NULL DEFAULT 1
  - revision bigint NOT NULL DEFAULT 1
  - phases: json NOT NULL
  - createdAt, updatedAt, deletedAt, createdBy, updatedBy, deletedBy
  - UNIQUE(key, version, revision)

• For each of these new models:
  Create CRUD endpoints for version:
  GET /projects/metadata/<modelname>/{key}/versions - lists all versions (permission logged in user)
  GET /projects/metadata/<modelname>/{key}/versions/{version} - get a particular version (permission  logged in user)
  GET /projects/metadata/<modelname>/{key} - get the latest version (permission  logged in user)
  - In all GET endpoints, we return the latest revision
  POST /projects/metadata/<modelname>/{key}/versions - create a new version (permission projectAdmin)
  - When we create a new version, the version should be increased by one. revision is set to 1 automatically.
  PATCH /projects/metadata/<modelname>/{key}/versions/{version}- update the specific version (permission projectAdmin)
  - When we update the specific version we should create a new object with revision increased by one instead of updating. The same time if we have more than 100 revisions, we should remove the old revisions. So at any moment we will have not more than 100 revisions. This limit should be configurable.
  DELETE /projects/metadata/<modelname>/{key}/versions/{version} - mark as deleted the specific version (permission projectAdmin)
  - When we delete a specific version, all the revisions of this version should be marked as deleted. NOTE: we use soft deleting.

• Create CRUD endpoints for revision:
  GET /projects/metadata/<modelname>/{key}/versions/{version}/revisions - lists all revisions of specific version (permission logged in user)
  GET /projects/metadata/<modelname>/{key}/versions/{version}/revisions/{revision} - get a particular revision of specific version (permission  logged in user)
  POST /projects/metadata/<modelname>/{key}/versions/{version}/revisions - create a new revision of specific version  (permission projectAdmin)
  - when we create revision using this endpoint we shouldn’t care about removing old ones as we did in PATH version endpoint.
  DELETE /projects/metadata/<modelname>/{key}/versions/{version}/revisions/{revision} - mark as deleted a particular  revisions of specific version (permission projectAdmin)
  - NOTE: we use soft deleting.

• Create a migration script for creating these tables in the DB inside /migrations folder

• Currently, we have /projects/metadata endpoint which returns various templates including projectTemplates. As now we moved out some data from projectTemplates we have to return it also. So /projects/metadata should additionally return new lists:
  - forms
  - priceConfigs
  - planConfigs
  By default, it should return only the latest versions for forms, priceConfigs and planConfigs.
  If we add an additional parameter includeAllRefered to the endpoint:
  /projects/metadata?includeAllRefered additionally to all the latest versions of forms, priceConfigs and planConfigs it should also return versions which are referred by any existent projectTemplate.
  In all the cases we return only the latest revisions of these objects.

2. Update ProjectTemplate model

We want to migrate to the way of storing projectTemplates gradually. So at the moment, we will let to store templates in the old way with scope and phase and a new way with form, priceConfig and planConfig. For this purpose we will do the next things:

• Make scope and phases properties optional (now they are NOT NULL).

• Add three more properties:
  planConfig: json (optional)
  priceConfig: json (optional)
  form: json (optional)

• Create a migration script for updating this table in the DB inside /migrations folder

3. Create a migration script

We want to have a migration script which admins could call for particular projectTemplate to migrate it from the old way of storing form, pricing, and phases to a new one. This should be done with a similar way to migrating V2 projects to V3 projects: https://github.com/topcoder-platform/tc-project-service/blob/dev/src/routes/projectUpgrade/create.js#L1.
So we should have an endpoint POST /v4/projects/metadata/projectTemplates/:templateId(\\d+)/upgrade which only admins can call with a particular projectTemplate id. And this upgrade endpoint should also allow specifying using existing Form/PriceConfig/PlanConfig and if user does not specify any one of them, it would extract the current content as a new corresponding entity.
The migration script would look like this:

• if Form is not specified: create a new Form and copy fields sections,  wizard,  preparedConditions from ProjectTemplate.scope to Form.scope

• set ProjectTemplate.form: { key: Form.key, version: Form.version }

• if PriceConfig is not specified: create a new PriceConfig and copy all the fields excpet of sections and  wizard from ProjectTemplate.scope to PriceConfig.config object (note we copy preparedConditions into both Form and PriceConfig)

• set ProjectTemplate.priceConfig: { key: PriceConfig.key, version: PriceConfig.version }

• if PlanConfig is not specified: create a new PlanConfig and copy ProjectTemplate.phases to  PlanConfig.phase

• set ProjectTemplate.planConfig: { key: PlanConfig.key, version: PlanConfig.version }

• set ProjectTemplate.scope to null

• set PlanConfig.phases to null

General requirements

• Implement unit test for all these new endpoints similar way as for existent endpoints

• Update postman collection to reflect all the changes

• Update swagger to reflect all the changes

• Lint should pass

• Existent unit tests should pass

Final Submission Guidelines

• Patch to the dev branch of Project Service (Please, avoid errors in the patch)

• Text file with the hash of the commit you use for the patch.