Project UBahn | Integrate Elasticsearch in API

Context

Project Context

Our client wants to create an employee management system to determine employees that are no longer working on active projects and to understand their qualifications and expertise for suitability in other projects

 

Challenge Context

Update existing API to integrate with Elasticsearch for all GET requests

 

Expected Outcome

The current API interfaces with Amazon QLDB only. After this contest, we expect the API to also interface with Elasticsearch for all GET requests.

 

Challenge Details

Technology Stack

• Nodejs version 12

• Elasticsearch version 6.x

• Amazon QLDB

 

Development Assets

• You can find the API code base here. Ensure that you are using the “develop” branch. You can also view the swagger documentation for the API at the “/docs” route

• You can find the Elasticsearch processor code base here. Ensure that you are using the “develop” branch. (Read the individual requirements below for when this is used).

 

Individual requirement

Requirement 1: Integrate Elasticsearch with API (Major Requirement)

• Currently, the API interfaces only with Amazon QLDB.

• For all GET requests, we would like the API to instead use Elasticsearch, before it uses QLDB.

• This means that when a GET request for any of the endpoints arrives (which earlier would require the api to query QLDB) will now instead query Elasticsearch. You will support existing filters (as defined in the swagger specification for each of the GET endpoints) too.

• If Elasticsearch does not return any data, or if elasticsearch reports an error (log the error), proceed to query QLDB, which is the current behaviour. You are not expected to make any updates to the current behaviour with QLDB. (This is to guard against any issues that you may discover with QLDB based queries - report them in the forum if you can but do not fix them)

• While the query / filter parameters will be the same as the ones that we are passing currently, in addition to these, users will also be able to pass pagination parameters. Currently, the api does not support pagination (because QLDB does not support pagination) but after integrating with elasticsearch, it will.

• We understand that, when we fall back to QLDB, the pagination parameters will not be taken into consideration.

 

Requirement 2: Enriched users endpoint (Major Requirement)

• For the GET /users and the GET /users/{userId} endpoints, you need to support an additional query parameter named “enriched”.

• When this value is true, the query to elasticsearch should be such that all data related to users is also returned, along with the users data

• The related models that need to be returned are:

  • User skills

  • Skills

  • Skill providers

  • User roles

  • Roles

  • External profiles

  • Organization

  • Achievements

  • Achievement providers

  • User attributes

  • Attributes

  • Attribute groups

• We can filter the users using the handle. Additionally, for each of the above models, users can also use nested filters. The list of nested filters are:

  • User skills

    • userSkill.skillId

    • userSkill.metricValue

    • userSkill.certifierId

    • userSkill.certifiedDate

  • Skills

    • skill.skillProviderId

    • skill.name

    • skill.externalId

    • skill.uri

  • Skill providers

    • skillProvider.name

  • User roles

    • userRole.roleId

  • Roles

    • role.name

  • External profiles

    • externalProfile.organizationId

    • externalProfile.uri

  • Organization

    • organization.name

  • Achievements

    • achievement.achievementProviderId

    • achievement.name

    • achievement.uri

    • achievement.certifierId

    • achievement.certifiedDate

  • Achievement providers

    • achievementProvider.name

  • User attribute

    • userAttribute.attributeId

    • userAttribute.value

  • Attributes

    • attribute.attributeGroupId

    • attribute.name

  • Attribute groups

    • attributeGroup.name

    • attributeGroup.organizationId

• Thus, if there is a request such as GET /users?enriched=true&attribute.name=location&userAttribute.value=London&skill.name=Java&userSkill.metricValue=90, the response should contain users, along with all the models mentioned above and it should only contain those users who have an attribute named Location with value London, who have a skill named Java with metric value of 90

• For GET /users/{userId}?enriched=true, there will be no filters. You will only return the enriched data for the requested user.

• The search / query will be case insensitive

• If the user does not pass “enriched=true”, then none of the query filters mentioned above should work. It would be treated like the existing GET /users endpoint with the handle, groupId and role filters.

 

Requirement 3: Pass data updates to Bus API (Major Requirement)

• For all endpoints that create, update or delete data, after the necessary action is carried out and before the api responds back, the api needs to pass the data to the Bus API.

• You don’t have to know the details about the Bus API (you can use this website or equivalent for your development) except understand that the data that you pass to the bus api is then passed as a message to Kafka.

• Another app (not in scope for this contest) listens on Kafka for incoming messages and processes it. The processed data is then stored in Elasticsearch.

• It is this data that Requirements 1 and 2 pass back to the user for an incoming GET request.

• Refer to submission api or lookup api. You will be implementing it similarly.

• Note that, in the data that is passed to the bus api, we will add a property named “resource”. This will be the name of the model associated with the data.

• Additionally note that, depending on the action (create, update or delete), the kafka topic to which the data is being sent to also is different. For our use case, the topics will be:

  • u-bahn.notification.create

  • u-bahn.notification.update

  • u-bahn.notification.delete

• The end to end workflow will thus be as follows:

  • Endpoints that create or update (including deleting) data will work only with QLDB

  • The API will post the data to Topcoder’s Bus API. This API can be treated as a black box. You are not expected to understand how Bus API works. Understand only that when you pass data to this api, it will send the data as a message in Kafka

  • We have a nodejs based processor, which we will call as the UBahn Elasticsearch Processor. This processor listens for messages in Kafka - same as the one to which the Bus API sends messages to. It will process the messages that arrive, and store the data in Elasticsearch

  • When our UBahn API (the api that you will be working on as part of this contest) receives a GET request, the api will first query Elasticsearch. You may need to convert the query to a format that elasticsearch will understand.

  • If Elasticsearch returns with no data, then we will query Amazon QLDB, as we are currently doing. We will basically fall back to QLDB only if Elasticsearch does not return any data or fails (example, if the Elasticsearch instance is down).

 

We currently have a similar setup for Topcoder’s other APIs. Examples:

• The submission api and the submission elasticsearch processor. In this case the API interfaces with Dynamodb and posts the data to Kafka, which is then processed by the processor and the data is stored / updated in Elasticsearch. The API itself uses Elasticsearch for read operations (before falling back on Dynamodb).

• The lookups api and the lookup api processor. Similar to the submission api and submission processor. Different data.

 

Requirement 4: Call Groups API (Minor Requirement)

• For Requirement 2 - Enriched users endpoints, in addition to the models that exist in the API, you also need to retrieve the groups associated with the users and return in the response, for each user.

• For this, you will call an external api named the groups api.

• You can find the specification for this api here.

• You need to call GET /groups using the userId under the memberId filter to get groups that the user belongs to.

• If no groups are returned for the user, you will return an empty array for the groups property.

 

Important Notes

• Use standard for your lint tool

• Use the async await pattern

• Tests are not needed.

• Since Elasticsearch is now being used, pagination will be supported for all GET requests that return an array of data.

• In case Elasticsearch reports any errors during the query, log the error and proceed to query QLDB. We understand that in this scenario, pagination cannot be supported and that is fine.

• You need to update swagger specification (for example, for pagination parameters) This  is a Minor Requirement

• For update operations, ensure that you pass the entire data for that model and not just the attribute that is to be updated. Example - when updating the skill model, the request may contain only the updated skill name. However, when passing the data to the bus api, you would need to pass other properties of the model too, such as skillProviderId, externalId and uri. The processor that listens to kafka messages will update the data in  Elasticsearch based on the data that is passed to it and thus, you need to pass all the properties of the model.

 

Deployment guide and validation document

Update the deployment guide as needed.

 

Scorecard Aid

Judging Criteria

• Against each requirement, we have identified Major and minor requirements.

• Reviewers will use the respective sections in the scorecard, depending on the type of requirement.

Final Submission Guidelines

You will only be updating the API code base (the one located here). You need to submit a zip file containing the git patch of the changes made, which will be applied to the “develop” branch. This is the recommended approach. However, if you face issues generating the patch, you can upload the entire code base too in the zip file.