Challenge Overview
Topcoder message service is a microservice that handles all discussions for Topcoder Connect. We have recently migrated away from using Discourse for storing messages and into our own Postgres database.
In this challenge we want to index all threads/posts into elasticsearch for faster retrieval. The goal is to have just one index for topics that also contains all posts. Here are the requirements:
- Index should be updated every time a topic/post is created or updated (ie PUT/POST /topics or /posts).
- Topics list/get endpoints will be updated to fetch data from elasticsearch - make sure to keep all the permission checks before actually fetching the data - don't introduce any regressions in this regard
-
Two new endpoints should be added to force fetching topic(s) from database instead of elasticsearch (so the two new endpoints will actually be doing what get/list topics endpoints are doing now). Endpoint paths will be /v5/topics/list/db and /v5/topics/{topicId}/read/db
- A custom endpoint should be implemented that will re-index existing topics/posts. It should take startId and endId ad parameters, iterate over all records with ids in the specified range currently in the database and add them to ES index. It should be a asynchronous process, ie, it should return success code (200) once the immediately after making call to elastic search. Only admin users should be able to access this endpoint.
- The microservice is already using several AWS services, so the AWS credential configurations are already present - only elasticsearch configuration should be added. In local deployment, we should use a local instance of Elasticsearch, so add ES service to local/docker-compose.yml . All new or updated endpoints/services must contain proper logging for both success scenario(debug level) and error scenario (error level) - error messages have to contain details on why the operation failed and any relevant identifiers like topic id, project id, etc.
- Any failed indexing operations should be logged to a DynamoDB table (we're already using dynamoDB in the service, just create a new table for logging indexing errors). Error log should contain topic id, index body, timestamp and error message.
- Unit tests for new/updated endpoints are required.
Base code is available here - use dev branch.
Final Submission Guidelines
Submit the updated code
Submit updated deployment/verification guide
Submit updated Postman collection for verification