Challenge Overview
For All Challenges Index, we used a single sql query to retrieve the data from informix database, and use aggregate filter to aggregate data for some array fields.
For this challenge, we'd like to try out splitting the sql query, and each splitted sql query will be used to populate different fields, so we can easily revise and improve the index and sync performance.
This assumes the easticsearch output will do partial update based on document id, please evaluate this approach and provide a solution.
- Following fields should be populated separated.
- challenges type
1. groupIds
2. userIds
3. hasUserSubmittedForReview
- srms type
1. userIds
2. isRatedForSRM
- mmatches type
1. useIds
2. isRatedForMM
- logstash conf files (conf/all-challenges*) should be updated.
- Please test against the standard elasticsearch output and aws_es output
How to Test
1. Make the changes and build the docker image locally like appiriodevops/tc-logstash-conf-public:dev
you can use different tag, but be sure to update the docker-compose.file below
2. Following https://github.com/appirio-tech/tc-common-tutorials/blob/master/docker/direct-app/ to run the services locally
2.1 build tc-direct
2.2 run services like `docker-compose up all-challenges-es-initial-load tc-direct-with-volume run-online-review`
3. Use TC direct to create active challenges to see if the data is properly populated in elasticsearch
- Verifiation Steps
For this challenge, we'd like to try out splitting the sql query, and each splitted sql query will be used to populate different fields, so we can easily revise and improve the index and sync performance.
This assumes the easticsearch output will do partial update based on document id, please evaluate this approach and provide a solution.
- Following fields should be populated separated.
- challenges type
1. groupIds
2. userIds
3. hasUserSubmittedForReview
- srms type
1. userIds
2. isRatedForSRM
- mmatches type
1. useIds
2. isRatedForMM
- logstash conf files (conf/all-challenges*) should be updated.
- Please test against the standard elasticsearch output and aws_es output
How to Test
1. Make the changes and build the docker image locally like appiriodevops/tc-logstash-conf-public:dev
you can use different tag, but be sure to update the docker-compose.file below
2. Following https://github.com/appirio-tech/tc-common-tutorials/blob/master/docker/direct-app/ to run the services locally
2.1 build tc-direct
2.2 run services like `docker-compose up all-challenges-es-initial-load tc-direct-with-volume run-online-review`
3. Use TC direct to create active challenges to see if the data is properly populated in elasticsearch
Final Submission Guidelines
- Changes To Logstash Conf Files- Verifiation Steps