Challenge Overview
Challenge Overview
In this challenge, we are going to update sagemaker processor to test marathon match submissions with Sagemaker.
Project Background
Topcoder currently has a processor built for scoring marathon matches. Now we would like to adopt SageMaker to replace current solution.
Technology Stack
- Node 10
- AWS Sagemaker
- Nodejs AWS SDK
- Docker
Individual requirements
The current sagemaker processor codebase is shared in the forum. The sagemaker processor reads message from kafka, download submission, build docker image, push docker image into ECR and train in Sagemaker. This process is for stage "train".
Now we need to create another stage "test". In "test" stage, the sagemaker processor reads message from kafka, create model and deploy it.
The sagemaker knows its stage by reading configuration.
(major) Update config
- Add STAGE: it could be either "train" or "test"
- rename INPUT_FOLDER_PREFIX to TRAIN_INPUT_FOLDER_PREFIX
- Add TEST_INPUT_FOLDER_PREFIX: test data folder in test stage
Add more config if needed and ask in the forum if you are not sure. Please keep the code clean and only add useful files.
(major) Update ProcessorService.js
- Add checking for STAGE, the existing process is for STAGE == "train". The following part is for STAGE == "test"
- Create a deployable model (https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-deploy-model.html#ex1-deploy-model-boto) This is example of Python code, we will use Nodejs in this challenge. Values for job name, container and model data could be created based on code in STAGE train, for example job name is "mm-processor-${submissionId}". Model name is training_job_name + '-mod'.
- Deploy the model with transform (https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-batch-transform.html#ex1-batch-transform-api-low-level). Batch input folder is in S3 bucket {INPUT_S3_BUCKET + TEST_INPUT_FOLDER_PREFIX}. You can assume there is only one input file in the folder. Output model is in S3 bucket {OUTPUT_S3_BUCKET + "test-" + submissionId}. Wait until its finish.
Important Notes
- Use async await pattern.
- Use standard as your linter. Ensure there are no lint errors in your submission.
- Typescript is not allowed. No build or compilation of code is expected.
- No tests are needed.
Final Submission Guidelines
- code for updated sagemaker processor
- README and VERIFICATION guides should be updated