Challenge Overview
For this challenge, we'd like to create a migration tool that helps us to migrate all data that is currently in our DynamoDB instances to Postgres.
The tool should be general for migration from one DynamoDB table to Postgres table.
This tool will be initially used to migration the Submission table in DynamoDB, following is an example document (see forum)
You can setup docker environment for ease of testing. like postgres and dynamoDB local
2. Configuration and Setup Guide to use the tool
3. the verification steps to making sure the migration is working properly.
The tool should be general for migration from one DynamoDB table to Postgres table.
What the tool should do is, given an input DynamoDB url, an output Postgres connection information, and a mapping file (preferably json format), insert rows in Postgres based on the data from DynamoDB.
A few things to note:
1. The field name in the source table could be different than the field in the target table
2. The data type in the source table may be different than the data type in the target table. For instance, the system should be able to convert from text to integer, from boolean to integer, etc
3. The source and target tables may have different structures. If there are additional fields in the target table default values may be supplied. If fields are missing in the target table, those may be ignored;
4. If error happen, allow the migration to continue, and generate a detailed error report at the end. Please provide a way that the migration tool can be retried to recover from the error documents.
1. The field name in the source table could be different than the field in the target table
2. The data type in the source table may be different than the data type in the target table. For instance, the system should be able to convert from text to integer, from boolean to integer, etc
3. The source and target tables may have different structures. If there are additional fields in the target table default values may be supplied. If fields are missing in the target table, those may be ignored;
4. If error happen, allow the migration to continue, and generate a detailed error report at the end. Please provide a way that the migration tool can be retried to recover from the error documents.
5. For the mapping file, please create a general guide can everyone can be easily followed to create new mapping files.
6. All fields from DynamoDB should map 1 to 1 to Postgres, for id, which should be a auto generated long in Postgres, instead of a String UUID. for fields with json should be a json field in Postgres, for time related fields, we will use datetime field type.
7. If there are already tools to perform such migration, please describe them clearly and demonstrate how they address our requirements.
6. All fields from DynamoDB should map 1 to 1 to Postgres, for id, which should be a auto generated long in Postgres, instead of a String UUID. for fields with json should be a json field in Postgres, for time related fields, we will use datetime field type.
7. If there are already tools to perform such migration, please describe them clearly and demonstrate how they address our requirements.
8. Our preference in technology to migrate the data is to use Logstash, but not limited.
About Testing
About Testing
This tool will be initially used to migration the Submission table in DynamoDB, following is an example document (see forum)
You can setup docker environment for ease of testing. like postgres and dynamoDB local
Final Submission Guidelines
1. Complete Source Code for Migration Tool2. Configuration and Setup Guide to use the tool
3. the verification steps to making sure the migration is working properly.