Challenge Overview
Previously, we have created a separate loader repository for loading data from TC to DW or Redshift. You can check https://github.com/appirio-tech/tcs-loader (Use load_tcs_to_redshift branch).
This is following the old loader logic, like
1. a Loader Utitlity class (com.topcoder.shared.util.dwload.TCLoadUtility) which will read XML configuration to start the specific loader.
2. an XML configuration file which will specify the DB related information and the real class doing the loader.
3. the real class that extends from com.topcoder.shared.util.dwload.TCLoad
But we saw some limitation about this approach,
1. if you check com.topcoder.utilities.dwload.TCLoadTCSRedshift and com.topcoder.utilities.dwload.TCLoadTCS, you can see they are big classes which actually doing separate data loading in each doXXXXXX methods.
2. a failure inside any doXXXX methods may throw exception, and will stop the loader, but it seems fine to keep on going to load data for other tables.
3. If we want to disable loading for one table, we need to update the class.
4. If we want to support new tables, we need to update the existing class.
So we'd like to revise the architecture, following are the general guide, but you are expecting to improve as you think it is better.
- A abstract class or interface which defines the way that every specific loader should be following.
- Each separated loading should be inside separated classes implementation.
- An xml configuration file will defines the class or classes to be loaded, an error in one loader class should not break the whole process.
- An general loader uitlity will read the xml confiuration file and start the loader classes.
With the above approach, we can easily add and test new loaders, we can easily disable existing loaders by simply changing the XML configuration file.
both com.topcoder.utilities.dwload.TCLoadTCSRedshift and com.topcoder.utilities.dwload.TCLoadTCS need to be refactored and splitted by following the new architecture.
About Testing
- For Informix, you can use docker to setup local env, see https://github.com/appirio-tech/tcs-loader/blob/master/README.md
- For Redshift, you need to create your own in AWS, there are free tier for new users.
Winner Responsiblity
Winner will be responsible to create PR in github for merge.
This is following the old loader logic, like
1. a Loader Utitlity class (com.topcoder.shared.util.dwload.TCLoadUtility) which will read XML configuration to start the specific loader.
2. an XML configuration file which will specify the DB related information and the real class doing the loader.
3. the real class that extends from com.topcoder.shared.util.dwload.TCLoad
But we saw some limitation about this approach,
1. if you check com.topcoder.utilities.dwload.TCLoadTCSRedshift and com.topcoder.utilities.dwload.TCLoadTCS, you can see they are big classes which actually doing separate data loading in each doXXXXXX methods.
2. a failure inside any doXXXX methods may throw exception, and will stop the loader, but it seems fine to keep on going to load data for other tables.
3. If we want to disable loading for one table, we need to update the class.
4. If we want to support new tables, we need to update the existing class.
So we'd like to revise the architecture, following are the general guide, but you are expecting to improve as you think it is better.
- A abstract class or interface which defines the way that every specific loader should be following.
- Each separated loading should be inside separated classes implementation.
- An xml configuration file will defines the class or classes to be loaded, an error in one loader class should not break the whole process.
- An general loader uitlity will read the xml confiuration file and start the loader classes.
With the above approach, we can easily add and test new loaders, we can easily disable existing loaders by simply changing the XML configuration file.
both com.topcoder.utilities.dwload.TCLoadTCSRedshift and com.topcoder.utilities.dwload.TCLoadTCS need to be refactored and splitted by following the new architecture.
About Testing
- For Informix, you can use docker to setup local env, see https://github.com/appirio-tech/tcs-loader/blob/master/README.md
- For Redshift, you need to create your own in AWS, there are free tier for new users.
Final Submission Guidelines
- Code Changes for the new apprachWinner Responsiblity
Winner will be responsible to create PR in github for merge.