PoC - Salesforce Orgs Data Mover Tool Part 1

Project Overview

Our Client needs a custom Salesforce data migration tool, the final product will be a cloud NodeJS/Angular.JS/BootstrapUI application that will utilize Force.com Bulk and Metadata APIs to alleviate the difficulty of working on multiple Salesforce.com organizations and the need to repeatedly, manually move data for various reasons. We want to build an automated process to address this need.

Client’s current process consists of:

1. identifying the data they need in the target development org, using data loader to extract the data from production to some staging location

2. transforming, obfuscating, or omitting data that can't be seen by contractors or third party developers

3. loading data into a target development org.

Note that this is manual and very time intensive. Also, there are various custom steps that must be done every time.

Challenge Task Overview

This challenge is the first in a series of challenges to build the Data Mover tool, in this challenge we are addressing a single use case requirement:

• Moving all data from one Salesforce organization to another organization.

Functionality Requirements

You will address the following in this challenge:

1. Create single Frontend page - using Bootstrap for UI, and angularjs as js framework - that communicate with a web Express.JS application.

2. Use Postgres database to store task and job information for tracking purpose.

3. You will use jsForce module to communicate with Salesforce API.

4. The a single page should follow following structure (following points explain the page functionality):
   

5. Obtain source/target salesforce information via OAuth 2.0 :

   1. Login buttons will be used to navigate user to Salesforce OAuth 2.0 authorization code flow to obtain User information to be used in jsForce module to initiate Connection.

      1. Use https://jsforce.github.io/document/#oauth2

      2. On successful execution, store returned userInfo and access/refresh Tokens data in database for both organizations, and render it in the panel.

6. Metadata Panel:

   1. This panel will list the objects to be moved between organiziations

   2. Initiate Connection object using source Salesforce organization userInfo data obtained in step 5

      1. Use this method https://jsforce.github.io/document/#access-token-with-refresh-token

   3. Use describeGlobal function to obtain list of objects, and populate the “Avaialble Objects” pick list.

      1. Refer to this https://jsforce.github.io/jsforce/doc/Connection.html#describeGlobal

      2. In this challenge the scope should be limited to standard objects (lookups) that has no parent-child relationships. It is preferred to add filtering to the list that only display these objects. Taking into account that this filtering will be removed in next challenge so make sure your implementation should not require lots of efforts to remove this filtering.

   4. User is expected to add objects from pick list to migrate it’s data to the table as shown in the page.

   5. “Save” button should be disabled if the table is empty.

   6. Clicking “Save” will store the selected objects in database.

   7. Action column should display ‘remove’ button for each added object.

      1. Clicking ‘remove’ will remove the object from table

7. Lst panel displays the Task status and related information.

   1. When clicking “Start task” button at top of the page it will start the data migration task.

   2. Refer to “Job Processing Flow” point below for the logic to be executed when starting the job.

   3. “Migration Jobs” table lists all created jobs in this task.

   4. This panel content should be refreshed on a regular basis via ajax. By default every 5 minutes.

8. This how page looks like when the migration task is started.
   

   1. User should not be able to add more objects to the task. disable ‘add’ button. We will re-enable this feature in future.

      1. removing an object from “objects to migrate” job won’t affect the “running” task

   2. Clicking ‘abort’ button in the table, will abort corresponding job.

      1. Refer to this function.

   3. User can abort the whole task by clicking ‘Abort Task’ button at top.

      1. Aborting a task should abort all currently running jobs

         1. The logic should iterate through each job in the task and abort it.

         2. update all jobs in the Migration jobs table to ‘aported’.

         3. update task status in backend to ‘aborted’.

         4. Display boostrap info popup to tell user if operation was successful or not.

         5. User will not be able to do anything further for this task. User should refresh the page to start new task.

   4. ‘view’ button in jobs table is out of scope of this challenge.

9. Job Processing Flow:

   1. When user click on “start task” button it will invoke a job migration process.

   2. The task should run asynchronously, meaning that when user click the button, then ajax call will be made to backend to delegate the work to a async service method, and response of the ajax will be either success or failure of the delegate.

   3. As mentioned earlier in the specs, the code will only address simple/lookup standard objects. So the flow to export/import should be straightforward.

   4. The flow described as follow :

   5. Use metadata#describe method to list Objects of both source/target Salesforce organization (if needed to prepare the job)

   6. To export data from source Salesforce organization you will use Bulk#query operation with SOQL (Salesforce Object Query Language). This method will be executed for each Object selected to be exported. The SOQL to be used would be “SELECT * FROM <Object_Name>”

      1. Create a new model in database, use metadata of the object (obtained in step b) and define a postgres model/table. The model name should be unique, so you need to append a random string to the name.

      2. Store the model name in task table

         1. Rename the primary key (salesforce id) by simply appending _external_id.

   7. To import data to target salesforce organization you need to implement the following:

      1. Create new bulk object a job for the target Salesforce org

      2. For each object to be moved :

         1. Create new Job

         2. Persist created JobInfo ID in db for tracking purpose

         3. Use load operation to start a bulk/batch. load of the object

            1. read records from database.

            2. User the primary id of the object as to set extIdField

         4. Persist created batch info in db for tracking purpose.

         5. We will use upsert operation always.

      3. When all jobs are completed, delete the model’s tables from database. Then update task status to ‘completed’

   8. In this version we have following requirements about data being migrated :

      1. We will migrate all data

      2. We will only work with standard (simple/lookup) objects that has no parent-child (foreign keys) relationships.

      3. We will replace existing data, basically the Bulk operation should use ‘upsert’ operation for all submitted batches

      4. It might now be documented in jsForce about the limitation of using Bulk API, so you need to look at this page and take limits into consideration. Specifically, we want to avoid locks.

10. All interaction between frontend page and backend will be via Ajax through Angularjs controller.

11. We will always have single task at a time, if user try to create new task (i.e. open the page in a new browser window to start creating new task), you can then add a middleware that check DB if there is an active task then return that task to the user. Or you can show error message that you cannot create a task while an active one is currently running.

    1. This is temp. solution, we will enable multiple tasks in future challenge.

12. The project will be hosted in heroku.

13. Provide a detailed documentation for how to setup and configure the application locally and on heroku.

14. You are expected to use environment variables to store sensitive information and environment-specific configurations.

Architecture Requirements

Here is a high level overview of the architecture :

• The architecture must be followed as is so we can extend and add more layers in future without much efforts.

• Create two controllers :

  • One that communicate with the frontend page data, call it “TaskController”

  • Another controller that deal with the “Salesforce OAuth” interaction/flow.

• Each service to be created in your solution should be a self-contained unit of functionality, we don’t accept a single service/helper class that encapsulate all the code, so take this into account. And to simplify the requirement we expect following services :

  • Service to process the ‘start task’ button action, call it ‘taskProcessor’ that will be called async, and will encapsulate all the logic to interact with the other services/helper classes to complete the task.

  • Service work as CRUD for Task table.

  • Service for exporting data.

  • Service for importing data.

The task metadata information we collect through the page should be stored in ‘Task’ model/table, you are expected to create a service with CRUD operations for this model, it should have following fields (Post in challenge forums if you want to make changes to the model structure) :

• ID - primary id, you can use auto-increment integer, or UUID

• TargetOrgInfo - json, store userInfo object of target salesforce org

• SourceOrgInfo - json, store userInfo object of source salesforce org

• Status - String, an enum should be one of these values ‘Created, Pending, Export Running, Import Running, Aborted, Failed, Completed’

• Message - String, used to store extra information about execution

• Metadata - array of json objects, each item stores information related to selected objects to be migrated, and it’s related job/batches. fields are :

  • Object : name of the object

  • Operation : it will be one of the values ‘insert', 'update', 'upsert', 'delete', or 'hardDelete'

  • JobInfo : json object, store the JobInfo returned when creating a Bulk Job

  • batchInfo : array of json objects, each item stores the batches submitted the job via Bulk Job.

Folder Structure and Configuration

Follow this folder structure :

• config/
  • config.js
• app.js
• controllers/
• models/
• services/
• helpers/
• README.md
• env-sample (don't include .env in your submission)
• .. other files if needed

For configuration, we expect routes and other sensitive config will be configured in config/config.js, we prefer if you use node-config module for that. but We will leave it up to you to use the proper approach.

References

• You will need more than one Salesforce developer account, signup here if you don't have one already.
• Salesforce API documentation

 

Final Submission Guidelines

Submission Deliverables

Below is an overview of the deliverables:

• A fully implemented application with all the functionality defined by the requirements above.

• Sample data.

• A complete and detailed readme document explaining how to deploy the application including configuration information (locally and heroku).

 

Final Submission

For each member, the final submission should be uploaded via the challenge detail page on topcoder.com.