Challenge Overview
The Department of Veterans Affairs' (VA) National Cemetery Administration (NCA) seeks to create an interactive digital experience that enables virtual memorialization of the millions of people interred at VA national cemeteries. This online memorial space will allow visitors to honor, cherish, share, and pay their respects and permit researchers, amateurs, students, and professionals to share information about Veterans.
Core application data will come from a public Gravesite Locations data set. In this challenge you will write a NodeJS tool that imports data from the public data set into our database. As data is released periodically by the VA office, the tool needs to be able to sync the data between our db and newly released data sets.
Input data is contained in csv files, one for each US state and file locations are available at a metadata set released by the VA (metadata about all datasets made available by VA office). We are only interested at data sets with programCode="029:001", keyword "burial data" and csv as distribution format.
The tool shall first load the latest metadata file (data.json) and get urls for the csv files (unique urls) and then download and process each one. Records in the csv files do not have any identifiers, so we will use first name, last name, birth date and death date as row identifiers. Other attributes can be changed and the tool should update them when syncing the data. On the first run, all the data should be imported to db. Running the tool again would just insert new data into the database and update existing rows.
For the database we will use Prostgres, but schema is up to you as long all the data is imported into the database. Make sure to use lookup tables when necessary and don't duplicate data (cemetaries, branches, ranks, wars).
Final Submission Guidelines
Submit the source code for the tool.Submit deployment and verification guide.