TCO19 Spacenet BuildingDetector3 - TCO19 BuildingDetector3

Problem Statement

Prize Distribution

  1st:          $25,000
  2nd:          $10,000
  3rd:           $5,000
  4th:           $4,000
  5th:           $3,000
  
  Top University Team or Individual:
                 $3,000
  Total Prizes: $50,000

Background

The commercialization of the geospatial industry has led to an explosive amount of data being collected to characterize our changing planet. One area for innovation is the application of computer vision and deep learning to extract information from satellite imagery at scale. The SpaceNet Partners, CosmiQ Works, DigitalGlobe and Radiant Solutions have partnered to release the SpaceNet Off-Nadir dataset to the public to enable developers and data scientists.

Today, map features such as roads, building footprints, and points of interest are primarily created through manual techniques. We believe that advancing automated feature extraction techniques will serve important downstream uses of map data. For example, this type of map data is needed for planning international events like the 2016 Rio Olympics, as well as humanitarian and disaster response as recently observed in Haiti to aid the response to Hurricane Matthew. Furthermore, we think that solving this challenge is an important stepping stone to unleashing the power of advanced computer vision algorithms applied to a variety of remote sensing data applications in both the public and private sector.

Further background information about the contest can be found at the SpaceNet Challenge minisite.

Objective

Can you help us automate mapping from off-nadir imagery (collect from angles that are not straight down)? In this challenge, competitors are tasked with finding automated methods for extracting map-ready building footprints from high-resolution satellite imagery from high off-nadir imagery. In many disaster scenarios the first post-event imagery is from a more off-nadir image than is used in standard mapping use cases. Also, emerging next generation earth imaging satellite constellations that will provide more frequent revisists will tend to have higher-off nadir imagery. The ability to use higher off-nadir imagery will allow for more flexibility in acquiring and using satellite imagery after a disaster. Moving towards more accurate fully automated extraction of building footprints will help bring innovation to computer vision methodologies applied to high-resolution satellite imagery, and ultimately help create better maps where they are needed most.

Your task will be to extract building footprints from increasingly off-nadir satellite images. The polygons you create will be compared to ground truth, and the quality of the solutions will be measured using the SpaceNet metric.

Input Files

This dataset contains 27 8-Band WorldView-2 images taken over Atlanta, GA on December 22nd, 2009. They range in off-nadir angle from 7 degrees to 54 degrees.

For the competition, the 27 images are broken into 3 segments based on their off-nadir angle:

• Nadir: 0-25 degrees
• Off-nadir: 26 degrees - 40 degrees
• Very Off-nadir 41-55 degrees

The entire set of images is tiled into 450m x 450m tiles.

The training data has 27 folders. ���Atlanta_nadir{Off-Nadir Angle}_catid_{CatalogID}���

Each Collection folder than has 3 types of images are available for the target areas:

1. PAN: panchromatic (single channel, 16-bit grayscale, ~50 cm resolution)
2. MS: 8-band multi-channel (8*16-bit, ~2m resolution). This is the equivalent of the 8-band images from the first and second competition.
3. Pan-Sharpen: pan-sharpened version of Red-Green-Blue+NIR1 bands from the multispectral product (4 channels, 4*16-bit, ~50 cm resolution). This is formed by using the PAN image to interpolate 4 bands of the MS dataset to increase the resolution of the Red, Green and Blue bands.

Building footprints

The location and shape of known buildings are referred to as ���ground truth��� in this document. These data are described in CSV files using the following format:

ImageId,BuildingId,PolygonWKT_Pix,Occluded

img1,1,"POLYGON ((103.7 205.4 0,107.8 201.9 0,
100.5 203.2 0,94.4 208.0 0,93.0 215.7 0,92.1 226.1 0,89.6 228.8 0,
95.0 233.9 0,92.4 236.6 0,95.4 239.8 0,116.7 221.0 0,116.7 216.1 0,
103.7 205.4 0))", False

img2,-1,POLYGON EMPTY, False

img339,1,"POLYGON ((213 269 0,184 221 0,
130 249 0,154 293 0,162 288 0,165 292 0,169 290 0,180 285 0,
213 269 0),(151 253 0,177 242 0,189 263 0,164 275 0,151 253 0))", True

(The sample above contains 4 lines of text, extra line breaks are added only for readability.)

• ImageId is a string that uniquely identifies the image.
• BuildingId is an integer that identifies a building in the image, it is unique only within an image. A special id of -1 is used to mean that there are no buildings in the image.
• PolygonWKT_Pix specifies the points of the shape that represents the building in Well Known Text format. Only the POLYGON object type of the WKT standard is supported. If BuildingIdis -1 then the POLYGON EMPTY construct is used in this column. (See line #3 for an example above.) The coordinate values represent pixels, the origin of the coordinate system is at the top left corner of the image, the first coordinate is the x value (positive is right), the second coordinate is the y value (positive is down), the third coordinate is always 0. Note that the polygons must be closed: the first and last entries in their point list must be the same.
• Occluded. The imagery in this challenge was taken with a large amount of tree cover. Some buildings in the image are partially covered by trees. In those images, the visible portion of the building was traced and the building was flagged as partially occluded.

Usually a building is defined by a single polygon that represents the exterior boundary edge of its shape. This is sometimes not enough, see building #32 on image Atlanta_nadir10_catid_1030010003993E00_741251_3740439 for an example shape that contains a hole. In such cases two (or more) polygons are needed, the first one always defines the exterior edge, the second (third, fourth, etc) defines the interior rings (holes). (See line #4 above for an example of the required syntax in this case.)

The PolygonWKT_Pix coordinates refer to the PAN and Pan-Sharpen images. Note that because of the different resolutions, the coordinates of the same object on the MS images are different. There are geojson���s in the training data that allow for additional geospatial manipulation.

Notes

• The way of representing holes is different from how the WKT standard specifies them: the standard mandates that the points of a polygon must be enumerated in an anti-clockwise order if the polygon represents a shape with positive area (exterior rings), and in a clockwise order if the polygon represents a shape with negative area (interior rings). However, what appears clockwise in the latitude/longitude based geographic coordinate system appears anti-clockwise in the image space where the y axis points downwards, so in the hope of causing less confusion we chose a simpler approach: the first polygon is always positive, the rest are always negative.
• The ground truth data was created manually and has gone through a rigorous QA/QC. Nevertheless, as in all real life problems, it may contain errors. Also you may annotate building footprints differently than how the annotators did.
• In this ground truth there are a large amount of trees. These trees make the outline of buildings difficult to detect in some cases. In cases where tree coverage overlaps a building the ground truth is flagged with Occluded=True. The 'occluded' flag has no significance in scoring and need not be predicted by your algorithms. It's an additional data field that you may or may not find useful in your model.
• The task of finding building footprints is different from finding the outline of the building���s shape on an image. In most cases the difference is not significant, but you should be aware of this difference, especially when a high building is viewed at an angle. This becomes more apparent at higher off-nadir images.

Downloads

Input files are available for download from the spacenet-dataset AWS bucket. A separate guide is available that details the process of obtaining the data. See also this page for description of the Spacenet AWS data and download instructions.

Note that the same bucket holds data for the previous SpaceNet challenges as well, you need only a subset of the bucket content now: download only the files in the SpaceNet_Off-Nadir_Dataset directory. You will need the following files:

In the SpaceNet_Off-Nadir_Dataset/SpaceNet-Off-Nadir_Train/ (186 GiB Total)

Atlanta_nadir10_catid_1030010003993E00.tar.gz   6.8     GiB
Atlanta_nadir10_catid_1030010003CAF100.tar.gz   6.9     GiB
Atlanta_nadir13_catid_1030010002B7D800.tar.gz   6.6     GiB
Atlanta_nadir14_catid_10300100039AB000.tar.gz   6.9     GiB
Atlanta_nadir16_catid_1030010002649200.tar.gz   6.5     GiB
Atlanta_nadir19_catid_1030010003C92000.tar.gz   7       GiB
Atlanta_nadir21_catid_1030010003127500.tar.gz   6.8     GiB
Atlanta_nadir23_catid_103001000352C200.tar.gz   6.8     GiB
Atlanta_nadir25_catid_103001000307D800.tar.gz   6.8     GiB
Atlanta_nadir27_catid_1030010003472200.tar.gz   7       GiB
Atlanta_nadir29_catid_1030010003315300.tar.gz   6.8     GiB
Atlanta_nadir30_catid_10300100036D5200.tar.gz   6.8     GiB
Atlanta_nadir32_catid_103001000392F600.tar.gz   6.5     GiB
Atlanta_nadir34_catid_1030010003697400.tar.gz   6.8     GiB
Atlanta_nadir36_catid_1030010003895500.tar.gz   7.1     GiB
Atlanta_nadir39_catid_1030010003832800.tar.gz   7.1     GiB
Atlanta_nadir42_catid_10300100035D1B00.tar.gz   6.9     GiB
Atlanta_nadir44_catid_1030010003CCD700.tar.gz   6.9     GiB
Atlanta_nadir46_catid_1030010003713C00.tar.gz   6.9     GiB
Atlanta_nadir47_catid_10300100033C5200.tar.gz   7.2     GiB
Atlanta_nadir49_catid_1030010003492700.tar.gz   7       GiB
Atlanta_nadir50_catid_10300100039E6200.tar.gz   7       GiB
Atlanta_nadir52_catid_1030010003BDDC00.tar.gz   7       GiB
Atlanta_nadir53_catid_1030010003193D00.tar.gz   7.2     GiB
Atlanta_nadir53_catid_1030010003CD4300.tar.gz   7       GiB
Atlanta_nadir7_catid_1030010003D22F00.tar.gz    6.6     GiB
Atlanta_nadir8_catid_10300100023BC100.tar.gz    6.9     GiB
geojson.tar.gz                                 14.5     MiB
summaryData.tar.gz                            129.5     MiB

In the SpaceNet_Off-Nadir_Dataset/SpaceNet-Off-Nadir_Test_Public/ (5.8 GiB Total)

SpaceNet-Off-Nadir_Test_Public.tar.gz (5.8 GiB) 

In the SpaceNet_Off-Nadir_Dataset/

SpaceNet-Off-Nadir_Sample.tar.gz (3.5GiB)

Where

• SpaceNet-Off-Nadir_Sample.tar.gz contains a few sample training images from all 27 image collections. Use this if you want to get familiar with the data without having to download any of the large files.
• Atlanta_nadir{nadir-angle}_catid_{catid}.tar.gz is the training set that belongs to each large satellite collect. It contains imagery in the 3 formats described above. Each training set contains imagery for the exact same locations.
• summaryData.tar.gz contains the building footprint annotations for each image collection in the training set.
• geojson.tar.gz contains the corresponding geojson building footprint labels.
• SpaceNet-Off-Nadir_Test_Public.tar.gz is the testing set. It contains the same 27 folders as the Training set.

A note on image ids and image names:

The format of an imageId is Atlanta_nadir{nadir-angle}_catid_{catid}_{x}_{y}

where {x} and {y} are UTM coordinates represented as integers. An example imageId:

Atlanta_nadir46_catid_1030010003713C00_740351_3734139

Images within the PAN, MS and Pan-Sharpen folders have names like PAN_<imageId>.tif, MS_<imageId>.tif and Pan-Sharpen_<imageId>.tif, respectively. All imageIds are case sensitive.

Output File

Your output must be a CSV file with almost identical format to the building footprint definition files.

ImageId,BuildingId,PolygonWKT_Pix,Confidence

Your output file may or may not include the above header line. The rest of the lines should specify the buildings your algorithm extracted, one per line.

The required fields are:

• ImageId is a string that uniquely identifies the image.
• BuildingId is an integer that identifies a building in the image, it should be unique within an image and must be positive unless the special id of -1 is used. -1 must be used to signal that there are no buildings in the image.
• PolygonWKT_Pix specifies the points of the shape that represents the building you found. The format is exactly the same as given above in the Input files section. Important to know that the coordinates must be given in the scale of the 3-band images. So if you find a building that has a corner at (40, 20) on the 3-band image and (10, 5) on the corresponding 8-band image then your output file should have a (40 20 0) coordinate triplet listed in the shape definition.
• Confidence is a positive real number, higher numbers mean you are more confident that this building is indeed present. See the details of scoring for how this value is used.

Your output must be a single file with .csv extension. Optionally the file may be zipped, in which case it must have .zip extension. The file must not be larger than 150MB and must not contain more than 2 million lines.

Your algorithm must process the image tiles of the test set one by one, that is when you are predicting building footprints you must not use information from other tiles of the test set.

Functions

This match uses the result submission style, i.e. you will run your solution locally using the provided files as input, and produce a CSV or ZIP file that contains your answer.

In order for your solution to be evaluated by Topcoder���s marathon system, you must implement a class named BuildingDetector3, which implements a single function: getAnswerURL(). Your function will return a String corresponding to the URL of your submission files. You may upload your files to a cloud hosting service such as Dropbox or Google Drive, which can provide a direct link to the file.

To create a direct sharing link in Dropbox, right click on the uploaded file and select share. You should be able to copy a link to this specific file which ends with the tag "?dl=0". This URL will point directly to your file if you change this tag to "?dl=1". You can then use this link in your getAnswerURL() function.

If you use Google Drive to share the link, then please use the following format: "https://drive.google.com/uc?export=download&id=" + id

Note that Google has a file size limit of 25MB and can���t provide direct links to files larger than this. (For larger files the link opens a warning message saying that automatic virus checking of the file is not done.)

You can use any other way to share your result file, but make sure the link you provide opens the filestream directly, and is available for anyone with the link (not only the file owner), to allow the automated tester to download and evaluate it.

An example of the code you have to submit, using Java:

public class BuildingDetector3 {
  public String getAnswerURL() {
    //Replace the returned String with your submission file's URL
    return "https://drive.google.com/uc?export=download&id=XYZ1";
  }
}

Keep in mind that your complete code that generates these results will be verified at the end of the contest if you achieve a score in the top 10, as described later in the "Requirements to Win a Prize" section, i.e. participants will be required to provide fully automated executable software to allow for independent verification of software performance and the metric quality of the output data.

Scoring

A full submission will be processed by the Topcoder Marathon test system, which will download, validate and evaluate your submission file.

Any malformed or inaccessible file, or one that exceeds the maximum file size (150 MB) or the maximum number of lines (2 million) will receive a zero score.

If your submission is valid, your solution will be scored using the following algorithm.

For each image in the test set:

1. Sort the building polygons you returned in decreasing order of confidence.
2. For each polygon find the best matching one from the set of the ground truth polygons. Loop over the truth polygons, and:
   1. Skip this truth polygon if it was already matched with another solution polygon.
   2. Skip this truth polygon if it belongs to a different ImageId than the solution polygon.
   3. Otherwise calculate the IOU (Intersection over Union, Jaccard index) of the two polygons.
   4. Note the truth polygon which has the highest IOU score if this score is higher than 0.5. Call this the 'matching' polygon.
3. If there is a matching polygon found above, increase the count of true positives by one (TP).
4. If there is no matching polygon found, increase the count of false positives by one (FP).
5. When all solution polygons are processed then for each truth polygon that left unmatched increase the count of false negatives by one (FN).

The TP, FP and FN counters are defined separately for all 3 nadir ranges ("Nadir", "Off-nadir", "Very off-nadir"). In the above calculations the correct set of counters are used, depending on the nadir range of the actual image.

Then 3 F-scores (corresponding to the 3 nadir ranges) are calculated based on the 3 sets of TP, FP and FN counters:

F-score is defined as 0 if TP is 0. Otherwise:

F_score = 2 * Precision * Recall / (Precision + Recall),
  where
    Precision = TP / (TP + FP)
    Recall = TP / (TP + FN)

F1-Score-Total = mean(F1-Score-Nadir, F1-Score-Off-Nadir, F1-Score-Very-Off-Nadir)

Your overall score is calculated as 1,000,000 * F_Score-Total.

Note that your returned confidence values are used only to sort the building footprints you extract so that buildings with higher confidence values are tried to be matched first. The exact values don't matter, they only establish a ranking among the buildings.

Note also that due to clipping building footprints at image chip boundaries very small buildings may appear. As it is not realistic to expect that your algorithm can find such buildings, all buildings with area less than 20 (measured in pixels on the PAN imagery) will be ignored both in the ground truth and in your solution.

For the exact algorithm of scoring see the visualizer source code.

Example submissions can be used to verify that your chosen approach to upload submissions works. The tester will verify that the returned String contains a valid URL, its content is accessible, i.e. the tester is able to download the file from the returned URL. If your file is valid, it will be evaluated, and precision, recall and F-score values will be available in the test results.

Example submissions must contain building footprint predictions for these 2 image tiles (both are present in the SpaceNet-Off-Nadir_Sample.tar.gz sample set):

• Atlanta_nadir10_catid_1030010003993E00_733601_3724689
• Atlanta_nadir50_catid_10300100039E6200_747101_3741339

Though recommended, it is not mandatory to create example submissions. The scores you achieve on example submissions have no effect on your provisional or final ranking. Example submissions can be created using the "Test Examples" button on TopCoder's submission uploader interface.

Full submissions must contain in a single file all the extracted building polygons that your algorithm found in all images of the SpaceNet-Off-Nadir_Test_Public/Atlanta_nadir{nadir_angle}_catid_{catalog id}

folders, i.e. all images of all 27 collections should be processed into a single submission file. Full submissions can be created using the "Submit" button on TopCoder's submission uploader interface.

Final Scoring

The top 10 competitors after the provisional testing phase will be invited to the final testing round. Within 10 days after the provisional testing phase you are required to submit a dockerized version of your code that we can use to test your system. The technical details of this process are described in a separate document.

Your solution will be subjected to three tests:

First, your solution will be validated, i.e. we will check if it produces the same output file as your last submission, using the same input files used in this contest. Note that this means that your solution must not be improved further after the provisional submission phase ends. (We are aware that it is not always possible to reproduce the exact same results. E.g. if you do online training then the difference in the training environments may result in different number of iterations, meaning different models. Also you may have no control over random number generation in certain 3rd party libraries. In any case, the results must be statistically similar, and in case of differences you must have a convincing explanation why the same result can not be reproduced.)

Second, your solution will be tested against a set of new image files. The number and size of these new set of images will be similar to the one you downloaded as testing data. Also the scene content will be similar.

Third, the resulting output from the steps above will be validated and scored. The final rankings will be based on this score alone.

Competitors who fail to provide their solution as expected will receive a zero score in this final scoring phase, and will not be eligible to win prizes.

Additional Resources

• A visualizer is available here that you can use to test your solution locally. It displays your extracted building footprints, the expected ground truth, and the difference of these two. It also calculates precision, recall and f-scores so it serves as an offline tester. (But note that the visualizer does not enforce the limits on allowed file size and number of lines.)
• See the algorithm description and source code of the winning entries from the first and second building detector contest here and here.

General Notes

• This match is rated.
• In this match you may use any programming language and libraries, including commercial solutions, provided Topcoder is able to run it free of any charge. You may also use open source languages and libraries, with the restrictions listed in the next section below. If your solution requires licenses, you must have these licenses and be able to legally install them in a testing VM (see "Requirements to Win a Prize" section). Submissions will be deleted/destroyed after they are confirmed. Topcoder will not purchase licenses to run your code. Prior to submission, please make absolutely sure your submission can be run by Topcoder free of cost, and with all necessary licenses pre-installed in your solution. Topcoder is not required to contact submitters for additional instructions if the code does not run. If we are unable to run your solution due to license problems, including any requirement to download a license, your submission might be rejected. Be sure to contact us right away if you have concerns about this requirement.
• You may use open source languages and libraries provided they are equally free for your use, use by another competitor, or use by the client.
• If your solution includes licensed software (e.g. commercial software, open source software, etc), you must include the full license agreements with your submission. Include your licenses in a folder labeled "Licenses". Within the same folder, include a text file labeled "README" that explains the purpose of each licensed software package as it is used in your solution.
• External data sets and pre-trained models are allowed for use in the competition provided the following are satisfied:
  • The external data and pre-trained models are unencumbered with legal restrictions that conflict with its use in the competition.
  • The data source or data used to train the pre-trained models is defined in the submission description.
  • The external data source must be declared in the competition forum in the first 45 days of the competition to be eligible in a final solution. References and instructions on how to obtain are valid declarations (for instance in the case of license restrictions). If you want to use a certain external data source, post a question in the forum thread titled "Requested Data Sources". Contest stakeholders will verify the request and if the use of the data source is approved then it will be listed in the forum thread titled "Approved Data Sources".
• Use the match forum to ask general questions or report problems, but please do not post comments and questions that reveal information about the problem itself or possible solution techniques.
• The stakeholders of this competition are especially interested in how the presence of 8-band imagery improves the quality of results. They believe that the non-standard/multi-band RGB images provide extra information that your algorithms can incorporate in their decisions even better than how a trained human could.

Requirements to Win a Prize

In order to receive a final prize, you must do all the following:

Achieve a score in the top five of the average score of all collections, according to final system test results. See the "Final scoring" section above.

Once the final scores are posted and winners are announced, the prize winner candidates have 7 days to submit a report outlining their final algorithm explaining the logic behind and steps to its approach. You will receive a template that helps creating your final report.

If you place in a prize winning rank but fail to do any of the above, then you will not receive a prize, and it will be awarded to the contestant with the next best performance who did all of the above.

Top University Individual or Team

The highest scoring university individual (someone actively pursuing a Bachelor's, Master's or doctoral degree), or team of such students is awarded $3,000.

The prize is based on the rankings after the final tests. The top university individual or team prize and the final prizes are not exclusive, the same contestant / team may win a final prize and also this award. For teams to be eligible for this award all team members must be eligible for the same award. Recipients must be able to reasonably prove enrollment in a current degree program to be eligible for this award.

Definition

Examples