2D image and 3D object matching - Fujitsu Object Matching Challenge Series 3

Background

Fujitsu has an application that matches CAD data (blueprint) with the actual manufactured product to inspect if the product is built correctly - as shown below.

Since this application is built as Windows desktop application, the client wants Topcoder community to build Web based system using browser as UI.
 

Overall Objective

The customer wants Topcoder community to create a prototype which implements the core features of the existing application with Web technology in multiple challenges.
The following diagram shows an illustration of the prototype:

The client is offering the following goals:
1. Algorithm and API to detect lines/curves in 2D images 2. Algorithm and API to detect lines/curves in 3D model 3. Algorithm and API to calculate transformation matrix of 3D model to match 2D image 4. UI to show the overlaid image of 2D image and transformed 3D model

Goal of this challenge

The client is interested in matching a 2D image and a 3D model. As we have obtained algorithms that can detect lines and curves in 2D images and 3D models separately through the previous contests. Now, we are going to match them.
This challenge consists of the following four subtasks. (The first two are the same ones as goals in the previous challenges.)

Overview image of the subtasks

1. Detecting lines/curves in 2D image

2. Detecting lines/curves in 3D object

3. Matching 2D image and 3D model with results of 1. and 2.

This is one of the main parts in this challenge. The goal of the task is to calculate the transformation matrices to transform 3D model to match to 2D image with lines and curves data extracted from 2D image and 3D model data in the task 1 and 2, and Camera calibration parameter.
The transformation matrices consists of translation, rotation and scaling matrix.
Input

a) Output of 1.
b) Output of 2.
c) Camera calibration parameter

Output

Transformation matrices including translation, rotation and scaling.

* Sometimes, the user may provide a wrong image or select a wrong model. If the algorithm cannot make the matching between the given image and model, please return some message like “cannot match”
* Although the output format is not specified, JSON is preferred. You can propose your own output format but you need to carefully document it.
* In the matching process, it’s expecting to utilize not only the lines, but also curves well.
* The existing customer-owned application is not able to process this in fully automatic. It requires users to choose some pair of edges, one is from 2D and another is from 3D, and to try to find the best match with this help. So, something to help for processing can be introduced, which is like pairs of edges chosen by human (although it needs UI), if it’s difficult to achieve the goal without it.

4. Displaying the 2D image and transformed 3D model overlaid in the browser

The last part is a UI stuff to show overall result in the web browser.
This viewer application is expected
- to show 2D image.
- to transform 3D model with matrices of translation, rotation and scaling.
- to show the transformed 3D model overlaid with transparency on the 2D image.
- to enable users to perceive how much the image and model are matched and differences between them.
The application is supposed to be built with the modern front end technologies: Bootstrap, AngularJS, Three.js, and etc.
Input

a) A 2D image of some object.
b) A 3D object model file. (Wavefront obj format)
c) Output of 3.

Output

An overlaid 2D image and 3D model shown in the browser.

Note

We provide some public examples as well as the winning solutions of both previous contests. They can be downloaded after your registration. You can also develop your own line/curve detection methods for 2D images and 3D models. And if you have other brilliant ideas to directly match the image and object without line/curve detections, you can also try.

 

Judging Criteria

Note that all codes in your submission should have propoer licenses. Otherwise, the submission will be considered as invalid. Please check the "Licenses & Attribution" section for details, or ask questions in the forum.

The submissions will be evaluated from different aspects:

1. Quality (60% = 20% + 30% + 10%)

2. Readability (20%)

3. Innovativity (20%)

Note

Requirements to Win a Prize

Final Submission:

Evaluation Criteria

You will be judged on the quality of your ideas, the quality of your description of the ideas, and how much benefit it can provide for the client.  The winner will be chosen by the most logical and convincing reasoning as to how and why the idea presented will meet the objective.
Note that, this contest will be judged subjectively by the client and Topcoder. However, the following criteria will largely be the basis for the judgement: (check Judging Criteria for more details)

Licenses & Attribution

Third-party assets used to build your item must be properly licensed or free for commercial use. MIT, some modified BSD, Apache 2 licenses are ok, but if a library is not commercial friendly you need to get our approval first. Generally, GNU License is not allowed in this challenge.
Sufficient information regarding third-party assets must be present in your documentation. This includes the author, license info and a direct link to the asset online.
 

Additional Information

- This contest is not rated.
- The review process with the client - The previous challenges:

• "Lines and Curves Detection in 2D images"
  "Lines and Curves Detection in 3D object model"

Q&A

Feel free to ask questions by posting in the contest forum.
 

Final Submission Guidelines