Ideation - ElasticSearch/Kibana Architecture with SharePoint

1. Context

 

Project Context


This challenge is a part of the project aimed to create an internal search engine for our client's documents hosted in SharePoint.
 

Challenge Context

The client works in the insurance domain and their employees need to refer to policy documents from time to time. These documents are generally in docx and pdf formats and are kept by the client in SharePoint.

This challenge aims to generate ideas and PoC code regarding a good architecture to implement ElasticSearch and Kibana based search engine that is in sync with their files in Microsoft SharePoint.  

2. Challenge Details

 

Overview


As discussed above, the current challenge aims to generate ideas and PoC code regarding a good architecture to implement ElasticSearch and Kibana based search engine that is in sync with their files in Microsoft SharePoint.



Project Goals

The goal of the architecture is to solve the client need for a web based search engine, which they can use to locate relevant content in the documents. These documents have not been shared with us and are confidential. These documents reside in Microsoft SharePoint, hosted by the client.

The goal of this project is to deliver an ElasticSearch based system which can automatically access client's SharePoint, and does not require manual indexing of files. This ElasticSearch instance will also be integrated with Kibana, and eventually we'll deliver a web based interface with multiple filters and a text box to enter search terms. The final code will be hosted on Azure.

Expected Outcome

The expected output of this challenge is a document as well as a demo PoC, detailing the architecture along with the exact technical steps that need to be followed to implement the architecture.

The contestants can basically think of their output as the plan and reference for actual development of the final architecture in a systematic step by step manner.

Particularly, the details in the document should be such that if those details were shared as challenge specification in a Topcoder challenge, the contestants should be able to go through it and generate the required output components and eventually the entire architecture.  

Proof-of-concept (PoC) Code

The demo can be a simple implementation with examples about how to set up a demo sharepoint with some demo files, and some code to demonstrate how ElasticSearch can integrate with it. Demo of Kibana integration is not essential, but recommended. If Kibana is not included in the demo, it should be ensured that the instructions about its installation etc are detailed enough to leave no room for ambiguity.

Demo Documents

In this challenge, we will not be sharing any subset of documents by the client. The contestants can use any docx, doc and pdf document and host them in demo SharePoint accounts for their reference and for the PoC. Note - in case the ideation document is exceptional and well detailed, then such submissions will be considered for the prize even in case it doesn't include the PoC.

3. Scorecard Aid

Judging Criteria

This ideation challenge will be reviewed subjectively. The primary criteria would mainly revolve around the following questions:

• Will this architecture be able to keep ElasticSearch up to date with the documents added in SharePoint?
• Is the architecture optimal? As in, does it use some slow and unnecessary steps in case easier and more efficient alternative are available?
• As a contestant, would a developer be able to use the details in the document and PoC as a reference and gameplan to build the actual code.
• Is the overall documentation and PoC readable and well-presented? (note - it doesn't need to have anything extraordinary in terms of presentation)
  

Final Submission Guidelines

A valid submission should contain the following:

• Ideation Document - With details about the proposed architecture, along with detailed technical steps.
• Demo PoC.