CSFV Sitemaps Generator

Project Overview

This project, the CSFV Community Gaming website project will provide the web site infrastructure and integration of the games. This will be a web site that is available on the Internet, and also will be delivered in a form where the server and the games can be installed and used behind a firewall.

We are building sitemap generator in this challenge.

Competition Task Overview

We need to build a new nodejs tool using expressjs framework to generate sitemap xml for all csfv websites, we have 6 websites that currently don't have sitemaps. Sitemaps are important for SEO.

Build a flexiable sitemap generator tool that address the following :

• It should be built using expressjs framework.
• It should have a job schedular that run the sitemap periodically. We should have a job schedular for each website, and each job will load it's own cofiguration.
  • We need the ability to generate multiple sitemaps for multiple sites at the same run.
  • The base url of the sitemap should be configurable for each job schedular.
  • The name of sitemap file to be generated should be configurable.
• The generator utility should be pluggable, you should build service for each area to generate links for it, and inject the generator to the utility via configuration. We need this flexibilty to adjust generators and add new ones without the need to recode the tool.
• The generator services needed for the following areas :
  • Fixed URLs generator:
    • It should include any url that is hardcoded and not dynamically generated i.e. home pages, faq, terms of use, about us, contact us, privacy .. etc.
    • Here is a list of pages that we scanned in the website :
      • landing page : usually accessed as *.verigeames.com/
      • Home page : usually accessed as *.verigames.com/home in all websites
      • My Home : this is in main website http://verigames.com/myHome
      • news : this is news listing page http://*.verigames.com/news
      • privacy page
      • about us page
      • FAQ page
      • terms of use page
      • login page
      • Forgot password page
      • wiki home page
      • wiki favorite page
      • games page : this is in main website http://verigames.com/games
      • play page : this page exists in mininsites http://verigames.com/play
      • play game page : this is specific for each game
    • The above urls varies from page to page, so it is preferred if the fixed URLS are configured as array of relative urls in configuration and loaded by generator, this will give us flexibility to specify which fixed urls to be used for each website.
    • It is preferred to crawl the page to generate it's title.
    • It is preffered that url prioirty is configured with the fixed URLs in configuration file.
  • News dynamic pages generator :
    • This will generate URLs for news details pages and news filtered by categories pages.
    • Relative paths are :
      • GET /newsDetails/:id or GET /static/newsDetails/:id - this is news details page
      • GET /news/filter/:categoryName - this is the news listing page filtered by category
    • You can refer to the csfv_frontend_module to understand how these URLs should be constructured
    • You can crawl the page to retrieve it's title
    • Use 0.8 as priority for the urls, make it configurable parameter.
  • Forums dynamic pages generator :
    • This will generate the forum threads listing pages filtered by categories, and forums posts pages URLs.
    • Forums paths are :
      • GET /forums/:category?/:subcategory? - this is used to generate forums threads listing page filtered by category, you will populate the urls recursively for each cateogry
      • GET /forums/threads/:id - this is the thread posts page
      • GET /forums/post/:id/edit-history - this is the history page of specified forum post :id
      • GET /forumHistory/:id - this show the forum history of specified user - :id represents username here, you can refer to the corresponding method in frontend module to understand how it works.
    • Root category shoudl be configured and when generating it's thread listing page it should not include the name in the path.
    • We need to generate url for no categories selected option, usually the url is http://verigames.com/- or http://verigames.com/flowjam/- which will list threads that are associated with root categories.
    • You can refer to csfv_frontend_module and csfv_forums_module to understand how the URLs above are constructured
    • You can crawl the page to retrieve it's title
    • Use 0.8 as priority for the urls, make it configurable parameter.
  • Wiki dynamic pages generator :
    • This will generate urls of dynamic wiki pages.
    • Wiki paths are :
      • GET /wiki/sp/:spaceKey - this is the wiki space page of the specified space key.
      • GET /wiki/sp/:spaceKey/ti/:title - this is the wiki article page of the specified space key and title
      • GET /wiki/sp/:spaceKey/ti/:title/history - this page show edit history of wiki article
    • You can refer to csfv_frontend_module and csfv_wiki_module to understand how the URLs above are constructured
    • You can crawl the page to retrieve it's title
    • Use 0.5 as priority for the urls, make it configurable parameter.
  • User and user activity pages generator :
    • This is generator for users and their activity pages in mainwebsite and minisites
      • Paths are :
        • GET /userPublicProfileData/:id - this is the url for user info in profile page, the id represents username.
        • GET /social/friends/:id - this is url of the friends list in user profile page, the id represents username.
        • GET /userPublicProfile/:id - this is the url of landing page of user profile page, the id represents username.
        • GET /activity/:id - this display the activity page, the ID represents the activity id.
        • GET /profile/:username - this is only specific to stormbound, (i.e. http://stormbound.verigames.com/profile/elkhawajah)
    • You can refer to frontend and minisite frontend modules/applications to understand how these urls are constructed.
    • You can crawl the page to retrieve it's title
    • Use 0.6 as priority for the urls, make it configurable parameter.
  • Robots.txt generator :
    • This will generate robots txt file for each website
    • Other URL paths not listed above should be added to disallow. You can refer to config/configuration.js of each website for rest of URLs
    • All rules to be added in robots should be applied to all websites to be generated.
    • Sitemap should be added to robots file.
• All generated sitemap files and robots files should be placed under public/ directory of express.js, practically, in production we will configure nginx to forward requests to robots and sitemap to this /public folder of this application.

Github Code

CSFV websites code exists in private github repository http://github.com/topcoderinc

• CSFV Mainsite
• CSFV Minisites (we have 5 branches, each website exists in a branch, the branches are minisite1, minisite2, minisite3, minisite4 and minisite5)
• CSFV CMS
• CSFV Test Data Generator
• Dependencies CSFV node modules are configured in package.json in minisites and mainsite.

Deployment Guide

Each website repository has details about deploying the code in README.md file, if you are still not sure specific deployment you can refer to docs folder under each repository in github as it contains deployment guides from past challenges, or you can post to challenge forums directly and Copilot will help asnwering questions.

Final Submission Guidelines