Topcoder Review App Design Challenge

Welcome to "Topcoder Review App Design Challenge". In this challenge, we are looking for your help to create the best UI/UX for our new Topcoder Review application where a Copilot, Admin, Members, and Reviewers can manage, collaborating, and doing review process to judging submissions in order to provide the best submission deliverables for the challenge that meet the client expectation.  

Read the challenge specification carefully and watch the forums for any questions or feedback concerning this challenge. Let us know if you have any questions in the challenge forum!

Round 1

Submit your initial design as much as possible for a Checkpoint Feedback

• As part of your checkpoint submission, you must upload your submission to MarvelApp so we can provide direct feedback on your designs. Please include the MarvelApp URL in your notes.txt. 
• Make sure all pages have the correct flow. Use the proper file numbers, (1, 2, 3, etc.)

Round 2

Submit your Final Design (All requirements) plus Checkpoint Updates

• As part of your Final submission, you must replace your checkpoint submission with the final submission into MarvelApp so we can provide direct feedback on your designs. Please include the MarvelApp URL in your notes.txt.
• Make sure all pages have the correct flow. Use the proper file numbers, (1, 2, 3, etc.)

• This project purpose is to replace the current Online Review platform with a new, fresh, reliable, and competent application that easier to be used by Copilot, Admin, Members and Reviewers when working and do review process in any Topcoder challenges.
• We're looking for a fresh idea, concepts and the best UI/UX for the new Review application.

• Create the Best possible UI/UX design for "Topcoder Review App" that able to offers a great collaboration process between each role and easier to use. 

• Copilot or Admin create and launch a challenge (Design / Development / Algorithm / Marathon) to Topcoder pipeline (Out of Scope)
• Each Challenge will open 2 slots of review position and it will appear in the challenge page or in the review opportunities page
• The reviewer will be able to see the review opportunities page, view the review opportunity of each challenge and apply to any open review position via system or manual request
• Copilot or Admin will be able to see the list of available reviewer that apply for a challenge review position
• Copilot or Admin will be able to assign reviewers to their challenge (usually each challenge will need 2 reviewers)
• The reviewer will get notified via email once they are selected to do Review in the challenge
• The reviewer will get notification about review phase started and will be able to see/view pending deliverables (how many submissions that needs to be reviewed in the challenge)
• The reviewer will open scorecard of each submission and submit their review (We've got plenty of Scorecard, we need a generic feeling of the scorecard)
• Once all Reviewer finish the review process, Reviewer will be able to see the complete review list (all reviewers scorecard to all submissions)
• Once review process finish, Members will be able to see their scorecard and able to do complaints/justification if they think the scoring did not make any sense or doubtful or the review is wrong via "Appeal Phase"
• Once the Appeal phase is close, the Reviewer will be able to provide their clarification via "Appeal Response" phase and decide whether they want to keep the score or adjust their scoring if the member appeal makes sense/correct
• After Appeal Response close, Copilot or Admin will be able to check the review process and able to edit review scorecard if necessary based on the facts available in the challenge
• A copilot or Admin should be able to re-run an automated check (eg AV Scan)

• Consistency with Topcoder Branding
• KISS principle needs to be applied here. We want a straight up nice UI
• Clean and streamlined design and user flow
• The application should provide an intuitive process
• Looking for a fresh user-experience!
• Screen Size: 1366px width and height as required (Desktop)

• If you have any doubts or questions regarding challenge requirements, please ask in our challenge forum:
• https://apps.topcoder.com/forums/?module=ThreadList&forumID=682121

• Please refer to this app to see the new look and feel for our new challenge page: https://submission-review.topcoder.com/
• The new application still under development so any ideas, concepts, and new UI propose is welcome!
• We need your ideas to define and creates a design for our new review page 
• For general information about Topcoder Review Process and Rules, please refer to this page: https://help.topcoder.com/hc/en-us/articles/222503827-Development-Reviewer-Role-Responsibilities 

• Copilot and Admin will be able to see a list of reviewers that applied to their challenge and see the same view with the “Apply review list page”, but with additional information like reviewer track record (how many review has been done, review success rate, rating, comment, etc)
• Show the name of the challenge, timeline so reviewers can measure their commitment, review payment, spec details, notes, countdown timer to apply for a review, notes, other reviewer applicants, etc
• If the review assignment needs to be done manually and not by the system automatically, there should be a button next to each reviewer applicant so Copilot and Admin can assign reviewer manually to their challenge (assignment limited to 2 reviewers only, with a primary reviewer and reviewer roles applied)

• All roles will need to do some interactions on this page
• Interactions will be vary depending on each role in the challenge
• We need some clear navigation where the user can navigate to this page from the challenge details page or from the review opportunity page
• Please see below for interactions needed in this page for each role:

• Will be able to see only their own submission in this page after they upload their submission via the upload page
• Once Review phase complete/done, they will be able to see the score from each reviewer
• Able to drill down into the scorecard to see their details score and information about the score
• Able to add their appeal text when Appeal phase open for any review that is not correct

• Will be able to see all submissions available in the challenge that needs their review 
• Ability to download submissions from the system
• Reviewer only able to see their own review scorecard (unable to see other reviewer scorecards)
• Ability to drilled down into each submission to judge the submissions using Topcoder review scorecard that being assigned to the challenge (doing review process)
• Show time/countdown timer to show remaining time for doing the review
• When appeal phase open, they will be able to see how many appeal text that they need to answer per submission in the Appeal response page
• Once review phase is done and winners announced, the reviewer will be able to see other reviewer scorecards

• Will be able to see all submissions and scorecard from members and reviewers all the time
• Ability to manage and monitor the whole review process from start to finish
• Ability to remove or add a reviewer to the challenge
• Ability to edit reviewer scorecard 

• A reviewer will need to open each submission scorecard and do judging and inspection to the submission based on criteria provided in the scorecard
• There will be a set of questions that ask whether the submission achieve the maximum expectation of Topcoder standard or if there is something missing in the submission. 
• Each question have different score weight depends on the importance of the questions, and the reviewer will need to do scoring based on the relevance between submission and the questions
• For the answer/scoring, we are going to use select box/dropdown to determine the score, it can be a range of variation between ‘Yes/No’, ‘0-3’, and some have ‘0-9’, etc. 
• For each question, there will be a hint to the reviewers that tells them about how to do the scoring perfectly and choose the correct score for each question
• Reviewers also will need to provide comments into each question, providing their expert explanation of why they choose the score for the submission in those particular questions. The reviewer can add notes/comments as much as needed, there should be 
• Reviewer comment needs to be ranked too with Required/Recommended/Comment value for comment ranking: 
• Required means the notes is crucial and needs to be fixed
• Recommended means the notes only for enhancement or to make submissions more easily to use
• Comment means the notes only for general information
• Reviewers will be able to save the scorecard as a draft and continue working on the scorecard for later without losing their current work 
• Once reviewer finish reviewing the submission, they can close the scorecard and mark it as completed
• Please see the following link to see sample scorecard for a challenge: https://software.topcoder.com/review/actions/ViewScorecard?scid=30002110
• The reviewer needs to answer all questions and provide notes for each of them
• Read this page to see how reviewer should fill the scorecard: https://help.topcoder.com/hc/en-us/articles/222503827-Development-Reviewer-Role-Responsibilities 

• After Review phase Ends, the challenge will enter the “Appeal” phase where members can check their scorecards and inspect the review result by the reviewers 
• They will be able to see the score added by the reviewer, and see notes/comments from the reviewers on each question in the scorecard
• If the member disagrees or have doubts regarding any reviewer score in each question, they can open the question and appeal the score for that question.
• Members will need to provide a clear explanation of why the score is not correct and suggest the rational score for the questions
• The user will be able to appeal any questions until the appeal phase reaches its deadline

• The next step after Appeal phase ends is the Appeal response phase
• In this phase, the reviewer will check back the scorecard from each submission and answer members doubt/complaints from the appeal phase
• Each response will be listed/saved and appear in the scorecard, from Reviewer comment in the review phase, Member complaints in the appeal phase, and the Reviewer Response text in the Appeal Response phase
• A reviewer will be able to decide whether they want to keep the score or adjust/raise the score for the particular questions that get appeal comment from the members
• Once all Appeal questions answered, the reviewer will be able to complete the appeal response

• Once all the review process complete, and if there is any case/issue happen in the review process, Copilot and Admin can jump in to fix the problem by doing an investigation to the problem  
• Copilot and Admin will be able to fix any score based on the requirements and facts gathered during the investigation
• Copilot and Admin still need to provide comments and clarification to the scorecard to explain their decision to change or keep the score
• A copilot or Admin should be able to re-run an automated check (eg AV Scan)

• Do you think the review process is perfect? If not, we encourage you to let us know how the review process should be done! 
• We are very welcome to any ideas that able to enhance and improve the review process for judging submissions
• Show us your creativity!

• Keep things consistent. This means all graphics styles should work together.
• All of the graphics should have a similar feel and general aesthetic appearance

• We need you to upload your screens to Marvel App
• Please send your marvel app request to fajar.mln@gmail.com (Challenge Copilot) 
• You MUST include your Marvel app URL (in your marvel app prototype, click on share and then copy the link & share it with your notes/comments this link while you upload).

• Follow Topcoder Branding Guidelines

• Google Drive: https://drive.google.com/drive/folders/1LNuHsgUzs6rUMy33tH3dkAqG9rh-xsGE?usp=sharing 
  • Screenshot of existing review platform (Online Review)
  • Topcoder Submission Review Sketch file

• Web/Desktop: 1366px width with Height adjusted accordingly

• Topcoder Reviewers, Members, Copilot, and Admin

• Your submission will be judged on the following criteria:
  • The overall idea and execution of concepts
  • How well does your design align with the objectives of the challenge
  • Execution and thoughtfulness put into solving the problem
  • The overall design, UI, and user experience
  • Cleanliness of screen design and user flow
  • Ease of use 

• Please create your preview image as one (1) 1024x1024px JPG or PNG file in RGB color mode at 72dpi and place a screenshot of your submission within it.

• Submit JPG/PNG image files based on Challenge submission requirements stated above
• MarvelApp link for review and to provide feedback

• All source files of all graphics created in Sketch and saved as an editable layer

• As part of the final fixes phase, you may be asked to modify your graphics (sizes or colors) or modify overall colors.