Challenge Overview
Challenge Objectives
-
Identify & log bugs and provide usability feedback on the provided iOS and Android applications.
About the Application
Sonicwall SonicExpress is an iOS/Android app that allows customers to monitor and manage SonicWall firewalls, services and their licenses. It’s in beta right now and SonicWall wants us to review the app, submit functionality defects, usability issues, workflow enhancement suggestions from an end user standpoint.
iOS App Access:
Since the app is currently in beta, access to the iOS app will need to be enabled by SonicWall who is using TestFlight to distribute the app for testing. Post your Last name, First name and Apple ID in the forums if you would like iOS app access and we will share your information with SonicWall, who will then add you to the TestFlight app. If you don’t get an e-mail within twelve hours, please check your spam folder. Please let us know in the forum, if you continue to face access issues.
Android App Access:
Since the app is currently in beta, it has not yet been hosted on the Android play store. If you want to test the Android app and provide useability feedback, you can download the APK from the challenge forum.
Other things you will need to test the app:
MySonicWall account:
In order to login to the App, you will need to have a MySonicWall account. If you don't have a MySonicWall account already, please login into mysonicwall.com and create an account.
Access to a SonicWall firewall:
SonicExpress app works only with the new version (version 7) of SonicOS which is the operating system that runs on SonicWall firewalls. To help you test SonicExpress, we have deployed SonicWall NSv OVAs on our ESXi servers. SonicWall NSv is the virtualized version of SonicWall next generation firewalls. Each of you will be given access to one of these virtual firewalls to test your app with.
Additionally, if you have access to a VMware ESXi server, we can also provide you with the OVA image that you can use to deploy the firewall on your own ESXi servers.
Serial numbers and Authentication code:
We will share serial numbers and authentication codes that are used to license and register the firewalls with your MySonicWall accounts. You can use the SonicExpress app to register the Serial number.
What to Test (Scope)
In this challenge, you need to test the provided iOS application, submit functionality defects, usability issues, workflow enhancement suggestions from an end user standpoint.
-
App URL: TestFlight for iOS app and we will provide the Android apk file
-
Accepted types of bugs: We will target functional, UI as well as usability issues for this bug hunt.
-
Primary target device(s): Mobile (iOS and Android)
-
Do not report edge cases. An edge case would be anything that does not reflect typical user behavior. They are accepted according to the impact to end-user and based on the workarounds available.
How to Create New Bug Report
-
Create an account on Atlassian (if you do not already have one): https://id.atlassian.com/signup
-
You can get access to the Jira project by posting your email on the challenge forum.
-
Issues/Bugs found in this application must be created through the Jira platform. DON'T use any other link to create new issues OR submit a document, they won't be counted and won't be paid.
Issue Reporting Guidelines
For each report of a bug, we need the following information:
-
Descriptive issue title
-
Device type and OS version
-
Steps to reproduce, including any needed information (Must list all the steps that need to reproduce the bug, DON'T list only the page without test data)
-
Current results before the bug is fixed
-
Expected results, after the bug is fixed
-
Screenshots, videos: Provide a URL of the screenshots/videos showing the bug
-
Bug priority
-
Bug type
IMPORTANT NOTE:
Missing or Incorrect details to ANY of the above fields will mark the bug report as INCOMPLETE.
For example, Incorrect Steps, Missing Screenshot/Screencast, Incorrect Actual and Expected results etc.
Be careful when you are providing only the page and not listing the steps to go to that particular page in 'Steps to reproduce' section. Sometimes the provided page with parameters won't load the page to the reviewer and the bug may get closed as 'CAN'T REPRODUCE'. So better to list all the steps till the end or double check the page is loading or not.
Issue Weights and Scoring
-
Scoring will be based on the number of bugs by weight. Be sure to correctly attach a weight to your bug. The copilot has the right to change a severity at his / her discretion.
-
Only verified issues will be counted. Tickets created for enhancements or that are not bugs will not be counted. Duplicate issues will be closed and not counted. Log issues according to the guidelines above. Issues that do not follow these guidelines may be rejected due to lack of information.
-
For challenge scoring, the user with the most verified issues will be selected as the winner. If two users submit the same issue, the user that submitted the issue first will receive credit.
-
Please focus on functionality, UI and usability testing All bug reports based on your own assumptions will be rejected.
There are no fixed prizes on this challenge. We will be awarding prizes based on the number of valis issues you will log. Each issue will have a different price based on its severity. You'll get paid for each issue based on the following pricing breakdown:
-
Blocker/ Critical: $10 [10 Points]
-
Major: $8 [8 Points]
-
Minor: $5 [5 Points]
-
Enhancement: $5 [5 Points]
P1 - Blocker: This bug causes the app to fail. No workaround exists. E.g. app crashes, app freezes.
P2 - Critical: This bug causes the app to fail for some specific cases. No workaround exists.
P3 - Major: This bug causes the app to fail, but there’s a workaround to prevent that issue.
P4 - Minor: This is an annoyance, but won’t prevent the app from running normally.
P5 - Enhancement: Something noticed by testers that should be fixed, workflow / end user experience can be improved, but isn’t considered a bug.
We will also offer $2 to each registrant, who makes at least 1 valid submission.
Important Notice
-
Follow the standard topcoder Bug Hunt Rules.
-
If you do not properly document your bug reports, they will likely be rejected due to lack of information or documentation. If you submit the same bug in multiple areas/pages, (for instance, Same validation issue of a form can be found in different pages/sections) you will likely get credit for the original bug report only. The others will all be closed as duplicates.
-
If you duplicate an issue on a platform or browser that hasn’t been tested yet, you should create a new issue and add a link/reference in the issue description to the existing issue number. Our copilot will review these items and consolidate them later. Please don’t make adjustments or change labels of existing issues logged by other competitors.
-
DON'T RE-OPEN the issues in the review phase and anyone who RE-OPENS a ticket will be disqualified from the challenge.
-
If Mobile and Tablet testing are available DON'T create the same issue on different platforms; instead, merge them into one; All the others will be marked as Duplicate.
-
If you see multiple broken links on the same page combine them into one ticket. Others will be marked as DUPLICATE.
-
You must not edit the bug report once created, so make sure you enter all the details at the time you create the issue, otherwise, your issue will be moved to the end of the queue. If you really need to edit an issue you must use the comments section for this (i.e. add a comment to describe any changes you want to make to the issue), and we'll decide whether the changes are major enough to move the issue to the end of the queue. You are allowed to add screenshots in the comments section though, assuming your issue report contains all the details when created.
-
You must specify the test data you have used in the 'Reproduction Steps', All the issues will be marked as 'Incomplete', if the correct test data is not provided.
-
Keep an eye on the issues being submitted by other participants to minimize the time you may be spending on duplicate efforts. Knowing what has already been reported will allow you to better focus your time on finding yet undiscovered issues.
-
There will be no appeals phase. The decision of Copilot for validity and severity of each filled issue will be final.
What to Submit
Submit all your bugs directly to Jira. When you are done with your submissions please submit a .txt file using the “Submit” button before the submission phase ends. In this file include:
-
Your Topcoder handle (The one displayed in the top right corner near the Profile picture)
-
Your Jira email/handle used to raise the issues.
- ALL SUBMISSIONS WITHOUT THE ABOVE INFORMATION WILL BE REJECTED AND WON’T BE PAID.
- IMPORTANT: Submit the above details before the Submission Phase ends.
- DON'T use any other link to create new issues OR submit as a document, they won't count and won't be paid.