Challenge Overview
Challenge Objectives
-
Help us create a robust automated E2E QA framework.
Project Background
Throughout the year, we ran many challenges to create an automated E2E framework.
We used spreadsheets to store the test scenarios/cases including step-by-step descriptions and sample data (You will find one, as an example, attached on the challenge forum).
With this approach, we faced the following issues:
-
It was very hard to maintain the test scenario/case descriptions. Especially maintaining the test scenario/case IDs.
-
It was very hard to create the automated tests as the test scenario/case descriptions had many duplicate steps for very similar test cases that could be grouped.
-
The spreadsheets were usually endless as most of their content was duplicated.
On the framework side, we had the following issues:
-
We had to manually trigger the tests by pushing a new commit to the framework’s repo. We want to be able to configure the framework to “watch” another repo (eg the connect-app repo) and when anyone pushes a commit to that targeted repo, we should run the tests.
-
The output was not organized in a meaningful way.
In this series of challenges, we’re revamping our E2E testing framework to help us test different applications within the Topcoder platform.
Currently, we target only the Connect app.
Technology Stack
Previously, we used the tech stack below but feel free to suggest another approach using different technologies/frameworks.
-
Node.js
-
Selenium
-
Webdriver.io
-
AWS S3
-
CircleCI
Code access
You can find the code that we have created so far in the following repo: https://github.com/appirio-tech/qa-framework-js
You will find a self-registration link on the challenge forum in case you don’t have access to the repository.
The code is provided just for reference. You’re not expected to work based on that.
Individual requirements
1. Test scenario/case descriptions
We need your help to come up with an idea on how to effectively collect test scenario/case description so:
-
It’s easy to maintain that scenarios/cases/data.
-
It’s easy to add more scenarios/cases/data without having to change everything or start from scratch.
-
We don’t have duplicated scenarios/cases/data.
-
It’s easy for the developers to follow the test scenario/case description and implement the automated tests.
The Connect app that we target in this phase has 3 user roles:
-
Manager
-
Client
-
Copilot
Your idea should support multiple user roles.
It’s up to you to group the test scenarios/cases.
Ideally, we should group the tests per page so we can have the ability to run the tests for a particular page without running all tests.
2. Framework
We need your help to make our QA framework robust.
We need solutions for the following problems:
-
Properly organize the tests to avoid code repetition.
-
Be able to to run tests for a specific page (or multiple pages)
-
Be able to trigger the tests by pushing a commit to the targeted repo (currently the connect-app repo). Ideally, we want to be able to run a limited set of test when pushing to the `dev` branch and the full set of tests when pushing to the `master` branch of the targeted repo.
-
Properly identify the failed tests. Previously, we used IDs to tag each scenario/case but this is very hard to maintain so we need another way to properly identify which part of the app the failed test was testing.
Important Notes
-
Note that the tests will run on CircleCI and not locally.
Scoring
The copilot will review your submission using the ideation scorecard.
There will be no appeals / appeals response phase and the copilot’s decision will be final.
Your submission will be scored based on:
-
Effectiveness of your solution
-
Feasibility of your solution
-
How well you explained your solution (we don’t want a list of bullet-points)
What To Submit
Submit a document explaining your ideas in details for the 2 sections explained in the individual requirements section.
You are also encouraged to accompany your ideas with code snippets, links or anything else that may help us understanding your idea(s).
Should you have any doubt, feel free to ask in the challenge forum!