How to help with testing

From Status Wiki
This is the approved revision of this page, as well as being the most recent.
Jump to: navigation, search
This article seems to be up-to-date on 06/06/2018

Check if you are ready to help

Short Details
app is unstable If you are not ready to deal with unstable/crashing/weird and other typical traits of the untested app then you should not help with testing
create backup of the device BEFORE testing If you are ready to help with testing then make sure you have a recent backup of your device as there is no guarantee that next build will not introduce critical issues that cause complete device reset and data loss.
2 devices for testing This is app with lot's of messaging functionality, so you need at least 2 devices to test it. Ideally, it's 2 real devices. Ok: 1 real and 1 emulator like Android emulator or Genymotion.
test on wifi Current versions consume lots of traffic and battery, make sure you are using wifi for testing.
do not send real ETH or SNT to the app We will test only in Testnets (Ropsten and Rinkeby). Test ether should be used to check transactions are working. We will never ask you to send real SNT or ETH for testing

Test all features in the nightly develop build

The most useful would be to check nightly develop build on Android. We expect that features merged in the develop build are stable enough and this build is free of obvious bugs. This is of course just an assumption and should be verified. There are end-to-end tests that cover some common scenario. For the rest we test manually once in 2-3 days, so anyone who can find and report not-known issue is welcome to the club. There is an outdated version of the smoke test cases (for version 0.9.10) available here that we will update soon. It can be used as a base to learn the features of the app.

Each European night there is a new nightly build created and it’s very important to take the latest because issues you will find in 2-days old build might be already fixed in the recent build. List of the builds is not sorted by the latest date, so to find the latest search by the date, e.g. 06-dec

When you report an issue it’s also important to mention that nightly build X was used. Just add an info about the date and link to the apk you’ve used in section Additional Information of the issue. When reporting an issue, please follow the guidelines from How to report a Bug.

Suggestions on the procedure to check dev build:

1 Perform fresh install to see that build works. Fresh install = previous build was uninstalled or never installed on the device and new version is installed. There is no old data from the previous version available in the app (user profile, chats etc).

Do not perform install of a new build on top of previously installed develop or PR build.

2 Test basic flows that typical user will do. You can use Smoke Tests checklist for this
3 Perform upgrade to see that older version can be upgraded using build under test. Upgrade = released version from the TestFlight/PlayStore was installed on the device and "build under test" is installed on top of it. No uninstall was performed, so old data like user profile, chats, contacts is available.

Data for upgrade is listed in Smoke Test checklist Before making upgrade: record (video or pictures) the state of the data as anything can be broken after upgrade. For example, if there is "requesting ether" message sent before upgrade then chat will contain empty boxes instead of this message, To identify and prove what was there you need to record initial state with video/screenshots.

4 After upgrade - check that old data is visible/usable. Test basic flows that typical user will do using old data and new data
5 Fresh install - do "crazy" things, not typical flows
6 Upgrade - do "crazy" things, not typical flows

Find and report issues in the stable released versions on iOS or Android

If you don’t want to install nightly apk build then you can help by finding and reporting issues in the released Android version available on PlayStore or in iOS version from TestFlight. This is less useful as we continue to fix known issues and modify existing areas, so it might happen that found issue is already fixed or area is being re-designed in the current develop build. When reporting an issue, please follow the guidelines from How to report a Bug

Reproduce issues with "Help to reproduce" label and add extra info

Sometimes we see the issues or get the bug reports but can't find the exact steps/model/condition to reproduce them. You can search in the Github repo by label Help to reproduce to see the list and try to reproduce bugs. If you can then add as much info as possible: exact steps, logs, video, info about the phone model/OS, anything else that can help us to reproduce and then to fix a bug.

Test new features in PR builds

Team is working on providing an access to the PR builds to the external contributors, including testers. However, PR builds are expected to be tested faster and thoroughly by someone who is very familiar with the app. This is what core test team is doing daily: testing PR builds that introduce changes/fixes/features. So, if you are just starting to help it would not be the best option to choose.

Check the Python end-to-end test suite and create new tests

Setup and run existing automated tests. Review the tests and create new ones in PR. You might also like to create an issue in Github repo marked Autotests in title to explain which test you want to automate, so core test team can verify you have the correct assumptions about expected results.