Testing FAQ (core team)
Current PR testing workflow for core team (will be updated)
All new functionality and features both are being delivered by PR builds. How to test PR? Steps below could help a bit!
- What is ready for testing? Good question and our team Board (https://github.com/status-im/status-react/projects/7) answers on that - check Column TO TEST
- Install the PR build with issue/feature you are about to test. Builds are basically avaliable on Diawi, our #internal-jenkins channel in Slack does contain all the links for builds that are ready. Keep in mind that builds may expire, so if you are going to test something that is ready for testing for a time-being (like 3 days and longer), you need to ask PR-author to rebase and create a new build for you. If you started a build and it fails:
- try to re-run build
- check if systematic, i.e. just one PR or many
- ask in #devops channel (especially if it seems systematic, and not code error that Clojure dev / core contrib can fix)
- ping author in #clojure channel if nothing above nelped
- if PR-author is external contributor - please comment the PR using @ [author-name] (so you are mentioning him directly and email notification for this will be received) what the problem is and ask for assistance.
- Move appropriate PR card to IN TESTING on the Board ( https://github.com/status-im/status-react/projects/7) and let people know you are on it - assign it to yourself! :)
- Once installed both Android and iOS, check the functionality current PR fixes / delivers (positive/negative tests related to the feature). In curtain cases it's worth to look in 'Files changed' tab in GitHub to check the list of what was changed to get understanding of the test coverage or "weak" places that have to be covered. Ask PR-author in #clojure channel in slack what was changed if it's not clear from the notes in PR.
- Check reasonable (and even not) regression around.
- Check upgrade feature. Try to upgrade latest released version (avaliable on google play and test flight) to current PR under test.
- No issues? Perfect! Put appropriate label to the PR ( Tested - OK) and move the PR instance to MERGE (review/merge). Once merged it will be shown in DONE on the Board. If PR fixed what addressed in the issue description - don't forget to close the Issue.
- Found issues? Check for duplicates before adding one. Hint: make sure the issue is really introduced by current PR - check latest develop build too. Issue exists in develop? Check existing issues list and make sure you are not adding duplicates before creating your own bug :) There are some tips how to create a remarkable issue - please check Bug Report. And one more thing - use labels, it is superhelpful! Which labels we have? Check out this list https://hackmd.io/s/HyopujPPZ . Once all issues are logged put label Tested-issues to the PR and notify developer that there are several problems that are preventing the PR to merge. Move the PR to CONTRIBUTOR on the board.
- To help others and developer please put the PR number to the end of issue title (for those which are introduced by PR). For example: Login button is disabled . That means the bug "lives" in PR 1234.
- Until we have Test Rail integrated with Github please add the following information to the PR when testing: App version, OS, checks that you have performed in briefly way (like logged in/ logged out/ recovered account), links to the issues logged / exising ones (if any)
Pool of daily testing tasks for core team
- Check automation tests results for nightly build and analyze. Log issues if necessary (check the known problems first to avoid duplicates)
- Check Board for TO TEST tasks (new features and bugfixes). Pick up any Unassigned and feel free to test! Do not forget to close the bug when bugfix is merged.
- Contribute to Ideas (https://github.com/status-im/ideas). If you are not assigned to any idea now - go ahead and choose it from the list (you may want to check with Test Lead which Idea is worth contributing right now)
- Review Open Issues list and close fixed/outdated/invalid ones.
- Schedule time (not less than 1 hour) to run regression on develop build.
- Create test cases and update our TestRail (https://ethstatus.testrail.net/index.php?/dashboard) with them. More we have - more of them will be automated! :)
- Review Instabug issues
- Add / maintain autotests. UI is changing so we need to keep our tests alive.
VM for upgrade testing
Below is the image on VM with 0.9.11 Status version with not so much, but lot of data: 1-1 chat, group chats, public chats. We may use this VM and account to test app upgrade scenarios.
How to install it:
- You need to have Genymotion
- You need to have VirtualBox
- Download this .ova file https://drive.google.com/open?id=1MBRaRx5hY0-TsAXiZqEiIn7QseS1SxAy
- In VirtualBox menu select File -> Import. Chose the .ova file you downloaded. And proceed import.
- Relaunch Genymotion so it pick up the VM.
- Run and locate Status app 0.9.11
User password: Aaaaaa
What data do we have inside this account:
- 10 1-1chats created. In each 1-1 chat >20 messages sent in total from both contacts. /location and /request used as well. Some of users confirmed
- 10 Group chats created with random contacts > 10 messages sent by initial user in each chat
- 5 public chats created and >5 text/emoji messages are sent by initial user
Get STT tokens on ropsten
Simply send a transaction (can be 0 ether) to the following address: 0x34358C45FbA99ef9b78cB501584E8cBFa6f85Cef