Testing FAQ (core team)
Current PR testing workflow for core team
All new functionality and features both are being delivered by PR builds. How to test PR? Steps below could help a bit!
- What is ready for testing? Good question and our team Board (
https://github.com/status-im/status-react/projects/7) answers on that - check Column
- Install the PR build with issue/feature you are about to test. Builds are basically available on Diawi, our
#internal-jenkinschannel in Slack does contain all the links for builds that are ready. Keep in mind that builds may expire, so if you are going to test something that is ready for testing for a time-being (like 3 days and longer), you need to ask PR-author to rebase and create a new build for you. If you started a build and it fails:
- try to re-run build
- check if systematic, i.e. just one PR or many
- ask in
#devopschannel (especially if it seems systematic, and not code error that Clojure dev / core contrib can fix)
- ping author in
#clojurechannel if nothing above helped
- if PR-author is external contributor - please comment the PR using
@ [author-name](so you are mentioning him directly and email notification for this will be received) what the problem is and ask for assistance.
- Move appropriate PR card to
IN TESTINGon the Board (https://github.com/status-im/status-react/projects/7) and let people know you are on it - assign it to yourself! :)
- Once installed both Android and iOS, check the functionality current PR fixes / delivers (positive/negative tests related to the feature). In curtain cases it's worth to look in 'Files changed' tab in GitHub to check the list of what was changed to get understanding of the test coverage or "weak" places that have to be covered. Ask PR-author in #clojure channel in slack what was changed if it's not clear from the notes in PR.
- Check reasonable regression using https://ethstatus.testrail.net/index.php?/suites/view/15&group_by=cases:section_id&group_order=asc&group_id=388 (most are automated). If more tests are needed - discuss in testteam slack channel, so we agree and update regression tests
- Check upgrade feature. Try to upgrade latest released version (avaliable on google play and test flight) to current PR under test.
- No issues? Perfect! Put appropriate label to the PR (
Tested - OK) and move the PR instance to MERGE (review/merge). Once merged it will be shown in
DONEon the Board. If PR fixed what addressed in the issue description - don't forget to close the Issue.
- Found issues? Check for duplicates before adding one. Hint: make sure the issue is really introduced by current PR - check latest develop build too. Issue exists in develop? Check existing issues list and make sure you are not adding duplicates before creating your own bug :) There are some tips how to create a remarkable issue - please check Bug Report. And one more thing - use labels, it is superhelpful! Which labels we have? Check out this list . Once all issues are logged put label Tested-issues to the PR and notify developer that there are several problems that are preventing the PR to merge. Move the PR to CONTRIBUTOR on the board.
- To help others and developer please put the PR number to the end of issue title (for those which are introduced by PR). For example: Login button is disabled . That means the bug "lives" in PR 1234.
- Until we have Test Rail integrated with Github please add the following information to the PR when testing: App version, OS, checks that you have performed in briefly way (like logged in/ logged out/ recovered account), links to the issues logged / exising ones (if any)
Pool of daily testing tasks for core team
- Check automation tests results for nightly build and analyze. Log issues if necessary (check the known problems first to avoid duplicates)
- Check Board for
TO TESTtasks (new features and bugfixes). Pick up any Unassigned and feel free to test! Do not forget to close the bug when bugfix is merged.
- Contribute to Ideas. If you are not assigned to any idea now - go ahead and choose it from the list (you may want to check with Test Lead which Idea is worth contributing right now)
- Review Open Issues list and close fixed/outdated/invalid ones.
- Schedule time to run regression on nightly build. Tester from the core team has 1 day during a week to dedicate only for nightly build testing and reviewing issues from status-everyday channel (where other team members write about issues they see on nightly build). Track test execution in TestRail. For slack channel - respond if issue is known and help to reproduce and report if issue is new.
- Create test cases and update our TestRail (https://ethstatus.testrail.net/index.php?/dashboard) with them. More we have - more of them will be automated! :)
- Review Instabug issues
- Add / maintain autotests. UI is changing so we need to keep our tests alive.
VM for upgrade testing
Below is the image on VM with
0.9.11 Status version with not so much, but lot of data: 1-1 chat, group chats, public chats. We may use this VM and account to test app upgrade scenarios.
How to install it:
- You need to have Genymotion
- You need to have VirtualBox
- Download this .ova file https://drive.google.com/open?id=1MBRaRx5hY0-TsAXiZqEiIn7QseS1SxAy
- In VirtualBox menu select File -> Import. Chose the .ova file you downloaded. And proceed import.
- Relaunch Genymotion so it pick up the VM.
- Run and locate Status app
What data do we have inside this account:
- 10 1-1chats created. In each 1-1 chat >20 messages sent in total from both contacts. /location and /request used as well. Some of users confirmed
- 10 Group chats created with random contacts > 10 messages sent by initial user in each chat
- 5 public chats created and >5 text/emoji messages are sent by initial user
Get test tokens on ropsten
For STT simply send a transaction (can be 0 ether) with Gas limit 105000 (you need to set the value manually!) to the following address:
HND Handy Test Token 0x9e47fb3049f0d9c953f5428ce2e6c3a8321780bf LXS Lucky XS Test 0xf29d2dc0687d7d49f57d4a731ac8bfb6edc23473 ADI Adi Test Token 0xd2a816110c1177478c7e644ae4853d8e80aaec35 WGN Wagner Test Token 0x65c69bc258afa0906683f42e576b272a95c203dd MDS Modest Test Token 0x972b0570d9cd8b7c41aa8349f707ec7356daa825
User can get 1000 of each of those by sending exactly 0 ether to the appropriate contract address.
How to find geth.log file on Android
On Android 6+ you will need to provide
Storage permission in app's settings (
Settings->Application manager->Status->Permissions->Storage). Log file can be found in