Testing FAQ (core team)

From Status Wiki
This is the approved revision of this page, as well as being the most recent.
Jump to: navigation, search

Current PR testing workflow for core team

All new functionality and features both are being delivered by PR builds. How to test PR? Steps below could help a bit!

  1. What is ready for testing? Good question and our team Board (https://github.com/status-im/status-react/projects/7) answers on that - check Column TO TEST
  2. Install the PR build with issue/feature you are about to test. Builds are basically available on Diawi, our #internal-jenkins channel in Slack does contain all the links for builds that are ready. Keep in mind that builds may expire, so if you are going to test something that is ready for testing for a time-being (like 3 days and longer), you need to ask PR-author to rebase and create a new build for you. If you started a build and it fails:
    1. try to re-run build
    2. check if systematic, i.e. just one PR or many
    3. ask in #devops channel (especially if it seems systematic, and not code error that Clojure dev / core contrib can fix)
    4. ping author in #clojure channel if nothing above helped
    5. if PR-author is external contributor - please comment the PR using @ [author-name] (so you are mentioning him directly and email notification for this will be received) what the problem is and ask for assistance.
  3. Move appropriate PR card to IN TESTING on the Board (https://github.com/status-im/status-react/projects/7) and let people know you are on it - assign it to yourself! :)
  4. Once installed both Android and iOS, check the functionality current PR fixes / delivers (positive/negative tests related to the feature). In curtain cases it's worth to look in 'Files changed' tab in GitHub to check the list of what was changed to get understanding of the test coverage or "weak" places that have to be covered. Ask PR-author in #clojure channel in slack what was changed if it's not clear from the notes in PR.
  5. Check reasonable regression using https://ethstatus.testrail.net/index.php?/suites/view/15&group_by=cases:section_id&group_order=asc&group_id=388 (most are automated). If more tests are needed - discuss in testteam slack channel, so we agree and update regression tests
  6. Check upgrade feature. Try to upgrade latest released version (avaliable on google play and test flight) to current PR under test.
  7. No issues? Perfect! Put appropriate label to the PR ( Tested - OK) and move the PR instance to MERGE (review/merge). Once merged it will be shown in DONE on the Board. If PR fixed what addressed in the issue description - don't forget to close the Issue.
  8. Found issues? Check for duplicates before adding one. Hint: make sure the issue is really introduced by current PR - check latest develop build too. Issue exists in develop? Check existing issues list and make sure you are not adding duplicates before creating your own bug :) There are some tips how to create a remarkable issue - please check Bug Report. And one more thing - use labels, it is superhelpful! Which labels we have? Check out this list . Once all issues are logged put label Tested-issues to the PR and notify developer that there are several problems that are preventing the PR to merge. Move the PR to CONTRIBUTOR on the board.
  9. To help others and developer please put the PR number to the end of issue title (for those which are introduced by PR). For example: Login button is disabled [1234]. That means the bug "lives" in PR 1234.
  10. Until we have Test Rail integrated with Github please add the following information to the PR when testing: App version, OS, checks that you have performed in briefly way (like logged in/ logged out/ recovered account), links to the issues logged / exising ones (if any)

Pool of daily testing tasks for core team

  1. Check automation tests results for nightly build and analyze. Log issues if necessary (check the known problems first to avoid duplicates)
  2. Check Board for TO TEST tasks (new features and bugfixes). Pick up any Unassigned and feel free to test! Do not forget to close the bug when bugfix is merged.
  3. Contribute to Ideas. If you are not assigned to any idea now - go ahead and choose it from the list (you may want to check with Test Lead which Idea is worth contributing right now)
  4. Review Open Issues list and close fixed/outdated/invalid ones.
  5. Schedule time to run regression on nightly build. Tester from the core team has 1 day during a week to dedicate only for nightly build testing and reviewing issues from status-everyday channel (where other team members write about issues they see on nightly build). Track test execution in TestRail. For slack channel - respond if issue is known and help to reproduce and report if issue is new.
  6. Create test cases and update our TestRail (https://ethstatus.testrail.net/index.php?/dashboard) with them. More we have - more of them will be automated! :)
  7. Review Instabug issues
  8. Add / maintain autotests. UI is changing so we need to keep our tests alive.

VM for upgrade testing

Below is the image on VM with 0.9.11 Status version with not so much, but lot of data: 1-1 chat, group chats, public chats. We may use this VM and account to test app upgrade scenarios.

How to install it:

  1. You need to have Genymotion
  2. You need to have VirtualBox
  3. Download this .ova file https://drive.google.com/open?id=1MBRaRx5hY0-TsAXiZqEiIn7QseS1SxAy
  4. In VirtualBox menu select File -> Import. Chose the .ova file you downloaded. And proceed import.
  5. Relaunch Genymotion so it pick up the VM.
  6. Run and locate Status app 0.9.11

User password: Aaaaaa

What data do we have inside this account:

  • 10 1-1chats created. In each 1-1 chat >20 messages sent in total from both contacts. /location and /request used as well. Some of users confirmed 
  • 10 Group chats created with random contacts > 10 messages sent by initial user in each chat
  • 5 public chats created and >5 text/emoji messages are sent by initial user

Get test tokens on ropsten

For STT simply send a transaction (can be 0 ether) with Gas limit 105000 (you need to set the value manually!) to the following address: 0x34358C45FbA99ef9b78cB501584E8cBFa6f85Cef

Handy Test Token

Lucky XS Test

Adi Test Token

Wagner Test Token

Modest Test Token

User can get 1000 of each of those by sending exactly 0 ether to the appropriate contract address.

Ropsten contract:

How to find geth.log file on Android

On Android 6+ you will need to provide Storage permission in app's settings (Settings->Application manager->Status->Permissions->Storage). Log file can be found in Download folder