Low Level Test Plan for v20.11 Release

Announce
#1

Facts

  • Code freeze planned for 4th November for Badger Dgraph and GraphQL
  • We need to prepare
    • Test cases: Done (Review required from Ibrahim)
      • Pawan and Aman already working on adding test cases on test rail for feature release
      • Sankalan and Sourav to monitor and improve the format in which the tests are added by the GraphQL Team
      • Regression: Owner (Sankalan/Sourav)
      • Feature testing: Owner (Ibrahim/Pawan)
      • Performance: Owner (Ahsan) (Sankalan and Sourav to shadow)
    • Datasets: 1 tb dataset stack overflow
      • As per Daniel’s comment on the google sheet, this dataset should be enough for load testing
      • Question: Would we need other datasets for load test coverage? @vvbalaji
    • Test harness/Playground
      • NOTE: If two machines are accessible to QA folks, we can push harness setup by a month, currently the QA Team has access to Harshil’s machine and both Sankalan and Sourav are using it for testing
      • Harness setup will include logging, monitoring, easy setup/teardown and a better test isolation for different scenarios using namespaces
      • We will need help from @dmai/@joaquin to setup the playground/harness which will include Kubernetes setup with helm as the deployment manager and other tools for monitoring
    • Report generation and Jira integration
      • Owner (@hardik)
        • Integrate JIRA with TestRail
        • Work on setting up a good reporting format
        • Sourav to setup a call for the above and revert on the discussion that follows

Plan:

  • Prepare a demo for using TestRail and present it to the team during the India all hands call so that the team is familiar with using TestRail (Owner: Sankalan/Sourav)
  • Segregate tests to Sankalan and Sourav and set owners for regression testing in TestRail
  • Reach out as required for support on regression testing from GraphQL Team as the volume of testing is high
  • Setup testing milestones and a timeline for reaching the milestones, @vijay to add inputs on the timeline and milestones
  • Ready the test harness/test playground for regression testing and other related testing
  • Start executing the tests latest by 5th November
  • 4-5 weeks of testing will be done by the QA team along with Dgraph and GraphQL team
  • When above is green, we will put the build on Slash Staging and test the build for 1 week on Slash
  • We will do a partial launch to the Slash production (@gja to decide on how this launch will happen for paying and free customers)
  • 2-3 weeks live user testing on Slash production and we will be ready to release the 20.11 build
  • Bug reporting will be done on test failure and will be reported on Jira, it will then be assigned to the respective developers. Once bug fixes are pushed to the release branch, the QA team will verify the bug fixes and run regression testing to verify breakages if any

Please provide feedback to improve this and also correct us if we have missed something or over-calculated the timelines.

3 Likes
#2

@aman-bansal are we adding the test cases on TestRail? The expectation is to add all manual tests that we are doing on TestRail.

#3

Yes I did have added my test cases here https://dgraph.testrail.io/index.php?/suites/view/17&group_by=cases:section_id&group_order=asc. I guess we do need some re-organization based on review from @sourav . I will do that :slight_smile:

#4

Can I get invited to this event?

#5

Hey Omar
We plan to give a demo to the bigger team as well but yes will invite you to this event!

1 Like
#6

Meeting notes from Test Case Review Meeting with Ibrahim and team:

  • Test run by Ibrahim’s team will be done on the 1 Tb dataset. Regression tests documented in TestRail will be executed by the QA Team. Using the 1 Tb dataset for regression testing will take too long to finish.
  • Our users would be running a 3 alpha 3 zero configuration in production so we need to run our tests on a similar setup. Need to decide on test harness to run these tests.
  • We need to run regression tests will cluster encryption ON and OFF.
  • Add various tests on Snapshot transfer and shard movement
  • Improve test cases cumulated in TestRail and get them reviewed by Daniel, Joaquin and Omar

Discussion topics for tomorrow’s call 20.11 Test Plan:

  • Discuss about QA Harness (test bed) for running a 3 alpha 3 zero cluster to mimic production usage, exposing metrics and setting up monitoring etc. Running tests on Google Kubernetes Engine would make this easy for the time being until we find a cheaper solution in house
  • Discuss about QA Team and Dev team running same tests and we can streamline that to reduce repeatability
  • Discuss about dataset sizes for different stages of testing, running regression tests on 1 Tb dataset would take too long for completion before release date. Should we use a lower size dataset to run the regression tests
  • Discuss if we want to run 1 Tb data load test for Mac OS, at the current state, we will fall short of resources for testing this
#7

Here are a few things we are going to focus on during the 20.11 testing:

  • Ibrahim’s team will work on high load testing with 1 tb datasets, they will then use the directories to run more tests on it
  • Regression tests will be run by QA team and members of the GraphQL team where tests will be run on a smaller dataset (21 million perhaps) for different platforms like Ubuntu and MacOS
  • A production level test setup will be created to replicate our users’ setup and different parts of regression tests will be run on the setup (mostly priority 0 tests)
  • Production level tests will be run on a 6 node cluster deployed on Kubernetes, each alpha will be given 16/32 gb memory (as per availability) when running these tests and we will use a medium to large dataset to run these tests
  • Chaos and scale tests like Jepsen, Bank and Flock will be run by Ibrahim’s team

We are setting up a test bed that resembles our users’ production setup so that we can mimic what users do in production. Let me know if there are any questions or feedback on the above.