Code freeze planned for 4th November for Badger Dgraph and GraphQL
We need to prepare
Test cases: Done (Review required from Ibrahim)
Pawan and Aman already working on adding test cases on test rail for feature release
Sankalan and Sourav to monitor and improve the format in which the tests are added by the GraphQL Team
Regression: Owner (Sankalan/Sourav)
Feature testing: Owner (Ibrahim/Pawan)
Performance: Owner (Ahsan) (Sankalan and Sourav to shadow)
Datasets: 1 tb dataset stack overflow
As per Daniel’s comment on the google sheet, this dataset should be enough for load testing
Question: Would we need other datasets for load test coverage? @vvbalaji
Test harness/Playground
NOTE: If two machines are accessible to QA folks, we can push harness setup by a month, currently the QA Team has access to Harshil’s machine and both Sankalan and Sourav are using it for testing
Harness setup will include logging, monitoring, easy setup/teardown and a better test isolation for different scenarios using namespaces
We will need help from @dmai/@joaquin to setup the playground/harness which will include Kubernetes setup with helm as the deployment manager and other tools for monitoring
Sourav to setup a call for the above and revert on the discussion that follows
Plan:
Prepare a demo for using TestRail and present it to the team during the India all hands call so that the team is familiar with using TestRail (Owner: Sankalan/Sourav)
Segregate tests to Sankalan and Sourav and set owners for regression testing in TestRail
Reach out as required for support on regression testing from GraphQL Team as the volume of testing is high
Setup testing milestones and a timeline for reaching the milestones, @vijay to add inputs on the timeline and milestones
Ready the test harness/test playground for regression testing and other related testing
Start executing the tests latest by 5th November
4-5 weeks of testing will be done by the QA team along with Dgraph and GraphQL team
When above is green, we will put the build on Slash Staging and test the build for 1 week on Slash
We will do a partial launch to the Slash production (@gja to decide on how this launch will happen for paying and free customers)
2-3 weeks live user testing on Slash production and we will be ready to release the 20.11 build
Bug reporting will be done on test failure and will be reported on Jira, it will then be assigned to the respective developers. Once bug fixes are pushed to the release branch, the QA team will verify the bug fixes and run regression testing to verify breakages if any
Please provide feedback to improve this and also correct us if we have missed something or over-calculated the timelines.
Meeting notes from Test Case Review Meeting with Ibrahim and team:
Test run by Ibrahim’s team will be done on the 1 Tb dataset. Regression tests documented in TestRail will be executed by the QA Team. Using the 1 Tb dataset for regression testing will take too long to finish.
Our users would be running a 3 alpha 3 zero configuration in production so we need to run our tests on a similar setup. Need to decide on test harness to run these tests.
We need to run regression tests will cluster encryption ON and OFF.
Add various tests on Snapshot transfer and shard movement
Improve test cases cumulated in TestRail and get them reviewed by Daniel, Joaquin and Omar
Discussion topics for tomorrow’s call 20.11 Test Plan:
Discuss about QA Harness (test bed) for running a 3 alpha 3 zero cluster to mimic production usage, exposing metrics and setting up monitoring etc. Running tests on Google Kubernetes Engine would make this easy for the time being until we find a cheaper solution in house
Discuss about QA Team and Dev team running same tests and we can streamline that to reduce repeatability
Discuss about dataset sizes for different stages of testing, running regression tests on 1 Tb dataset would take too long for completion before release date. Should we use a lower size dataset to run the regression tests
Discuss if we want to run 1 Tb data load test for Mac OS, at the current state, we will fall short of resources for testing this
Here are a few things we are going to focus on during the 20.11 testing:
Ibrahim’s team will work on high load testing with 1 tb datasets, they will then use the directories to run more tests on it
Regression tests will be run by QA team and members of the GraphQL team where tests will be run on a smaller dataset (21 million perhaps) for different platforms like Ubuntu and MacOS
A production level test setup will be created to replicate our users’ setup and different parts of regression tests will be run on the setup (mostly priority 0 tests)
Production level tests will be run on a 6 node cluster deployed on Kubernetes, each alpha will be given 16/32 gb memory (as per availability) when running these tests and we will use a medium to large dataset to run these tests
Chaos and scale tests like Jepsen, Bank and Flock will be run by Ibrahim’s team
We are setting up a test bed that resembles our users’ production setup so that we can mimic what users do in production. Let me know if there are any questions or feedback on the above.