[Deepsea-users] Pull-requests and automated testing

[Jan Fajerski]

On Fri, Dec 02, 2016 at 12:00:52PM +0000, Ricardo Dias wrote:
>Hi folks,
>
>Now that DeepSea (DS) is starting to get public attention and will start
>getting more and more contributions, it is very important to "agile-ize"
>the reviewing process of pull-requests submissions.
>
>Every time a pull-request (PR) is submitted, besides reviewing the code,
>the assigned reviewer must also deploy DS and test it manually to check
>if nothing stops working due to the PR changes.
>While this process has worked until now, it will not scale when we start
>to get several PR submissions.
>
>A good starting point to solve this scaling problem, is to rely on
>GitHub's "hooks", and PR's "status checks", to run a set of automated
>tests that will ensure that everything is working correctly and save the
>reviewers time from doing the manual testing. This is similar to what
>the Ceph project in github is doing with PR submissions.
>
>Of course all of this assumes that we have tests to run, and we should
>start planning how unit tests should be developed for DS, based on Salt
>unit tests. But for the start, I think we should have some way to pick
>the changes from the submitted PR, spawn some virtual machines and
>deploy a Ceph cluster, and respective services with DS+changes, and
>check that everything is working correctly.
I'm not sure if unit tests will be sufficient. salt is hard to test and to my 
knowledge we can only test runners with unit tests. The real question is how we 
will test the rest.
>
>A side effect of having status checks activated for the PRs is that we
>protect the master branch from "force" pushes and from deletion, which
>is also nice.
>
>Now what do we need to actually put this plan to work?
>Here is a checklist (maybe incomplete):
>* public jenkins instance
>* computing resources for testing DS
>* Scripts to glue everything with github
Being on my way back from the QA/Testing/Automation meeting in Prag, I would 
strongly suggest we look into integrating DeepSea testing into Teuthology. This 
has various advantages and shouldn't be too hard. Ceph deploy is tested in 
Teuthology too...so why not DS.
Imho the tests to run (besides unit tests for the runners) would simply exercise 
DeepSeas functionality.
>
>If anyone has suggestions, see any problem with this plan, or knows about
>other components/software to achieve the same plan, please go forth and
>present it.
>
>I truly believe that this is an important matter to allow a faster
>development cycle for DS.
Absolutely agreed...we need to get this on the road.
>
>Thanks,
>Ricardo Dias
>
>_______________________________________________
>Deepsea-users mailing list
>Deepsea-users at lists.suse.com
>http://lists.suse.com/mailman/listinfo/deepsea-users

-- 
Jan Fajerski
Engineer Enterprise Storage
SUSE Linux GmbH
jfajerski at suse.com