[Deepsea-users] Pull-requests and automated testing
jfajerski at suse.com
Fri Dec 2 09:11:50 MST 2016
On Fri, Dec 02, 2016 at 12:00:52PM +0000, Ricardo Dias wrote:
>Now that DeepSea (DS) is starting to get public attention and will start
>getting more and more contributions, it is very important to "agile-ize"
>the reviewing process of pull-requests submissions.
>Every time a pull-request (PR) is submitted, besides reviewing the code,
>the assigned reviewer must also deploy DS and test it manually to check
>if nothing stops working due to the PR changes.
>While this process has worked until now, it will not scale when we start
>to get several PR submissions.
>A good starting point to solve this scaling problem, is to rely on
>GitHub's "hooks", and PR's "status checks", to run a set of automated
>tests that will ensure that everything is working correctly and save the
>reviewers time from doing the manual testing. This is similar to what
>the Ceph project in github is doing with PR submissions.
>Of course all of this assumes that we have tests to run, and we should
>start planning how unit tests should be developed for DS, based on Salt
>unit tests. But for the start, I think we should have some way to pick
>the changes from the submitted PR, spawn some virtual machines and
>deploy a Ceph cluster, and respective services with DS+changes, and
>check that everything is working correctly.
I'm not sure if unit tests will be sufficient. salt is hard to test and to my
knowledge we can only test runners with unit tests. The real question is how we
will test the rest.
>A side effect of having status checks activated for the PRs is that we
>protect the master branch from "force" pushes and from deletion, which
>is also nice.
>Now what do we need to actually put this plan to work?
>Here is a checklist (maybe incomplete):
>* public jenkins instance
>* computing resources for testing DS
>* Scripts to glue everything with github
Being on my way back from the QA/Testing/Automation meeting in Prag, I would
strongly suggest we look into integrating DeepSea testing into Teuthology. This
has various advantages and shouldn't be too hard. Ceph deploy is tested in
Teuthology too...so why not DS.
Imho the tests to run (besides unit tests for the runners) would simply exercise
>If anyone has suggestions, see any problem with this plan, or knows about
>other components/software to achieve the same plan, please go forth and
>I truly believe that this is an important matter to allow a faster
>development cycle for DS.
Absolutely agreed...we need to get this on the road.
>Deepsea-users mailing list
>Deepsea-users at lists.suse.com
Engineer Enterprise Storage
SUSE Linux GmbH
jfajerski at suse.com
More information about the Deepsea-users