[Deepsea-users] Pull-requests and automated testing

[Ricardo Dias]

Hi folks,

Now that DeepSea (DS) is starting to get public attention and will start
getting more and more contributions, it is very important to "agile-ize"
the reviewing process of pull-requests submissions.

Every time a pull-request (PR) is submitted, besides reviewing the code,
the assigned reviewer must also deploy DS and test it manually to check
if nothing stops working due to the PR changes.
While this process has worked until now, it will not scale when we start
to get several PR submissions.

A good starting point to solve this scaling problem, is to rely on
GitHub's "hooks", and PR's "status checks", to run a set of automated
tests that will ensure that everything is working correctly and save the
reviewers time from doing the manual testing. This is similar to what
the Ceph project in github is doing with PR submissions.

Of course all of this assumes that we have tests to run, and we should
start planning how unit tests should be developed for DS, based on Salt
unit tests. But for the start, I think we should have some way to pick
the changes from the submitted PR, spawn some virtual machines and
deploy a Ceph cluster, and respective services with DS+changes, and
check that everything is working correctly.

A side effect of having status checks activated for the PRs is that we
protect the master branch from "force" pushes and from deletion, which
is also nice.

Now what do we need to actually put this plan to work?
Here is a checklist (maybe incomplete):
* public jenkins instance
* computing resources for testing DS
* Scripts to glue everything with github

If anyone has suggestions, see any problem with this plan, or knows about
other components/software to achieve the same plan, please go forth and
present it.

I truly believe that this is an important matter to allow a faster
development cycle for DS.

Thanks,
Ricardo Dias