[Deepsea-users] Pull-requests and automated testing

[Jan Fajerski]

On Fri, Dec 02, 2016 at 04:32:30PM +0000, Ricardo Dias wrote:
>On 2016-12-02 17:11:50, Jan Fajerski wrote:
>> On Fri, Dec 02, 2016 at 12:00:52PM +0000, Ricardo Dias wrote:
>> > Of course all of this assumes that we have tests to run, and we should
>> > start planning how unit tests should be developed for DS, based on Salt
>> > unit tests. But for the start, I think we should have some way to pick
>> > the changes from the submitted PR, spawn some virtual machines and
>> > deploy a Ceph cluster, and respective services with DS+changes, and
>> > check that everything is working correctly.
>> I'm not sure if unit tests will be sufficient. salt is hard to test and to
>> my knowledge we can only test runners with unit tests. The real question is
>> how we will test the rest.
>
>As a first step, I will be happy to just deploy a Ceph cluster using the PR
>branch and check for HEALTH_OK.
>Also, we can check if each stage has succeeded.
Yeah same here. I'm also looking to somewhat automate this process with vagrant.
>
>> >
>> > A side effect of having status checks activated for the PRs is that we
>> > protect the master branch from "force" pushes and from deletion, which
>> > is also nice.
>> >
>> > Now what do we need to actually put this plan to work?
>> > Here is a checklist (maybe incomplete):
>> > * public jenkins instance
>> > * computing resources for testing DS
>> > * Scripts to glue everything with github
>> Being on my way back from the QA/Testing/Automation meeting in Prag, I would
>> strongly suggest we look into integrating DeepSea testing into Teuthology.
>> This has various advantages and shouldn't be too hard. Ceph deploy is tested
>> in Teuthology too...so why not DS.
>> Imho the tests to run (besides unit tests for the runners) would simply
>> exercise DeepSeas functionality.
>
>Yes, testing DS in teuthology is an important goal, which I think it's
>still a bit far away from being reached. There are two ways of testing
>DS within teuthology in my opinion: a) swap ceph-ansible for DS to
>deploy the cluster as configured in each test suite; b) test DS as a
>single suite test. Either approaches will require significant work,
>specially a).
We probably want to have both really. From what I understood from Nathans 
explanations re. Teuthology, it doesn't actually use Teuthology for cluster 
deployment. There is a ceph-cm-ansible component that takes care of provisioning 
the nodes, but does not actually deploy it. Is ceph-ansible also used?
>
>But I think using teuthology to test DeepSea in the context of a GitHub
>PR's status check is not an adequate solution. We need something to be
>run quickly, no more than 30 minutes in my opinion.
Would this be impossible with Teuthology? Can't we just run our DS specific test 
suite? If that gets so extensive, that it takes so long we can use Teuthologies 
approach of partitioning suites and run a partition at a time.
>
>> >
>> > _______________________________________________
>> > Deepsea-users mailing list
>> > Deepsea-users at lists.suse.com
>> > http://lists.suse.com/mailman/listinfo/deepsea-users
>>
>> --
>> Jan Fajerski
>> Engineer Enterprise Storage
>> SUSE Linux GmbH
>> jfajerski at suse.com
>> _______________________________________________
>> Deepsea-users mailing list
>> Deepsea-users at lists.suse.com
>> http://lists.suse.com/mailman/listinfo/deepsea-users
>_______________________________________________
>Deepsea-users mailing list
>Deepsea-users at lists.suse.com
>http://lists.suse.com/mailman/listinfo/deepsea-users