From susheel.gokhale2 at hpe.com  Thu Jan 18 22:00:35 2018
From: susheel.gokhale2 at hpe.com (Gokhale, Susheel)
Date: Fri, 19 Jan 2018 05:00:35 +0000
Subject: [Deepsea-users] Missing keyring files during stage 3
In-Reply-To: <5813810.ZpRKKL9bz3@fury.home>
References: <CS1PR8401MB0822352D3DC40B100DDDC963C6E90@CS1PR8401MB0822.NAMPRD84.PROD.OUTLOOK.COM>
	<5813810.ZpRKKL9bz3@fury.home>
Message-ID: <CS1PR8401MB0822C4C6BFCDE082075D9493C6EF0@CS1PR8401MB0822.NAMPRD84.PROD.OUTLOOK.COM>

Hi Eric,

Thanks for the quick debug help. 

Earlier, before posting to the list, I did try the individual steps that you suggest below.
The 1st step worked successfully, 2nd step would always fail. 
Unfortunately I did not save the debug  logs from that time.

In the meantime, I have un-installed the whole environment [deepsea/ ceph/ salt] and  repeated the install steps.
This time I did not get any errors in the stage 3 during  'ceph.admin' step to copy the keyrings. 

I followed the same steps from the SES5 Deployment Guide on both occasions. 
Now I am not sure what may have been broken before.

Moving on to setting up all the services now ....

regards
---
susheel.gokhale at hpe.com

> -----Original Message-----
> From: deepsea-users-bounces at lists.suse.com [mailto:deepsea-users-
> bounces at lists.suse.com] On Behalf Of Eric Jackson
> Sent: Thursday, 18 January 2018 7:53 PM
> To: deepsea-users at lists.suse.com
> Subject: Re: [Deepsea-users] Missing keyring files during stage 3
> 
> Hi Susheel,
> 
> 
> On Wednesday, January 17, 2018 12:54:30 AM EST Gokhale, Susheel wrote:
> > Hi,
> >
> > I am testing SES-5 setup on a VM cluster.
> > Running the latest updates from SES repos :
> >
> > deepsea-0.8+git.31.d2243db21-2.3.1.noarch
> > ceph-12.2.1+git.1510221942.af9ea5e715-2.5.1.x86_64
> > etc ....
> >
> >
> > When running stage 3 it reports errors in accessing keyring files :
> > -----
> > Starting stage: ceph.stage.3
> > Parsing ceph.stage.3 steps...
> >
> > [6/17]    ceph.admin on
> >           ses5-ceph-2.workshop.............................. ? (0.7s)
> >           ses5-ceph-3.workshop.............................. ? (0.7s)
> >           ses5-ceph-4.workshop.............................. ? (0.7s)
> >
> > [7/17]    ceph.mgr.keyring on
> >           ses5-ceph-4.workshop.............................. ? (0.5s)
> >
> > Ended stage: ceph.stage.3 succeeded=5/17 failed=2/17 time=38.1s
> >
> > Failures summary:
> >
> > ceph.admin (/srv/salt/ceph/admin):
> >   ses5-ceph-2.workshop:
> >     /etc/ceph/ceph.client.admin.keyring: Unable to manage file: none
> > of the specified sources were found
> >
> 
> This seems strange.  Here are the two steps that really happening for the admin
> keyring:
> 
> salt 'ses5*2*' state.apply ceph.admin.key salt 'ses5*2*' state.apply ceph.admin
> 
> The first uses the Jinja template in /srv/salt/ceph/admin/files/keyring.j2 and
> produces /srv/salt/ceph/admin/cache/ceph.client.admin.keyring.  Does this file
> exist?
> 
> The second step copies the generated admin keyring to all minions that need it.
> In the general case, it's only the master.
> 
> To debug this, enable debugging on the minion on ses5-ceph-2:
> 
> # vi /etc/salt/minion
> Set log_level: debug
> # systemctl restart salt-minion
> 
> Then, run the steps above and check /var/log/salt/minion.  I am expecting
> something odd like an unexplained "permission denied". (And this would be
> strange since the minion runs as root.)
> 
> Eric
> 
> > ----
> > ceph.mgr.keyring (/srv/salt/ceph/mgr/keyring):
> >   ses5-ceph-3.workshop:
> >     /var/lib/ceph/mgr/ceph-ses5-ceph-3/keyring: Unable to manage file:
> > none of the specified sources were found
> >
> > -----
> >
> > I noticed that all the keyrings were generated correctly and populated
> > on the master node  : /srv/salt/ceph/<role>/cache/*.keyring
> >
> > But they are not being pushed into the "/etc/ceph/" or
> > "/var/lib/ceph/<role>"  directories on the master node or to the
> > minions , where the stage 3 script is expecting to find them ..
> >
> > Where is this failing ??
> > How to debug it in more detail ??
> >
> > Not able to finish the setup due to stage 3 failing in setting up the
> > keyrings
> >
> > Thanks
> > ---
> > susheel.gokhale at hpe.com<mailto:susheel.gokhale at hpe.com>