[Deepsea-users] Stage 2 error on Azure
Tim Serong
tserong at suse.com
Sun Sep 23 23:23:58 MDT 2018
Have a look in /etc/hosts, and make sure the system hostname does *not*
appear on either the IPv4 or IPv6 localhost lines. That was the problem
last time I saw this issue (presumably something was resolving the
hostname, getting 127.0.0.1 or ::1 back, then doing a reverse lookup on
that address, and finishing up with localhost).
Regards,
Tim
On 09/23/2018 03:35 PM, David Byte wrote:
> I know you are going to love this, but I suspect you have stuff cached that needs to be cleaned up. The issues you are seeing are exactly things where I use my cleanit.sh script to wipe it all out and allow me to start over.
>
>
> David Byte
> Sr. Technology Strategist
> SCE Enterprise Linux
> SCE Enterprise Storage
> Alliances and SUSE Embedded
> dbyte at suse.com
> 918.528.4422
>
> On 9/22/18, 11:26 PM, "deepsea-users-bounces at lists.suse.com on behalf of Kevin Ayres" <deepsea-users-bounces at lists.suse.com on behalf of kevin.ayres at suse.com> wrote:
>
> Thanks Eric. No joy, I'm missing something here..
>
> salt:~ # salt-key -L
> Accepted Keys:
> igw1
> mon1
> mon2
> mon3
> osd1
> osd2
> osd3
> salt
>
> salt:~ # ls -a /srv/salt/ceph/mgr/cache
> . ..
>
> salt:~ # salt-key -d salt
> The following keys are going to be deleted:
> Accepted Keys:
> salt
>
> salt:~ # vi /etc/salt/minion_id
>
> salt:~ # salt-key -A
> The following keys are going to be accepted:
> Unaccepted Keys:
> salt
> Proceed? [n/Y] y
> Key for minion salt accepted.
>
> salt:~ # salt 'admin*' state.apply ceph.mgr.key
> No minions matched the target. No command was sent, no jid was assigned.
> ERROR: No return received
>
> salt:~ # salt 'salt*' state.apply ceph.mgr.key
> salt:
> Data failed to compile:
> ----------
> Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
> ERROR: Minions returned with non-zero exit code
>
> salt:~ # salt-run state.orch ceph.stage.configure
> deepsea_minions : valid
> yaml_syntax : valid
> profiles_populated : valid
> public network : 172.19.20.0/24
> cluster network : 172.19.20.0/24
> [ERROR ] Run failed on minions: salt
> Failures:
> salt:
> Data failed to compile:
> ----------
> Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
>
> salt_master:
> Name: push.proposal - Function: salt.runner - Result: Changed Started: - 05:22:09.015323 Duration: 1330.864 ms
> Name: refresh_pillar1 - Function: salt.state - Result: Changed Started: - 05:22:10.346301 Duration: 759.065 ms
> Name: advise.networks - Function: salt.runner - Result: Clean Started: - 05:22:11.105561 Duration: 1699.562 ms
> Name: admin key - Function: salt.state - Result: Clean Started: - 05:22:12.805243 Duration: 391.792 ms
> Name: mon key - Function: salt.state - Result: Changed Started: - 05:22:13.197149 Duration: 381.892 ms
> ----------
> ID: mgr key
> Function: salt.state
> Result: False
> Comment: Run failed on minions: salt
> Failures:
> salt:
> Data failed to compile:
> ----------
> Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
> Started: 05:22:13.579153
> Duration: 2510.463 ms
> Changes:
>
> Summary for salt_master
> ------------
> Succeeded: 5 (changed=3)
> Failed: 1
> ------------
> Total states run: 6
> Total run time: 7.074 s
>
> ~ Kevin
>
> On 9/21/18, 11:56 AM, "deepsea-users-bounces at lists.suse.com on behalf of Eric Jackson" <deepsea-users-bounces at lists.suse.com on behalf of ejackson at suse.com> wrote:
>
> Hi Kevin,
> Check your minion names. Try 'salt-key -L'. The reason for the "Conflicting
> ID" is that Salt will unroll a Jinja loop. For example, if you have three
> minions assigned the mgr role, then the preprocessing of /srv/salt/ceph/mgr/
> key/default.sls will create three separate stanzas. Yaml requires unique
> identifiers.
> The key file names you expected in /srv/salt/ceph/mgr/cache would be
> minion1.keyring, minion2.keyring and minon3.keyring. However, you are getting
> localhost.keyring. So, you have at least two and likely three minions all
> replying to the Salt master that they are "localhost".
>
> Check on each of your salt minions the value in /etc/salt/minion_id. If
> that is incorrect (and says "localhost"), delete the minion from the Salt
> master, correct the minion_id file, restart the salt minion and then accept the
> key on the Salt master. The commands would be
>
> admin# salt-key -d ID
> minion1# vi /etc/salt/minion_id
> minion1#
> admin# salt-key -A
>
> Once that is resolved, you can run just the ceph.mgr.key step to verify.
>
> admin# salt 'admin*' state.apply ceph.mgr.key
>
> When that works, try Stage 2 again.
>
> Eric
>
> On Friday, September 21, 2018 2:18:06 PM EDT Kevin Ayres wrote:
> > I redeployed this SES 5 cluster and ripped off all of the Azure naming junk
> > and moved a local DNS server on the Adm node. So, DNS, NTP, resolv.conf and
> > hosts and .ssh/ files appear to be correct throughout. Stages 0, 1
> > complete. Stage 2 fails.
>
> > I’m troubleshooting this stage 2 error. Something to do with keyring caching
> > and possibly the manager role running on admin node? I’ve restarted
> > services, node, etc. It seems to be minion issue on the salt master (adm)
> > node. Any guidance is appreciated. Googling madly..
>
> >
> > salt:
> >
> > Data failed to compile:
> >
> > ----------
> >
> > Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID
> > '/srv/salt/ceph/mgr/cache/localhost.keyring'
>
> > salt:~ # ls -a /srv/salt/ceph/mgr/cache/
> > . ..
> > No such folder..
> >
> > salt:~ # cat /srv/pillar/ceph/proposals/policy.cfg
> > ...
> > role-mgr/cluster/mon*.sls
> >
> > salt:~ # cat /srv/pillar/ceph/proposals/role-mgr/cluster/salt.sls
> > roles:
> > - mgr
> >
> > Thank you!
> > ~ Kevin
> >
> > Complete sage 2 output:
> >
> > salt:~ # salt-run state.orch ceph.stage.configure
> > deepsea_minions : valid
> > yaml_syntax : valid
> > profiles_populated : valid
> > public network : 172.19.20.0/24
> > cluster network : 172.19.20.0/24
> > [ERROR ] Run failed on minions: salt
> > Failures:
> > salt:
> > Data failed to compile:
> > ----------
> > Rendering SLS 'base:ceph.mgr.key.default' failed:
> > Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
>
> > salt_master:
> > Name: push.proposal - Function: salt.runner - Result: Changed
> > Started: - 17:04:46.183848 Duration: 1392.712 ms
> Name: refresh_pillar1 -
> > Function: salt.state - Result: Changed Started: - 17:04:47.576691 Duration:
> > 784.477 ms Name: advise.networks - Function: salt.runner - Result: Clean
> > Started: - 17:04:48.361300 Duration: 1766.973 ms Name: admin key -
> > Function: salt.state - Result: Clean Started: - 17:04:50.128391 Duration:
> > 404.333 ms Name: mon key - Function: salt.state - Result: Changed Started:
> > - 17:04:50.532926 Duration: 563.858 ms ----------
> > ID: mgr key
> > Function: salt.state
> > Result: False
> > Comment: Run failed on minions: salt
> > Failures:
> > salt:
> > Data failed to compile:
> > ----------
> > Rendering SLS 'base:ceph.mgr.key.default'
> > failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
> > Started: 17:04:51.096948
> > Duration: 2690.64 ms
> > Changes:
> >
> > Summary for salt_master
> > ------------
> > Succeeded: 5 (changed=3)
> > Failed: 1
> > ------------
> > Total states run: 6
> > Total run time: 7.603 s
> >
> > salt:~ # service salt-minion status
> > * salt-minion.service - The Salt Minion
> > Loaded: loaded (/usr/lib/systemd/system/salt-minion.service;
> > enabled; vendor preset: disabled)
> Active: active (running) since Thu
> > 2018-09-20 23:53:42 UTC; 17h ago Main PID: 13456 (salt-minion)
> > Tasks: 6 (limit: 512)
> > CGroup: /system.slice/salt-minion.service
> >
> > |-13456 /usr/bin/python /usr/bin/salt-minion
> > |-13462 /usr/bin/python /usr/bin/salt-minion
> >
> > `-13465 /usr/bin/python /usr/bin/salt-minion
> >
> > Sep 21 13:03:37 salt salt-minion[13456]: [ERROR ] Exception
> > occurred while handling stream: [Errno 0] Success
> Sep 21 13:05:57 salt
> > salt-minion[13456]: [ERROR ] Exception occurred while handling stream:
> > [Errno 0] Success Sep 21 13:50:19 salt salt-minion[13456]: [ERROR ]
> > Exception occurred while handling stream: [Errno 0] Success Sep 21 13:53:44
> > salt salt-minion[13456]: [ERROR ] Function cephimages.list in
> > mine_functions failed to execute Sep 21 14:47:32 salt salt-minion[13456]:
> > [ERROR ] Exception occurred while handling stream: [Errno 0] Success Sep
> > 21 14:53:44 salt salt-minion[13456]: [ERROR ] Function cephimages.list in
> > mine_functions failed to execute Sep 21 15:05:03 salt salt-minion[13456]:
> > [ERROR ] Exception occurred while handling stream: [Errno 0] Success Sep
> > 21 15:53:44 salt salt-minion[13456]: [ERROR ] Function cephimages.list in
> > mine_functions failed to execute Sep 21 16:53:44 salt salt-minion[13456]:
> > [ERROR ] Function cephimages.list in mine_functions failed to execute Sep
> > 21 17:04:53 salt salt-minion[13456]: [CRITICAL] Rendering SLS
> > 'base:ceph.mgr.key.default' failed: Conflicting ID
> > '/srv/salt/ceph/mgr/cache/localhost.keyring'
>
>
>
> _______________________________________________
> Deepsea-users mailing list
> Deepsea-users at lists.suse.com
> http://lists.suse.com/mailman/listinfo/deepsea-users
>
>
> _______________________________________________
> Deepsea-users mailing list
> Deepsea-users at lists.suse.com
> http://lists.suse.com/mailman/listinfo/deepsea-users
>
--
Tim Serong
Senior Clustering Engineer
SUSE
tserong at suse.com
More information about the Deepsea-users
mailing list