[Deepsea-users] Stage 2 error on Azure
David Byte
dbyte at suse.com
Sat Sep 22 23:35:17 MDT 2018
I know you are going to love this, but I suspect you have stuff cached that needs to be cleaned up. The issues you are seeing are exactly things where I use my cleanit.sh script to wipe it all out and allow me to start over.
David Byte
Sr. Technology Strategist
SCE Enterprise Linux
SCE Enterprise Storage
Alliances and SUSE Embedded
dbyte at suse.com
918.528.4422
On 9/22/18, 11:26 PM, "deepsea-users-bounces at lists.suse.com on behalf of Kevin Ayres" <deepsea-users-bounces at lists.suse.com on behalf of kevin.ayres at suse.com> wrote:
Thanks Eric. No joy, I'm missing something here..
salt:~ # salt-key -L
Accepted Keys:
igw1
mon1
mon2
mon3
osd1
osd2
osd3
salt
salt:~ # ls -a /srv/salt/ceph/mgr/cache
. ..
salt:~ # salt-key -d salt
The following keys are going to be deleted:
Accepted Keys:
salt
salt:~ # vi /etc/salt/minion_id
salt:~ # salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
salt
Proceed? [n/Y] y
Key for minion salt accepted.
salt:~ # salt 'admin*' state.apply ceph.mgr.key
No minions matched the target. No command was sent, no jid was assigned.
ERROR: No return received
salt:~ # salt 'salt*' state.apply ceph.mgr.key
salt:
Data failed to compile:
----------
Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
ERROR: Minions returned with non-zero exit code
salt:~ # salt-run state.orch ceph.stage.configure
deepsea_minions : valid
yaml_syntax : valid
profiles_populated : valid
public network : 172.19.20.0/24
cluster network : 172.19.20.0/24
[ERROR ] Run failed on minions: salt
Failures:
salt:
Data failed to compile:
----------
Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
salt_master:
Name: push.proposal - Function: salt.runner - Result: Changed Started: - 05:22:09.015323 Duration: 1330.864 ms
Name: refresh_pillar1 - Function: salt.state - Result: Changed Started: - 05:22:10.346301 Duration: 759.065 ms
Name: advise.networks - Function: salt.runner - Result: Clean Started: - 05:22:11.105561 Duration: 1699.562 ms
Name: admin key - Function: salt.state - Result: Clean Started: - 05:22:12.805243 Duration: 391.792 ms
Name: mon key - Function: salt.state - Result: Changed Started: - 05:22:13.197149 Duration: 381.892 ms
----------
ID: mgr key
Function: salt.state
Result: False
Comment: Run failed on minions: salt
Failures:
salt:
Data failed to compile:
----------
Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
Started: 05:22:13.579153
Duration: 2510.463 ms
Changes:
Summary for salt_master
------------
Succeeded: 5 (changed=3)
Failed: 1
------------
Total states run: 6
Total run time: 7.074 s
~ Kevin
On 9/21/18, 11:56 AM, "deepsea-users-bounces at lists.suse.com on behalf of Eric Jackson" <deepsea-users-bounces at lists.suse.com on behalf of ejackson at suse.com> wrote:
Hi Kevin,
Check your minion names. Try 'salt-key -L'. The reason for the "Conflicting
ID" is that Salt will unroll a Jinja loop. For example, if you have three
minions assigned the mgr role, then the preprocessing of /srv/salt/ceph/mgr/
key/default.sls will create three separate stanzas. Yaml requires unique
identifiers.
The key file names you expected in /srv/salt/ceph/mgr/cache would be
minion1.keyring, minion2.keyring and minon3.keyring. However, you are getting
localhost.keyring. So, you have at least two and likely three minions all
replying to the Salt master that they are "localhost".
Check on each of your salt minions the value in /etc/salt/minion_id. If
that is incorrect (and says "localhost"), delete the minion from the Salt
master, correct the minion_id file, restart the salt minion and then accept the
key on the Salt master. The commands would be
admin# salt-key -d ID
minion1# vi /etc/salt/minion_id
minion1#
admin# salt-key -A
Once that is resolved, you can run just the ceph.mgr.key step to verify.
admin# salt 'admin*' state.apply ceph.mgr.key
When that works, try Stage 2 again.
Eric
On Friday, September 21, 2018 2:18:06 PM EDT Kevin Ayres wrote:
> I redeployed this SES 5 cluster and ripped off all of the Azure naming junk
> and moved a local DNS server on the Adm node. So, DNS, NTP, resolv.conf and
> hosts and .ssh/ files appear to be correct throughout. Stages 0, 1
> complete. Stage 2 fails.
> I’m troubleshooting this stage 2 error. Something to do with keyring caching
> and possibly the manager role running on admin node? I’ve restarted
> services, node, etc. It seems to be minion issue on the salt master (adm)
> node. Any guidance is appreciated. Googling madly..
>
> salt:
>
> Data failed to compile:
>
> ----------
>
> Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID
> '/srv/salt/ceph/mgr/cache/localhost.keyring'
> salt:~ # ls -a /srv/salt/ceph/mgr/cache/
> . ..
> No such folder..
>
> salt:~ # cat /srv/pillar/ceph/proposals/policy.cfg
> ...
> role-mgr/cluster/mon*.sls
>
> salt:~ # cat /srv/pillar/ceph/proposals/role-mgr/cluster/salt.sls
> roles:
> - mgr
>
> Thank you!
> ~ Kevin
>
> Complete sage 2 output:
>
> salt:~ # salt-run state.orch ceph.stage.configure
> deepsea_minions : valid
> yaml_syntax : valid
> profiles_populated : valid
> public network : 172.19.20.0/24
> cluster network : 172.19.20.0/24
> [ERROR ] Run failed on minions: salt
> Failures:
> salt:
> Data failed to compile:
> ----------
> Rendering SLS 'base:ceph.mgr.key.default' failed:
> Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
> salt_master:
> Name: push.proposal - Function: salt.runner - Result: Changed
> Started: - 17:04:46.183848 Duration: 1392.712 ms
Name: refresh_pillar1 -
> Function: salt.state - Result: Changed Started: - 17:04:47.576691 Duration:
> 784.477 ms Name: advise.networks - Function: salt.runner - Result: Clean
> Started: - 17:04:48.361300 Duration: 1766.973 ms Name: admin key -
> Function: salt.state - Result: Clean Started: - 17:04:50.128391 Duration:
> 404.333 ms Name: mon key - Function: salt.state - Result: Changed Started:
> - 17:04:50.532926 Duration: 563.858 ms ----------
> ID: mgr key
> Function: salt.state
> Result: False
> Comment: Run failed on minions: salt
> Failures:
> salt:
> Data failed to compile:
> ----------
> Rendering SLS 'base:ceph.mgr.key.default'
> failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
> Started: 17:04:51.096948
> Duration: 2690.64 ms
> Changes:
>
> Summary for salt_master
> ------------
> Succeeded: 5 (changed=3)
> Failed: 1
> ------------
> Total states run: 6
> Total run time: 7.603 s
>
> salt:~ # service salt-minion status
> * salt-minion.service - The Salt Minion
> Loaded: loaded (/usr/lib/systemd/system/salt-minion.service;
> enabled; vendor preset: disabled)
Active: active (running) since Thu
> 2018-09-20 23:53:42 UTC; 17h ago Main PID: 13456 (salt-minion)
> Tasks: 6 (limit: 512)
> CGroup: /system.slice/salt-minion.service
>
> |-13456 /usr/bin/python /usr/bin/salt-minion
> |-13462 /usr/bin/python /usr/bin/salt-minion
>
> `-13465 /usr/bin/python /usr/bin/salt-minion
>
> Sep 21 13:03:37 salt salt-minion[13456]: [ERROR ] Exception
> occurred while handling stream: [Errno 0] Success
Sep 21 13:05:57 salt
> salt-minion[13456]: [ERROR ] Exception occurred while handling stream:
> [Errno 0] Success Sep 21 13:50:19 salt salt-minion[13456]: [ERROR ]
> Exception occurred while handling stream: [Errno 0] Success Sep 21 13:53:44
> salt salt-minion[13456]: [ERROR ] Function cephimages.list in
> mine_functions failed to execute Sep 21 14:47:32 salt salt-minion[13456]:
> [ERROR ] Exception occurred while handling stream: [Errno 0] Success Sep
> 21 14:53:44 salt salt-minion[13456]: [ERROR ] Function cephimages.list in
> mine_functions failed to execute Sep 21 15:05:03 salt salt-minion[13456]:
> [ERROR ] Exception occurred while handling stream: [Errno 0] Success Sep
> 21 15:53:44 salt salt-minion[13456]: [ERROR ] Function cephimages.list in
> mine_functions failed to execute Sep 21 16:53:44 salt salt-minion[13456]:
> [ERROR ] Function cephimages.list in mine_functions failed to execute Sep
> 21 17:04:53 salt salt-minion[13456]: [CRITICAL] Rendering SLS
> 'base:ceph.mgr.key.default' failed: Conflicting ID
> '/srv/salt/ceph/mgr/cache/localhost.keyring'
_______________________________________________
Deepsea-users mailing list
Deepsea-users at lists.suse.com
http://lists.suse.com/mailman/listinfo/deepsea-users
More information about the Deepsea-users
mailing list