[Deepsea-users] Stage 2 error on Azure

Tim Serong tserong at suse.com
Sun Sep 23 23:23:58 MDT 2018


Have a look in /etc/hosts, and make sure the system hostname does *not*
appear on either the IPv4 or IPv6 localhost lines.  That was the problem
last time I saw this issue (presumably something was resolving the
hostname, getting 127.0.0.1 or ::1 back, then doing a reverse lookup on
that address, and finishing up with localhost).

Regards,

Tim

On 09/23/2018 03:35 PM, David Byte wrote:
> I know you are going to love this, but I suspect you have stuff cached that needs to be cleaned up. The issues you are seeing are exactly things where I use my cleanit.sh script to wipe it all out and allow me to start over.
> 
> 
> David Byte
> Sr. Technology Strategist
> SCE Enterprise Linux 
> SCE Enterprise Storage
> Alliances and SUSE Embedded
> dbyte at suse.com
> 918.528.4422
> 
> On 9/22/18, 11:26 PM, "deepsea-users-bounces at lists.suse.com on behalf of Kevin Ayres" <deepsea-users-bounces at lists.suse.com on behalf of kevin.ayres at suse.com> wrote:
> 
>     Thanks Eric. No joy, I'm missing something here.. 
>     
>     salt:~ # salt-key -L
>     Accepted Keys:
>     igw1
>     mon1
>     mon2
>     mon3
>     osd1
>     osd2
>     osd3
>     salt
>     
>     salt:~ # ls -a /srv/salt/ceph/mgr/cache
>     .  ..
>     
>     salt:~ # salt-key -d salt
>     The following keys are going to be deleted:
>     Accepted Keys:
>     salt
>     
>     salt:~ # vi /etc/salt/minion_id
>     
>     salt:~ # salt-key -A
>     The following keys are going to be accepted:
>     Unaccepted Keys:
>     salt
>     Proceed? [n/Y] y
>     Key for minion salt accepted.
>     
>     salt:~ # salt 'admin*' state.apply ceph.mgr.key
>     No minions matched the target. No command was sent, no jid was assigned.
>     ERROR: No return received
>     
>     salt:~ # salt 'salt*' state.apply ceph.mgr.key
>     salt:
>         Data failed to compile:
>     ----------
>         Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
>     ERROR: Minions returned with non-zero exit code
>     
>     salt:~ # salt-run state.orch ceph.stage.configure
>     deepsea_minions          : valid
>     yaml_syntax              : valid
>     profiles_populated       : valid
>     public network           : 172.19.20.0/24
>     cluster network          : 172.19.20.0/24
>     [ERROR   ] Run failed on minions: salt
>     Failures:
>         salt:
>             Data failed to compile:
>         ----------
>             Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
>     
>     salt_master:
>       Name: push.proposal - Function: salt.runner - Result: Changed Started: - 05:22:09.015323 Duration: 1330.864 ms
>       Name: refresh_pillar1 - Function: salt.state - Result: Changed Started: - 05:22:10.346301 Duration: 759.065 ms
>       Name: advise.networks - Function: salt.runner - Result: Clean Started: - 05:22:11.105561 Duration: 1699.562 ms
>       Name: admin key - Function: salt.state - Result: Clean Started: - 05:22:12.805243 Duration: 391.792 ms
>       Name: mon key - Function: salt.state - Result: Changed Started: - 05:22:13.197149 Duration: 381.892 ms
>     ----------
>               ID: mgr key
>         Function: salt.state
>           Result: False
>          Comment: Run failed on minions: salt
>                   Failures:
>                       salt:
>                           Data failed to compile:
>                       ----------
>                           Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
>          Started: 05:22:13.579153
>         Duration: 2510.463 ms
>          Changes:   
>     
>     Summary for salt_master
>     ------------
>     Succeeded: 5 (changed=3)
>     Failed:    1
>     ------------
>     Total states run:     6
>     Total run time:   7.074 s
>     
>     ~ Kevin
>     
>     On 9/21/18, 11:56 AM, "deepsea-users-bounces at lists.suse.com on behalf of Eric Jackson" <deepsea-users-bounces at lists.suse.com on behalf of ejackson at suse.com> wrote:
>     
>         Hi Kevin,
>           Check your minion names.  Try 'salt-key -L'.  The reason for the "Conflicting 
>         ID" is that Salt will unroll a Jinja loop.  For example, if you have three 
>         minions assigned the mgr role, then the preprocessing of /srv/salt/ceph/mgr/
>         key/default.sls will create three separate stanzas.  Yaml requires unique 
>         identifiers.  
>           The key file names you expected in /srv/salt/ceph/mgr/cache would be 
>         minion1.keyring, minion2.keyring and minon3.keyring.  However, you are getting 
>         localhost.keyring.  So, you have at least two and likely three minions all 
>         replying to the Salt master that they are "localhost".
>         
>           Check on each of your salt minions the value in /etc/salt/minion_id.  If 
>         that is incorrect (and says "localhost"), delete the minion from the Salt 
>         master, correct the minion_id file, restart the salt minion and then accept the 
>         key on the Salt master.  The commands would be
>         
>         admin# salt-key -d ID
>         minion1# vi /etc/salt/minion_id
>         minion1#  
>         admin# salt-key -A
>         
>           Once that is resolved, you can run just the ceph.mgr.key step to verify.
>         
>         admin# salt 'admin*' state.apply ceph.mgr.key
>         
>           When that works, try Stage 2 again.
>         
>         Eric
>         
>         On Friday, September 21, 2018 2:18:06 PM EDT Kevin Ayres wrote:
>         > I redeployed this SES 5 cluster and ripped off all of the Azure naming junk
>         > and moved a local DNS server on the Adm node. So, DNS, NTP, resolv.conf and
>         > hosts and .ssh/ files appear to be correct throughout. Stages 0, 1
>         > complete. Stage 2 fails.
>          
>         > I’m troubleshooting this stage 2 error. Something to do with keyring caching
>         > and possibly the manager role running on admin node? I’ve restarted
>         > services, node, etc.  It seems to be minion issue on the salt master (adm)
>         > node. Any guidance is appreciated. Googling madly..
>          
>         > 
>         >     salt:
>         > 
>         >         Data failed to compile:
>         > 
>         >     ----------
>         > 
>         >         Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID
>         > '/srv/salt/ceph/mgr/cache/localhost.keyring'
>          
>         > salt:~ # ls -a /srv/salt/ceph/mgr/cache/
>         > .  ..
>         > No such folder..
>         > 
>         > salt:~ # cat /srv/pillar/ceph/proposals/policy.cfg
>         > ...
>         > role-mgr/cluster/mon*.sls
>         > 
>         > salt:~ # cat /srv/pillar/ceph/proposals/role-mgr/cluster/salt.sls
>         > roles:
>         > - mgr
>         > 
>         > Thank you!
>         > ~ Kevin
>         > 
>         > Complete sage 2 output:
>         > 
>         >         salt:~ # salt-run state.orch ceph.stage.configure
>         >         deepsea_minions          : valid
>         >         yaml_syntax              : valid
>         >         profiles_populated       : valid
>         >         public network           : 172.19.20.0/24
>         >         cluster network          : 172.19.20.0/24
>         >         [ERROR   ] Run failed on minions: salt
>         >         Failures:
>         >             salt:
>         >                 Data failed to compile:
>         >             ----------
>         >                 Rendering SLS 'base:ceph.mgr.key.default' failed:
>         > Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
>          
>         >         salt_master:
>         >           Name: push.proposal - Function: salt.runner - Result: Changed
>         > Started: - 17:04:46.183848 Duration: 1392.712 ms
>          Name: refresh_pillar1 -
>         > Function: salt.state - Result: Changed Started: - 17:04:47.576691 Duration:
>         > 784.477 ms Name: advise.networks - Function: salt.runner - Result: Clean
>         > Started: - 17:04:48.361300 Duration: 1766.973 ms Name: admin key -
>         > Function: salt.state - Result: Clean Started: - 17:04:50.128391 Duration:
>         > 404.333 ms Name: mon key - Function: salt.state - Result: Changed Started:
>         > - 17:04:50.532926 Duration: 563.858 ms ----------
>         >                   ID: mgr key
>         >             Function: salt.state
>         >               Result: False
>         >              Comment: Run failed on minions: salt
>         >                       Failures:
>         >                           salt:
>         >                               Data failed to compile:
>         >                           ----------
>         >                               Rendering SLS 'base:ceph.mgr.key.default'
>         > failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'
>         > Started: 17:04:51.096948
>         >             Duration: 2690.64 ms
>         >              Changes:
>         > 
>         >         Summary for salt_master
>         >         ------------
>         >         Succeeded: 5 (changed=3)
>         >         Failed:    1
>         >         ------------
>         >         Total states run:     6
>         >         Total run time:   7.603 s
>         > 
>         >         salt:~ # service salt-minion status
>         >         * salt-minion.service - The Salt Minion
>         >            Loaded: loaded (/usr/lib/systemd/system/salt-minion.service;
>         > enabled; vendor preset: disabled)
>          Active: active (running) since Thu
>         > 2018-09-20 23:53:42 UTC; 17h ago Main PID: 13456 (salt-minion)
>         >             Tasks: 6 (limit: 512)
>         >            CGroup: /system.slice/salt-minion.service
>         > 
>         >                    |-13456 /usr/bin/python /usr/bin/salt-minion
>         >                    |-13462 /usr/bin/python /usr/bin/salt-minion
>         > 
>         >                    `-13465 /usr/bin/python /usr/bin/salt-minion
>         > 
>         >         Sep 21 13:03:37 salt salt-minion[13456]: [ERROR   ] Exception
>         > occurred while handling stream: [Errno 0] Success
>          Sep 21 13:05:57 salt
>         > salt-minion[13456]: [ERROR   ] Exception occurred while handling stream:
>         > [Errno 0] Success Sep 21 13:50:19 salt salt-minion[13456]: [ERROR   ]
>         > Exception occurred while handling stream: [Errno 0] Success Sep 21 13:53:44
>         > salt salt-minion[13456]: [ERROR   ] Function cephimages.list in
>         > mine_functions failed to execute Sep 21 14:47:32 salt salt-minion[13456]:
>         > [ERROR   ] Exception occurred while handling stream: [Errno 0] Success Sep
>         > 21 14:53:44 salt salt-minion[13456]: [ERROR   ] Function cephimages.list in
>         > mine_functions failed to execute Sep 21 15:05:03 salt salt-minion[13456]:
>         > [ERROR   ] Exception occurred while handling stream: [Errno 0] Success Sep
>         > 21 15:53:44 salt salt-minion[13456]: [ERROR   ] Function cephimages.list in
>         > mine_functions failed to execute Sep 21 16:53:44 salt salt-minion[13456]:
>         > [ERROR   ] Function cephimages.list in mine_functions failed to execute Sep
>         > 21 17:04:53 salt salt-minion[13456]: [CRITICAL] Rendering SLS
>         > 'base:ceph.mgr.key.default' failed: Conflicting ID
>         > '/srv/salt/ceph/mgr/cache/localhost.keyring'
>         
>         
>     
>     _______________________________________________
>     Deepsea-users mailing list
>     Deepsea-users at lists.suse.com
>     http://lists.suse.com/mailman/listinfo/deepsea-users
>     
> 
> _______________________________________________
> Deepsea-users mailing list
> Deepsea-users at lists.suse.com
> http://lists.suse.com/mailman/listinfo/deepsea-users
> 


-- 
Tim Serong
Senior Clustering Engineer
SUSE
tserong at suse.com


More information about the Deepsea-users mailing list