<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 2 6 3 5 4 5 2 3 4;}
@font-face
{font-family:Monaco;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Courier New";
color:windowtext;
font-weight:normal;
font-style:normal;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
background:#0C0503;
font-size:9.0pt;
font-family:Monaco;
color:#28F6FF;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.s2
{mso-style-name:s2;}
span.s1
{mso-style-name:s1;
color:#28F6FF;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0in;
margin-bottom:.0001pt;
background:#0C0503;
font-size:9.0pt;
font-family:Monaco;
color:#F4F4F4;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">I redeployed this SES 5 cluster and ripped off all of the Azure naming junk and moved a local DNS server on the Adm node. So, DNS, NTP, resolv.conf and hosts and .ssh/ files appear
to be correct throughout. Stages 0, 1 complete. Stage 2 fails. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">I’m troubleshooting this stage 2 error. Something to do with keyring caching and possibly the manager role running on admin node? I’ve restarted services, node, etc. It seems to
be minion issue on the salt master (adm) node. Any guidance is appreciated. Googling madly..
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="p1"><span class="apple-converted-space"><span style="color:#F4F4F4">
</span></span><span class="s2">salt:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"><span style="color:#F4F4F4">
</span></span><span class="s2">Data failed to compile:</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"><span style="color:#F4F4F4">
</span></span><span class="s2">----------</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s2">Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#28F6FF">salt:~ #</span><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4"> ls -a /srv/salt/ceph/mgr/cache/<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">. ..<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">No such folder..
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#28F6FF">salt:~ #</span><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4"> cat /srv/pillar/ceph/proposals/policy.cfg<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">...<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">role-mgr/cluster/mon*.sls<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#28F6FF">salt:~ #</span><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4"> cat /srv/pillar/ceph/proposals/role-mgr/cluster/salt.sls<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">roles:<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">- mgr<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Thank you! <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">~ Kevin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Complete sage 2 output:
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"> salt:~ # salt-run state.orch ceph.stage.configure<o:p></o:p></p>
<p class="MsoNormal"> deepsea_minions : valid<o:p></o:p></p>
<p class="MsoNormal"> yaml_syntax : valid<o:p></o:p></p>
<p class="MsoNormal"> profiles_populated : valid<o:p></o:p></p>
<p class="MsoNormal"> public network : 172.19.20.0/24<o:p></o:p></p>
<p class="MsoNormal"> cluster network : 172.19.20.0/24<o:p></o:p></p>
<p class="MsoNormal"><span style="color:red"> [ERROR ] Run failed on minions: salt<o:p></o:p></span></p>
<p class="MsoNormal"> Failures:<o:p></o:p></p>
<p class="MsoNormal"> salt:<o:p></o:p></p>
<p class="MsoNormal"> Data failed to compile:<o:p></o:p></p>
<p class="MsoNormal"> ----------<o:p></o:p></p>
<p class="MsoNormal"> <span style="color:red">Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> salt_master:<o:p></o:p></p>
<p class="MsoNormal"> Name: push.proposal - Function: salt.runner - Result: Changed Started: - 17:04:46.183848 Duration: 1392.712 ms<o:p></o:p></p>
<p class="MsoNormal"> Name: refresh_pillar1 - Function: salt.state - Result: Changed Started: - 17:04:47.576691 Duration: 784.477 ms<o:p></o:p></p>
<p class="MsoNormal"> Name: advise.networks - Function: salt.runner - Result: Clean Started: - 17:04:48.361300 Duration: 1766.973 ms<o:p></o:p></p>
<p class="MsoNormal"> Name: admin key - Function: salt.state - Result: Clean Started: - 17:04:50.128391 Duration: 404.333 ms<o:p></o:p></p>
<p class="MsoNormal"> Name: mon key - Function: salt.state - Result: Changed Started: - 17:04:50.532926 Duration: 563.858 ms<o:p></o:p></p>
<p class="MsoNormal"> ----------<o:p></o:p></p>
<p class="MsoNormal"> ID: mgr key<o:p></o:p></p>
<p class="MsoNormal"> Function: salt.state<o:p></o:p></p>
<p class="MsoNormal"> Result: False<o:p></o:p></p>
<p class="MsoNormal"> Comment: Run failed on minions: salt<o:p></o:p></p>
<p class="MsoNormal"> Failures:<o:p></o:p></p>
<p class="MsoNormal"> salt:<o:p></o:p></p>
<p class="MsoNormal"> Data failed to compile:<o:p></o:p></p>
<p class="MsoNormal"> ----------<o:p></o:p></p>
<p class="MsoNormal"> <span style="color:red">Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'<o:p></o:p></span></p>
<p class="MsoNormal"> Started: 17:04:51.096948<o:p></o:p></p>
<p class="MsoNormal"> Duration: 2690.64 ms<o:p></o:p></p>
<p class="MsoNormal"> Changes: <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Summary for salt_master<o:p></o:p></p>
<p class="MsoNormal"> ------------<o:p></o:p></p>
<p class="MsoNormal"> Succeeded: 5 (changed=3)<o:p></o:p></p>
<p class="MsoNormal"> Failed: 1<o:p></o:p></p>
<p class="MsoNormal"> ------------<o:p></o:p></p>
<p class="MsoNormal"> Total states run: 6<o:p></o:p></p>
<p class="MsoNormal"> Total run time: 7.603 s<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> salt:~ # service salt-minion status<o:p></o:p></p>
<p class="MsoNormal"> * salt-minion.service - The Salt Minion<o:p></o:p></p>
<p class="MsoNormal"> Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled)<o:p></o:p></p>
<p class="MsoNormal"> Active: active (running) since Thu 2018-09-20 23:53:42 UTC; 17h ago<o:p></o:p></p>
<p class="MsoNormal"> Main PID: 13456 (salt-minion)<o:p></o:p></p>
<p class="MsoNormal"> Tasks: 6 (limit: 512)<o:p></o:p></p>
<p class="MsoNormal"> CGroup: /system.slice/salt-minion.service<o:p></o:p></p>
<p class="MsoNormal"> |-13456 /usr/bin/python /usr/bin/salt-minion<o:p></o:p></p>
<p class="MsoNormal"> |-13462 /usr/bin/python /usr/bin/salt-minion<o:p></o:p></p>
<p class="MsoNormal"> `-13465 /usr/bin/python /usr/bin/salt-minion<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Sep 21 13:03:37 salt salt-minion[13456]: [ERROR ] Exception occurred while handling stream: [Errno 0] Success<o:p></o:p></p>
<p class="MsoNormal"> Sep 21 13:05:57 salt salt-minion[13456]: [ERROR ] Exception occurred while handling stream: [Errno 0] Success<o:p></o:p></p>
<p class="MsoNormal"> Sep 21 13:50:19 salt salt-minion[13456]: [ERROR ] Exception occurred while handling stream: [Errno 0] Success<o:p></o:p></p>
<p class="MsoNormal"> Sep 21 13:53:44 salt salt-minion[13456]: [ERROR ] <span style="color:red">
Function cephimages.list in mine_functions failed to execute</span><o:p></o:p></p>
<p class="MsoNormal"> Sep 21 14:47:32 salt salt-minion[13456]: [ERROR ] Exception occurred while handling stream: [Errno 0] Success<o:p></o:p></p>
<p class="MsoNormal"> Sep 21 14:53:44 salt salt-minion[13456]: [ERROR ] Function cephimages.list in mine_functions failed to execute<o:p></o:p></p>
<p class="MsoNormal"> Sep 21 15:05:03 salt salt-minion[13456]: [ERROR ] Exception occurred while handling stream: [Errno 0] Success<o:p></o:p></p>
<p class="MsoNormal"> Sep 21 15:53:44 salt salt-minion[13456]: [ERROR ] Function cephimages.list in mine_functions failed to execute<o:p></o:p></p>
<p class="MsoNormal"> Sep 21 16:53:44 salt salt-minion[13456]: [ERROR ] Function cephimages.list in mine_functions failed to execute<o:p></o:p></p>
<p class="MsoNormal"> Sep 21 17:04:53 salt salt-minion[13456]: [CRITICAL] Rendering SLS 'base:ceph.mgr.key.default' failed: Conflicting ID '/srv/salt/ceph/mgr/cache/localhost.keyring'<o:p></o:p></p>
</div>
</body>
</html>