<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:MingLiU;
panose-1:2 2 5 9 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 2 6 3 5 4 5 2 3 4;}
@font-face
{font-family:Monaco;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"\@MingLiU";
panose-1:2 1 6 9 0 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
background:#0C0503;
font-size:9.0pt;
font-family:Monaco;
color:#F4F4F4;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0in;
margin-bottom:.0001pt;
background:#0C0503;
font-size:9.0pt;
font-family:Monaco;
color:#FF3E1F;}
p.p3, li.p3, div.p3
{mso-style-name:p3;
margin:0in;
margin-bottom:.0001pt;
background:#0C0503;
font-size:9.0pt;
font-family:Monaco;
color:#C63A22;}
p.p4, li.p4, div.p4
{mso-style-name:p4;
margin:0in;
margin-bottom:.0001pt;
background:#0C0503;
font-size:9.0pt;
font-family:Monaco;
color:#F4F4F4;}
p.p5, li.p5, div.p5
{mso-style-name:p5;
margin:0in;
margin-bottom:.0001pt;
background:#0C0503;
font-size:9.0pt;
font-family:Monaco;
color:#8EFB29;}
p.p6, li.p6, div.p6
{mso-style-name:p6;
margin:0in;
margin-bottom:.0001pt;
background:#0C0503;
font-size:9.0pt;
font-family:Monaco;
color:#36BE28;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Courier New";
color:windowtext;
font-weight:normal;
font-style:normal;}
span.s1
{mso-style-name:s1;
color:#C63A22;}
span.s3
{mso-style-name:s3;
color:#F4F4F4;}
span.s4
{mso-style-name:s4;
color:#2FE822;}
span.s2
{mso-style-name:s2;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle31
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle32
{mso-style-type:personal-reply;
font-family:"Courier New";
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Thanks Joel, yes DNS, NTP is configured and behaving correctly. SP3/SES5 from current repo. salt-api service, master, minion service running (with one error.)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">I’m walking through the Deployment guide line by line with same result, now on my second freshly built master node. Salt output is at the bottom of this message. Key: After stage
0, the */proposals directory has NOT been created. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Here’s my build on a single flat network(Azure vNet 172.19.20.0/24):
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Root ssh enabled and key based login from master to all nodes as root. All nodes rebooted before salt stage.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">All nodes using identical image and fully patched CPE_NAME="cpe:/o:suse:sles:12:sp3", firewall off, etc. - the Azure instance defaults.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Salt (and all nodes):~ # zypper lr -E<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Repository priorities are without effect. All enabled repositories share the same priority.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""># | Alias | Name | Enabled | GPG Check | Refresh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">---+--------------------------------------------------------------------+-----------------------------------+---------+-----------+--------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> 3 | SUSE_Enterprise_Storage_5_x86_64:SUSE-Enterprise-Storage-5-Pool | SUSE-Enterprise-Storage-5-Pool | Yes | (r ) Yes | No
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> 5 | SUSE_Enterprise_Storage_5_x86_64:SUSE-Enterprise-Storage-5-Updates | SUSE-Enterprise-Storage-5-Updates | Yes | (r ) Yes | Yes
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> 8 | SUSE_Linux_Enterprise_Server_12_SP3_x86_64:SLES12-SP3-Pool | SLES12-SP3-Pool | Yes | (r ) Yes | No
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> 10 | SUSE_Linux_Enterprise_Server_12_SP3_x86_64:SLES12-SP3-Updates | SLES12-SP3-Updates | Yes | (r ) Yes | Yes
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Courier New"">**DNS**</span></b><span style="font-size:10.0pt;font-family:"Courier New""> all nodes resolve bidirectionally. Azure cares for DNS but I’ve also updated hosts files.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:~ # hostname<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:~ # ping salt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">PING salt.acylew2ti3nulm1e5a1hcxdv0h.bx.internal.cloudapp.net (127.0.0.1) 56(84) bytes of data.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">64 bytes from salt.acylew2ti3nulm1e5a1hcxdv0h.bx.internal.cloudapp.net (127.0.0.1): icmp_seq=1 ttl=64 time=0.030 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">104.211.27.224 Outside NAT to 172.19.20.10 salt.acylew2ti3nulm1e5a1hcxdv0h.bx.internal.cloudapp.net salt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">172.19.20.12 mon1.acylew2ti3nulm1e5a1hcxdv0h.bx.internal.cloudapp.net mon1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">172.19.20.13 mon2.acylew2ti3nulm1e5a1hcxdv0h.bx.internal.cloudapp.net mon2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">172.19.20.14 mon3.acylew2ti3nulm1e5a1hcxdv0h.bx.internal.cloudapp.net mon3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">172.19.20.15 osd1.acylew2ti3nulm1e5a1hcxdv0h.bx.internal.cloudapp.net osd1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">172.19.20.16 osd2.acylew2ti3nulm1e5a1hcxdv0h.bx.internal.cloudapp.net osd2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">172.19.20.17 osd3.acylew2ti3nulm1e5a1hcxdv0h.bx.internal.cloudapp.net osd3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">172.19.20.18 igw1.acylew2ti3nulm1e5a1hcxdv0h.bx.internal.cloudapp.net ogw1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Courier New"">**NTP**</span></b><span style="font-size:10.0pt;font-family:"Courier New""> all nodes 5 minutes sync interval to same Stratum 1 server in same GEO as Azure AZ: (US East) navobs1.gatech.edu
as shown: <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">bash-3.2$ pssh -h pssh-hosts -l sesuser -i sudo ntpq -p<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[1] 11:15:27 [SUCCESS] mon3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> remote refid st t when poll reach delay offset jitter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">==============================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">*navobs1.gatech. .GPS. 1 u 19 64 1 15.596 -4.863 0.333<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[2] 11:15:27 [SUCCESS] salt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> remote refid st t when poll reach delay offset jitter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">==============================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">navobs1.gatech. .GPS. 1 u 42 64 1 17.063 -6.702 0.000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[3] 11:15:27 [SUCCESS] igw1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> remote refid st t when poll reach delay offset jitter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">==============================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">*navobs1.gatech. .GPS. 1 u 18 64 1 17.394 -27.874 7.663<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[4] 11:15:27 [SUCCESS] osd1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> remote refid st t when poll reach delay offset jitter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">==============================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">*navobs1.gatech. .GPS. 1 u 21 64 1 16.962 -3.755 0.813<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[5] 11:15:27 [SUCCESS] osd2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> remote refid st t when poll reach delay offset jitter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">==============================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">*navobs1.gatech. .GPS. 1 u 22 64 1 15.832 -4.709 3.062<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[6] 11:15:27 [SUCCESS] osd3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> remote refid st t when poll reach delay offset jitter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">==============================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">*navobs1.gatech. .GPS. 1 u 26 64 1 15.877 -3.252 19.131<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[7] 11:15:27 [SUCCESS] mon1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> remote refid st t when poll reach delay offset jitter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">==============================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">navobs1.gatech. .GPS. 1 u 2 64 1 16.120 -4.263 0.000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">[8] 11:15:27 [SUCCESS] mon2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> remote refid st t when poll reach delay offset jitter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">==============================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">navobs1.gatech. .GPS. 1 u 2 64 1 16.108 -7.713 0.959<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Courier New"">**SALT**
<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:~ # systemctl status salt-api salt-master salt-minion |grep 'active (running)'
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Active: active (running) since Wed 2018-09-19 18:14:39 UTC; 13min ago<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Active: active (running) since Wed 2018-09-19 18:14:42 UTC; 13min ago<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Active: active (running) since Wed 2018-09-19 18:14:41 UTC; 13min ago<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:~ # systemctl status salt-api salt-master salt-minion |grep ERROR
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Sep 19 18:14:49
<b><span style="color:red">salt salt-minion[1413]: [ERROR ] Function cephimages.list in mine_functions failed to execute</span></b><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:~ # salt-key --list-all<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Accepted Keys:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> igw1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> mon1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> mon2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> mon3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> osd1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> osd2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> osd3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> salt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Denied Keys:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Unaccepted Keys:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Rejected Keys:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:~ # salt '*' test.ping<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> True<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">osd2:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> True<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mon3:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> True<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">osd3:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> True<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">osd1:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> True<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mon2:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> True<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">igw1:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> True<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">mon1:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> True<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:~ # cat /srv/pillar/ceph/master_minion.sls<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">master_minion: salt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:~ # cat /srv/pillar/ceph/deepsea_minions.sls<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""># Choose all minions<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">deepsea_minions: '*'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Courier New"">**SALT STAGES**
<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Stage 0 is successful with no errors but does not create the proposals folder.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:~ # salt-run state.orch ceph.stage.prep<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> deepsea_minions : valid<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> master_minion : valid<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ceph_version : valid<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> [WARNING ] All minions are ready<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> salt_master:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: sync master - Function: salt.state - Result: Changed Started: - 18:44:20.440255 Duration: 949.98 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: salt-api - Function: salt.state - Result: Changed Started: - 18:44:21.390365 Duration: 3256.749 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: repo master - Function: salt.state - Result: Clean Started: - 18:44:24.647227 Duration: 351.0 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: metapackage master - Function: salt.state - Result: Clean Started: - 18:44:24.998333 Duration: 1127.063 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: prepare master - Function: salt.state - Result: Changed Started: - 18:44:26.125514 Duration: 4109.917 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: filequeue.remove - Function: salt.runner - Result: Changed Started: - 18:44:30.235610 Duration: 2071.199 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: restart master - Function: salt.state - Result: Clean Started: - 18:44:32.306972 Duration: 1006.268 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: filequeue.add - Function: salt.runner - Result: Changed Started: - 18:44:33.313369 Duration: 1352.98 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: minions.ready - Function: salt.runner - Result: Changed Started: - 18:44:34.666528 Duration: 1891.677 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: repo - Function: salt.state - Result: Clean Started: - 18:44:36.558363 Duration: 553.342 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: metapackage minions - Function: salt.state - Result: Clean Started: - 18:44:37.111825 Duration: 3993.733 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: common packages - Function: salt.state - Result: Clean Started: - 18:44:41.105706 Duration: 2434.079 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: sync - Function: salt.state - Result: Changed Started: - 18:44:43.539897 Duration: 1381.692 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: mines - Function: salt.state - Result: Clean Started: - 18:44:44.921708 Duration: 1657.019 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: updates - Function: salt.state - Result: Changed Started: - 18:44:46.578853 Duration: 11183.347 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: restart - Function: salt.state - Result: Clean Started: - 18:44:57.762346 Duration: 1553.957 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Name: mds restart noop - Function: test.nop - Result: Clean Started: - 18:44:59.316442 Duration: 0.348 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Summary for salt_master<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> -------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Succeeded: 17 (changed=8)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Failed: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> -------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Total states run: 17<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Total run time: 38.874 s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Courier New";color:red">Before running Stage 1, the /srv/pillar/ceph/proposals directory does not exist.
<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">salt:~ # ls /srv/pillar/ceph/proposals/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> ls: cannot access '/srv/pillar/ceph/proposals/': No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">That’s where I’m at – Googling..
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">~ Kevin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">From: <deepsea-users-bounces@lists.suse.com> on behalf of Joel Zhou <joel.zhou@suse.com><br>
Reply-To: Discussions about the DeepSea management framework for Ceph <deepsea-users@lists.suse.com><br>
Date: Tuesday, September 18, 2018 at 11:34 PM<br>
To: Discussions about the DeepSea management framework for Ceph <deepsea-users@lists.suse.com><br>
Subject: Re: [Deepsea-users] stage 1 errors on Azure<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Hi Kevin,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">My short answer is,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Step 1, before stage 0, check your salt-api service on salt-master node first.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">```bash<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">zypper install -y salt-api<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">systemctl enable salt-api.service<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">systemctl start salt-api.service<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Step 2, make sure NTP service works correctly on all nodes, which means time synchronized correctly on all nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Step 3, reboot all your nodes, if acceptable. In case of kernel updated somehow.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Step 4, then you have to start over again from stage 0 to 5.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Basically, deepsea is a bunch of salt scripts, and salt based on python2 and/or python3.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">I have no clues about your whole running stack, so assume SLES 12 sp3 + SES 5, which works fine and supported.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">More info would be helpful, and also your purpose, such as for practice on your own, or for PoC/testing to meet customer’s demands.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">-- <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Joel Zhou
</span><span style="font-size:10.0pt;font-family:"MS Gothic"">周</span><span style="font-size:10.0pt;font-family:MingLiU">维伟</span><span style="font-size:10.0pt;font-family:"Courier New""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Senior Storage Technologist, APJ<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:SimSun">Mobile: +86 18514577601</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:SimSun">Email: joel.zhou@suse.com</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black"><deepsea-users-bounces@lists.suse.com> on behalf of Kevin Ayres <kevin.ayres@suse.com><br>
<b>Reply-To: </b>Discussions about the DeepSea management framework for Ceph <deepsea-users@lists.suse.com><br>
<b>Date: </b>Tuesday, September 18, 2018 at 4:49 PM<br>
<b>To: </b>"deepsea-users@lists.suse.com" <deepsea-users@lists.suse.com><br>
<b>Subject: </b>[Deepsea-users] stage 1 errors on Azure</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Hey guys, I can’t seem to get past stage 1. Stage 0 complete successfully. Same output with deepsea command. The master and minion service are running and bidirectional host resolution
are good. Keys are all accepted. From what I can determine, the default files are not created by stage 0 for some reason. Thoughts? What I’m seeing is that it fails to create the /srv/pillar/ceph/proposals
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">I’m running through this doc line by line:
<a href="https://www.suse.com/documentation/suse-enterprise-storage-5/singlehtml/book_storage_deployment/book_storage_deployment.html#deepsea.cli">
https://www.suse.com/documentation/suse-enterprise-storage-5/singlehtml/book_storage_deployment/book_storage_deployment.html#deepsea.cli</a>
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">~ Kevin</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">salt:~ #</span><span class="s2"> salt-run state.orch ceph.stage.discovery</span><o:p></o:p></p>
<p class="p2"><span class="s3">salt-api </span><span class="apple-converted-space"><span style="color:#F4F4F4">
</span></span><span class="s3">: </span><span class="s2">["Salt API is failing to authenticate - try 'systemctl restart salt-master': list index out of range"]</span><o:p></o:p></p>
<p class="p1"><span class="s2">deepsea_minions</span><span class="apple-converted-space">
</span><span class="s2">: </span><span class="s4">valid</span><o:p></o:p></p>
<p class="p1"><span class="s2">master_minion</span><span class="apple-converted-space">
</span><span class="s2">: </span><span class="s4">valid</span><o:p></o:p></p>
<p class="p1"><span class="s2">ceph_version </span><span class="apple-converted-space">
</span><span class="s2">: </span><span class="s4">valid</span><o:p></o:p></p>
<p class="p1"><span class="s2">[ERROR </span><span class="apple-converted-space">
</span><span class="s2">] No highstate or sls specified, no execution made</span><o:p></o:p></p>
<p class="p3"><span class="s2">salt_master:</span><o:p></o:p></p>
<p class="p3"><span class="s2">----------</span><o:p></o:p></p>
<p class="p3"><span class="apple-converted-space"><span style="color:#F4F4F4">
</span> </span><span class="s2">ID: salt-api failed</span><o:p></o:p></p>
<p class="p3"><span class="apple-converted-space"><span style="color:#F4F4F4">
</span></span><span class="s2">Function: salt.state</span><o:p></o:p></p>
<p class="p3"><span class="apple-converted-space"><span style="color:#F4F4F4">
</span> </span><span class="s2">Name: just.exit</span><o:p></o:p></p>
<p class="p3"><span class="apple-converted-space"><span style="color:#F4F4F4">
</span> </span><span class="s2">Result: False</span><o:p></o:p></p>
<p class="p3"><span class="apple-converted-space"><span style="color:#F4F4F4"> </span></span><span class="s2"> Comment: No highstate or sls specified, no execution made</span><o:p></o:p></p>
<p class="p3"><span class="apple-converted-space"><span style="color:#F4F4F4"> </span></span><span class="s2"> Started: 22:30:53.628882</span><o:p></o:p></p>
<p class="p3"><span class="apple-converted-space"><span style="color:#F4F4F4">
</span></span><span class="s2">Duration: 0.647 ms</span><o:p></o:p></p>
<p class="p3"><span class="apple-converted-space"> </span><span class="s2">Changes:</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p4"> <o:p></o:p></p>
<p class="p5"><span class="s2">Summary for salt_master</span><o:p></o:p></p>
<p class="p5"><span class="s2">------------</span><o:p></o:p></p>
<p class="p6"><span class="s2">Succeeded: 0</span><o:p></o:p></p>
<p class="p3"><span class="s2">Failed:</span><span class="apple-converted-space">
</span><span class="s2">1</span><o:p></o:p></p>
<p class="p5"><span class="s2">------------</span><o:p></o:p></p>
<p class="p5"><span class="s2">Total states run: </span><span class="apple-converted-space">
</span><span class="s2">1</span><o:p></o:p></p>
<p class="p5"><span class="s2">Total run time: </span><span class="apple-converted-space">
</span><span class="s2">0.647 ms</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#C63A22">salt:~ #</span><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4"> !tail</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">tail -f /var/log/salt/master</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">2018-09-18 22:29:08,797 [salt.loaded.ext.runners.validate][WARNING ][8499] role-igw/cluster/igw*.sls matched no files</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">2018-09-18 22:29:08,797 [salt.loaded.ext.runners.validate][WARNING ][8499] role-openattic/cluster/salt.sls matched no files</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">2018-09-18 22:29:08,797 [salt.loaded.ext.runners.validate][WARNING ][8499] config/stack/default/global.yml matched no files</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">2018-09-18 22:29:08,798 [salt.loaded.ext.runners.validate][WARNING ][8499] config/stack/default/ceph/cluster.yml matched no files</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">2018-09-18 22:29:08,798 [salt.loaded.ext.runners.validate][WARNING ][8499] cluster/*.sls matched no files</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">2018-09-18 22:29:08,798 [salt.loaded.ext.runners.validate][WARNING ][8499] stack/default/ceph/minions/*.yml matched no files</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">2018-09-18 22:29:08,822 [salt.state ][ERROR ][8499] No highstate or sls specified, no execution made</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">2018-09-18 22:29:52,472 [salt.transport.ipc][ERROR ][5672] Exception occurred while handling stream: [Errno 0] Success</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">2018-09-18 22:29:56,797 [salt.state ][ERROR ][8759] No highstate or sls specified, no execution made</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">2018-09-18 22:30:53,629 [salt.state ][ERROR ][9272] No highstate or sls specified, no execution made</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4"> </span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">There’s also some issue with the salt-minion.service:
</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:"Times New Roman",serif;color:#36BE28">●</span><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4"> salt-minion.service - The Salt Minion</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4"> Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled)</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4"> Active:
</span><span style="font-size:9.0pt;font-family:Monaco;color:#36BE28">active (running)</span><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4"> since Tue 2018-09-18 22:46:54 UTC; 12s ago</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4"> Main PID: 11082 (salt-minion)</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">…</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">.....</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">Sep 18 22:46:54 salt systemd[1]: Started The Salt Minion.</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#0C0503"><span style="font-size:9.0pt;font-family:Monaco;color:#F4F4F4">Sep 18 22:47:00 salt salt-minion[11082]: [ERROR ] Function cephimages.list in mine_functions failed to execute</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</body>
</html>