<p dir="ltr">Hi Allen,</p>
<p dir="ltr">I think that you need empty disks for deepsea to 'target' them.</p>
<p dir="ltr">Can you wipe the partition's beginning, disk beginning and disk end ?</p>
<p dir="ltr">Should be something like:</p>
<p dir="ltr">for partition in /dev/sdX[0-9]*<br>
do <br>
dd if=/dev/zero of=$partition bs=4096 count=1 oflag=direct done </p>
<p dir="ltr"> dd if=/dev/zero of=/dev/sdX bs=512 count=34 oflag=direct<br></p>
<p dir="ltr"> dd if=/dev/zero of=/dev/sdX bs=512 count=33 \
<br>
seek=$((`blockdev --getsz /dev/sdX` - 33)) oflag=direct</p>
<p dir="ltr">And then create a gpt partition table:</p>
<p dir="ltr">sgdisk -Z --clear -g /dev/sdX</p>
<p dir="ltr">Source: <a href="https://www.google.bg/url?sa=t&source=web&rct=j&url=https://www.suse.com/documentation/suse-enterprise-storage-5/pdfdoc/book_storage_deployment/book_storage_deployment.pdf&ved=2ahUKEwj_2ouC4p7jAhWkwqYKHd7OBJUQFjAAegQIARAB&usg=AOvVaw3g9_lOOBwwzqK3siEkNbnF">https://www.google.bg/url?sa=t&source=web&rct=j&url=https://www.suse.com/documentation/suse-enterprise-storage-5/pdfdoc/book_storage_deployment/book_storage_deployment.pdf&ved=2ahUKEwj_2ouC4p7jAhWkwqYKHd7OBJUQFjAAegQIARAB&usg=AOvVaw3g9_lOOBwwzqK3siEkNbnF</a></p>
<p dir="ltr">Best Regards,<br>
Strahil Nikolov</p>
<div class="quote">On Jul 6, 2019 00:41, Allen Sellars <asellars@vigilantnow.com> wrote:<br type='attribution'><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<p><span style="font-size:11pt">I have a cisco UCS S3260 with 52 6TB spinning disks and 4 SSDs as DB disks.</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">I have no profile-* configs in the proposals directory.</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">I’ve obscured FQDNs</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">Stages 0-2 run fine with no failures. I see the following in stage 3:</span></p>
<p><span style="font-size:11pt">When I run salt-run state.orch ceph.stage.3 my salt-master return this:</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">firewall : disabled</span></p>
<p><span style="font-size:11pt">apparmor : disabled</span></p>
<p><span style="font-size:11pt">subvolume : skipping</span></p>
<p><span style="font-size:11pt">DEV_ENV : True</span></p>
<p><span style="font-size:11pt">fsid : valid</span></p>
<p><span style="font-size:11pt">public_network : valid</span></p>
<p><span style="font-size:11pt">public_interface : valid</span></p>
<p><span style="font-size:11pt">cluster_network : valid</span></p>
<p><span style="font-size:11pt">cluster_interface : valid</span></p>
<p><span style="font-size:11pt">ip_version : valid</span></p>
<p><span style="font-size:11pt">monitors : valid</span></p>
<p><span style="font-size:11pt">mgrs : valid</span></p>
<p><span style="font-size:11pt">storage : valid</span></p>
<p><span style="font-size:11pt">storage_role : valid</span></p>
<p><span style="font-size:11pt">rgw : valid</span></p>
<p><span style="font-size:11pt">ganesha : valid</span></p>
<p><span style="font-size:11pt">master_role : valid</span></p>
<p><span style="font-size:11pt">time_server : valid</span></p>
<p><span style="font-size:11pt">fqdn : valid</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">Found DriveGroup <drive_group_default></span></p>
<p><span style="font-size:11pt">Calling dg.deploy on compound target I@roles:storage</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">[ERROR ] {'out': 'highstate', 'ret': {'s1103': {'module_|-wait for osd processes_|-cephprocesses.wait_|-run': {'name': 'cephprocesses.wait', 'changes': {'ret': False}, 'comment': 'Module function cephprocesses.wait
executed', 'result': False, '__sls__': 'ceph.processes.osd.default', '__run_num__': 0, 'start_time': '21:11:52.567915', 'duration': 933229.3, '__id__': 'wait for osd processes'}}}}</span></p>
<p><span style="font-size:11pt">admin_master:</span></p>
<p><span style="font-size:11pt"> Name: populate scrape configs - Function: salt.state - Result: Changed Started: - 21:03:44.756802 Duration: 6245.346 ms</span></p>
<p><span style="font-size:11pt"> Name: populate alertmanager peers - Function: salt.state - Result: Changed Started: - 21:03:51.002480 Duration: 2236.262 ms</span></p>
<p><span style="font-size:11pt"> Name: fileserver.clear_file_list_cache - Function: salt.runner - Result: Changed Started: - 21:03:53.239063 Duration: 795.163 ms</span></p>
<p><span style="font-size:11pt"> Name: install prometheus - Function: salt.state - Result: Clean Started: - 21:03:54.034548 Duration: 62470.849 ms</span></p>
<p><span style="font-size:11pt"> Name: push scrape configs - Function: salt.state - Result: Changed Started: - 21:04:56.505708 Duration: 1903.123 ms</span></p>
<p><span style="font-size:11pt"> Name: install alertmanager - Function: salt.state - Result: Clean Started: - 21:04:58.409127 Duration: 38233.69 ms</span></p>
<p><span style="font-size:11pt"> Name: populate grafana config fragments - Function: salt.state - Result: Clean Started: - 21:05:36.643118 Duration: 2777.631 ms</span></p>
<p><span style="font-size:11pt"> Name: install grafana - Function: salt.state - Result: Clean Started: - 21:05:39.420958 Duration: 109235.045 ms</span></p>
<p><span style="font-size:11pt"> Name: time - Function: salt.state - Result: Changed Started: - 21:07:28.656309 Duration: 46245.898 ms</span></p>
<p><span style="font-size:11pt"> Name: configuration check - Function: salt.state - Result: Clean Started: - 21:08:14.902504 Duration: 920.39 ms</span></p>
<p><span style="font-size:11pt"> Name: create ceph.conf - Function: salt.state - Result: Changed Started: - 21:08:15.823194 Duration: 5960.673 ms</span></p>
<p><span style="font-size:11pt"> Name: configuration - Function: salt.state - Result: Changed Started: - 21:08:21.784167 Duration: 2212.586 ms</span></p>
<p><span style="font-size:11pt"> Name: admin - Function: salt.state - Result: Clean Started: - 21:08:23.996960 Duration: 962.78 ms</span></p>
<p><span style="font-size:11pt"> Name: mgr keyrings - Function: salt.state - Result: Clean Started: - 21:08:24.960033 Duration: 1011.557 ms</span></p>
<p><span style="font-size:11pt"> Name: monitors - Function: salt.state - Result: Changed Started: - 21:08:25.971888 Duration: 17081.809 ms</span></p>
<p><span style="font-size:11pt"> Name: mgr auth - Function: salt.state - Result: Changed Started: - 21:08:43.054000 Duration: 5225.853 ms</span></p>
<p><span style="font-size:11pt"> Name: mgrs - Function: salt.state - Result: Changed Started: - 21:08:48.280055 Duration: 44721.595 ms</span></p>
<p><span style="font-size:11pt"> Name: install ca cert in mgr minions - Function: salt.state - Result: Changed Started: - 21:09:33.001952 Duration: 2575.651 ms</span></p>
<p><span style="font-size:11pt"> Name: retry.cmd - Function: salt.function - Result: Changed Started: - 21:09:35.577911 Duration: 1095.445 ms</span></p>
<p><span style="font-size:11pt"> Name: dashboard - Function: salt.state - Result: Changed Started: - 21:09:36.673648 Duration: 7548.494 ms</span></p>
<p><span style="font-size:11pt"> Name: osd auth - Function: salt.state - Result: Changed Started: - 21:09:44.222446 Duration: 2545.84 ms</span></p>
<p><span style="font-size:11pt"> Name: sysctl - Function: salt.state - Result: Changed Started: - 21:09:46.768599 Duration: 758.771 ms</span></p>
<p><span style="font-size:11pt"> Name: set osd keyrings - Function: salt.state - Result: Clean Started: - 21:09:47.527655 Duration: 749.077 ms</span></p>
<p><span style="font-size:11pt"> Name: disks.deploy - Function: salt.runner - Result: Changed Started: - 21:09:48.277007 Duration: 30133.606 ms</span></p>
<p><span style="font-size:11pt"> Name: mgr tuned - Function: salt.state - Result: Changed Started: - 21:10:18.410965 Duration: 19570.008 ms</span></p>
<p><span style="font-size:11pt"> Name: mon tuned - Function: salt.state - Result: Changed Started: - 21:10:37.981281 Duration: 18494.201 ms</span></p>
<p><span style="font-size:11pt"> Name: osd tuned - Function: salt.state - Result: Changed Started: - 21:10:56.475786 Duration: 13570.12 ms</span></p>
<p><span style="font-size:11pt"> Name: pools - Function: salt.state - Result: Changed Started: - 21:11:10.046234 Duration: 6984.189 ms</span></p>
<p><span style="font-size:11pt"> Name: wait until mon2 with role mon can be restarted - Function: salt.state - Result: Changed Started: - 21:11:17.030734 Duration: 7574.266 ms</span></p>
<p><span style="font-size:11pt"> Name: check if mon processes are still running on mon2 after restarting mons - Function: salt.state - Result: Changed Started: - 21:11:24.605309 Duration: 992.112 ms</span></p>
<p><span style="font-size:11pt"> Name: restarting mons on mon2 - Function: salt.state - Result: Clean Started: - 21:11:25.597732 Duration: 1626.034 ms</span></p>
<p><span style="font-size:11pt"> Name: wait until mon3 with role mon can be restarted - Function: salt.state - Result: Changed Started: - 21:11:27.224053 Duration: 6971.541 ms</span></p>
<p><span style="font-size:11pt"> Name: check if mon processes are still running on mon3 after restarting mons - Function: salt.state - Result: Changed Started: - 21:11:34.195896 Duration: 994.762 ms</span></p>
<p><span style="font-size:11pt"> Name: restarting mons on mon3 - Function: salt.state - Result: Clean Started: - 21:11:35.190958 Duration: 1620.595 ms</span></p>
<p><span style="font-size:11pt"> Name: wait until mon1 with role mon can be restarted - Function: salt.state - Result: Changed Started: - 21:11:36.811852 Duration: 7322.913 ms</span></p>
<p><span style="font-size:11pt"> Name: check if mon processes are still running on mon1 after restarting mons - Function: salt.state - Result: Changed Started: - 21:11:44.135071 Duration: 988.726 ms</span></p>
<p><span style="font-size:11pt"> Name: restarting mons on mon1 - Function: salt.state - Result: Clean Started: - 21:11:45.124101 Duration: 1625.383 ms</span></p>
<p><span style="font-size:11pt"> Name: wait until mon2 with role mgr can be restarted - Function: salt.state - Result: Changed Started: - 21:11:46.749778 Duration: 6997.631 ms</span></p>
<p><span style="font-size:11pt"> Name: check if mgr processes are still running on mon2 after restarting mgrs - Function: salt.state - Result: Changed Started: - 21:11:53.747726 Duration: 990.767 ms</span></p>
<p><span style="font-size:11pt"> Name: restarting mgr on mon2 - Function: salt.state - Result: Clean Started: - 21:11:54.738787 Duration: 1666.851 ms</span></p>
<p><span style="font-size:11pt"> Name: wait until mon3 with role mgr can be restarted - Function: salt.state - Result: Changed Started: - 21:11:56.405937 Duration: 6978.253 ms</span></p>
<p><span style="font-size:11pt"> Name: check if mgr processes are still running on mon3 after restarting mgrs - Function: salt.state - Result: Changed Started: - 21:12:03.384521 Duration: 2233.584 ms</span></p>
<p><span style="font-size:11pt"> Name: restarting mgr on mon3 - Function: salt.state - Result: Clean Started: - 21:12:05.618424 Duration: 1572.339 ms</span></p>
<p><span style="font-size:11pt"> Name: wait until mon1 with role mgr can be restarted - Function: salt.state - Result: Changed Started: - 21:12:07.191059 Duration: 6980.657 ms</span></p>
<p><span style="font-size:11pt"> Name: check if mgr processes are still running on mon1 after restarting mgrs - Function: salt.state - Result: Changed Started: - 21:12:14.172025 Duration: 992.372 ms</span></p>
<p><span style="font-size:11pt"> Name: restarting mgr on mon1 - Function: salt.state - Result: Clean Started: - 21:12:15.164731 Duration: 1618.461 ms</span></p>
<p><span style="font-size:11pt"> Name: wait until s1103 with role osd can be restarted - Function: salt.state - Result: Changed Started: - 21:12:16.783490 Duration: 6973.487 ms</span></p>
<p><span style="font-size:11pt">----------</span></p>
<p><span style="font-size:11pt"> ID: check if osd processes are still running on s1103 after restarting osds</span></p>
<p><span style="font-size:11pt"> Function: salt.state</span></p>
<p><span style="font-size:11pt"> Result: False</span></p>
<p><span style="font-size:11pt"> Comment: Run failed on minions: s1103</span></p>
<p><span style="font-size:11pt"> Started: 21:12:23.757280</span></p>
<p><span style="font-size:11pt"> Duration: 934163.423 ms</span></p>
<p><span style="font-size:11pt"> Changes:</span></p>
<p><span style="font-size:11pt"> s1103:</span></p>
<p><span style="font-size:11pt"> ----------</span></p>
<p><span style="font-size:11pt"> ID: wait for osd processes</span></p>
<p><span style="font-size:11pt"> Function: module.run</span></p>
<p><span style="font-size:11pt"> Name: cephprocesses.wait</span></p>
<p><span style="font-size:11pt"> Result: False</span></p>
<p><span style="font-size:11pt"> Comment: Module function cephprocesses.wait executed</span></p>
<p><span style="font-size:11pt"> Started: 21:11:52.567915</span></p>
<p><span style="font-size:11pt"> Duration: 933229.3 ms</span></p>
<p><span style="font-size:11pt"> Changes:</span></p>
<p><span style="font-size:11pt"> ----------</span></p>
<p><span style="font-size:11pt"> ret:</span></p>
<p><span style="font-size:11pt"> False</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> Summary for s1103</span></p>
<p><span style="font-size:11pt"> ------------</span></p>
<p><span style="font-size:11pt"> Succeeded: 0 (changed=1)</span></p>
<p><span style="font-size:11pt"> Failed: 1</span></p>
<p><span style="font-size:11pt"> ------------</span></p>
<p><span style="font-size:11pt"> Total states run: 1</span></p>
<p><span style="font-size:11pt"> Total run time: 933.229 s</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">Summary for admin_master</span></p>
<p><span style="font-size:11pt">-------------</span></p>
<p><span style="font-size:11pt">Succeeded: 47 (changed=33)</span></p>
<p><span style="font-size:11pt">Failed: 1</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">When running salt-run disks.report I get this:</span></p>
<p><span style="font-size:11pt">salt-run disks.report</span></p>
<p><span style="font-size:11pt">Found DriveGroup <drive_group_default></span></p>
<p><span style="font-size:11pt">Calling dg.report on compound target I@roles:storage</span></p>
<p><span style="font-size:11pt">|_</span></p>
<p><span style="font-size:11pt"> ----------</span></p>
<p><span style="font-size:11pt"> s1103:</span></p>
<p><span style="font-size:11pt"> |_</span></p>
<p><span style="font-size:11pt"> - 0</span></p>
<p><span style="font-size:11pt"> -</span></p>
<p><span style="font-size:11pt"> Total OSDs: 13</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> Solid State VG:</span></p>
<p><span style="font-size:11pt"> Targets: block.db Total size: 185.00 GB</span></p>
<p><span style="font-size:11pt"> Total LVs: 13 Size per LV: 14.23 GB</span></p>
<p><span style="font-size:11pt"> Devices: /dev/sdbc</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> Type Path LV Size % of device</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdaa 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdae 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdai 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdam 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdaq 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdau 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sday 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdc 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdg 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdk 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdo 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sds 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdw 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 14.23 GB 7%</span></p>
<p><span style="font-size:11pt"> |_</span></p>
<p><span style="font-size:11pt"> - 0</span></p>
<p><span style="font-size:11pt"> -</span></p>
<p><span style="font-size:11pt"> Total OSDs: 13</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> Solid State VG:</span></p>
<p><span style="font-size:11pt"> Targets: block.db Total size: 371.00 GB</span></p>
<p><span style="font-size:11pt"> Total LVs: 13 Size per LV: 28.54 GB</span></p>
<p><span style="font-size:11pt"> Devices: /dev/sdbd</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> Type Path LV Size % of device</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdab 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdaf 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdaj 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdan 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdar 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdav 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdaz 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdd 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdh 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdl 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdp 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdt 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdx 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> |_</span></p>
<p><span style="font-size:11pt"> - 0</span></p>
<p><span style="font-size:11pt"> -</span></p>
<p><span style="font-size:11pt"> Total OSDs: 13</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> Solid State VG:</span></p>
<p><span style="font-size:11pt"> Targets: block.db Total size: 371.00 GB</span></p>
<p><span style="font-size:11pt"> Total LVs: 13 Size per LV: 28.54 GB</span></p>
<p><span style="font-size:11pt"> Devices: /dev/sdbe</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> Type Path LV Size % of device</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdac 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdag 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdak 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdao 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdas 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdaw 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdba 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sde 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdi 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdm 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdq 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdu 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdy 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> |_</span></p>
<p><span style="font-size:11pt"> - 0</span></p>
<p><span style="font-size:11pt"> -</span></p>
<p><span style="font-size:11pt"> Total OSDs: 13</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> Solid State VG:</span></p>
<p><span style="font-size:11pt"> Targets: block.db Total size: 371.00 GB</span></p>
<p><span style="font-size:11pt"> Total LVs: 13 Size per LV: 28.54 GB</span></p>
<p><span style="font-size:11pt"> Devices: /dev/sdbf</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> Type Path LV Size % of device</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdad 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdah 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdal 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdap 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdat 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdax 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdbb 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdf 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdj 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdn 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdr 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdv 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> ----------------------------------------------------------------------------------------------------</span></p>
<p><span style="font-size:11pt"> [data] /dev/sdz 5.46 TB 100.0%</span></p>
<p><span style="font-size:11pt"> [block.db] vg: vg/lv 28.54 GB 7%</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">Here’s my drive_groups.yml:</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">drive_group_default:</span></p>
<p><span style="font-size:11pt"> target: 'I@roles:storage'</span></p>
<p><span style="font-size:11pt"> data_devices:</span></p>
<p><span style="font-size:11pt"> size: '2TB:'</span></p>
<p><span style="font-size:11pt"> rotational: 1</span></p>
<p><span style="font-size:11pt"> db_devices:</span></p>
<p><span style="font-size:11pt"> rotational: 0</span></p>
<p><span style="font-size:11pt"> size: '180GB:'</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">Here’s what I see on the minion itself:</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">2019-07-05 20:52:27,750 [salt.utils.decorators:613 ][WARNING ][65394] The function "module.run" is using its deprecated version and will expire in version "Sodium".</span></p>
<p><span style="font-size:11pt">2019-07-05 21:07:26,314 [salt.utils.decorators:613 ][WARNING ][66129] The function "module.run" is using its deprecated version and will expire in version "Sodium".</span></p>
<p><span style="font-size:11pt">2019-07-05 21:11:52,568 [salt.utils.decorators:613 ][WARNING ][70185] The function "module.run" is using its deprecated version and will expire in version "Sodium".</span></p>
<p><span style="font-size:11pt">2019-07-05 21:11:52,692 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:11:55,801 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:12:01,899 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:12:11,001 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:12:23,104 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:12:38,210 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:12:56,321 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:13:17,437 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:13:41,557 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:14:08,676 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:14:38,801 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:15:11,930 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:15:48,031 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:16:27,128 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:17:09,232 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:17:54,371 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:18:42,510 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:19:33,653 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:20:27,799 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:21:24,951 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:22:25,107 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:23:25,270 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:24:25,425 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:25:25,582 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:26:25,737 [salt.loaded.ext.module.cephprocesses:252 ][ERROR ][70185] ERROR: process ceph-osd for role storage is not running</span></p>
<p><span style="font-size:11pt">2019-07-05 21:27:25,796 [salt.loaded.ext.module.cephprocesses:389 ][ERROR ][70185] Timeout expired</span></p>
<p><span style="font-size:11pt">2019-07-05 21:27:25,797 [salt.state :323 ][ERROR ][70185] {'ret': False}</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt">What am I missing here? The OSDs were deploying with deepsea and ceph 13 mimic and the storage profiles in profile-*. I seem to be missing something with the new drive_group setup</span></p>
<p><span style="font-size:11pt"> </span></p>
<p><span style="font-size:11pt"> </span></p>
<p><i><span style="font-size:11pt;color:black">Allen</span></i><span style="color:black"></span></p>
<p> </p>
</div>
</div>
</blockquote></div>