[Deepsea-users] Adding OSDs using Salt

Robert Grosschopff Robert.Grosschopff at suse.com
Thu Mar 16 09:13:39 MDT 2017


What I did at first was to simply add the new profile (..2Disk50GB...)
in profile.cfg :

profile-1Disk50GB-1/cluster/ses4-[1234]*.sls
profile-1Disk50GB-1/stack/default/ceph/minions/ses4-[1234]*.yml
profile-2Disk50GB-1/cluster/ses4-[1234]*.sls
profile-2Disk50GB-1/stack/default/ceph/minions/ses4-[1234]*.yml

Then I ran stage.2 :

Succeeded: 12 (changed=4)
Failed:     0

salt '*' pillar.items shows :

    storage:
        ----------
        data+journals:
        osds:
            - /dev/vdb
            - /dev/vdb
            - /dev/vdc

So, vdb was added again.

stage.3 throws a lot of failure messages since it cannot add vdb again:

cephadm at salt:~> sudo salt-run state.orch ceph.stage.3
firewall                 : disabled
fsid                     : valid
public_network           : valid
public_interface         : valid
cluster_network          : valid
cluster_interface        : valid
monitors                 : valid
storage                  : valid
master_role              : valid
mon_host                 : valid
mon_initial_members      : valid
time_server              : disabled
fqdn                     : valid
[WARNING ] Could not write out jid file for job 20170316122026087326.
Retrying.
[WARNING ] Could not write out jid file for job 20170316122026087326.
Retrying.
[WARNING ] Could not write out jid file for job 20170316122026087326.
Retrying.
[WARNING ] Could not write out jid file for job 20170316122026087326.
Retrying.
[WARNING ] Could not write out jid file for job 20170316122026087326.
Retrying.
[ERROR   ] prep_jid could not store a jid after 5 tries.
[ERROR   ] Could not store job cache info. Job details for this run may
be unavailable.
[ERROR   ] Run failed on minions: ses4-3.local.site, ses4-4.local.site, 
ses4-1.local.site, ses4-2.local.site
Failures:
    ses4-3.local.site:
        Data failed to compile:
    ----------
        Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID
'prepare /dev/vdb'
    ses4-4.local.site:
        Data failed to compile:
    ----------
        Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID
'prepare /dev/vdb'
    ses4-1.local.site:
        Data failed to compile:
    ----------
        Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID
'prepare /dev/vdb'
    ses4-2.local.site:
        Data failed to compile:
    ----------
        Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID
'prepare /dev/vdb'

salt.local.site_master:
  Name: packages - Function: salt.state - Result: Clean Started: -
12:20:27.760260 Duration: 1259.428 ms
  Name: configuration check - Function: salt.state - Result: Clean
Started: - 12:20:29.019844 Duration: 177.061 ms
  Name: configuration - Function: salt.state - Result: Clean Started: -
12:20:29.197064 Duration: 598.674 ms
  Name: admin - Function: salt.state - Result: Clean Started: -
12:20:29.795890 Duration: 190.272 ms
  Name: monitors - Function: salt.state - Result: Changed Started: -
12:20:29.986315 Duration: 454.657 ms
  Name: osd auth - Function: salt.state - Result: Changed Started: -
12:20:30.441126 Duration: 332.438 ms
----------
          ID: storage
    Function: salt.state
      Result: False
     Comment: Run failed on minions: ses4-3.local.site, ses4-
4.local.site, ses4-1.local.site, ses4-2.local.site
              Failures:
                  ses4-3.local.site:
                      Data failed to compile:
                  ----------
                      Rendering SLS 'base:ceph.osd.default' failed:
Conflicting ID 'prepare /dev/vdb'
                  ses4-4.local.site:
                      Data failed to compile:
                  ----------
                      Rendering SLS 'base:ceph.osd.default' failed:
Conflicting ID 'prepare /dev/vdb'
                  ses4-1.local.site:
                      Data failed to compile:
                  ----------
                      Rendering SLS 'base:ceph.osd.default' failed:
Conflicting ID 'prepare /dev/vdb'
                  ses4-2.local.site:
                      Data failed to compile:
                  ----------
                      Rendering SLS 'base:ceph.osd.default' failed:
Conflicting ID 'prepare /dev/vdb'
     Started: 12:20:30.773721
    Duration: 396.396 ms
     Changes:

Summary for salt.local.site_master
------------
Succeeded: 6 (changed=2)
Failed:    1
------------
Total states run:     7
Total run time:   3.409 s

So I had to remove the old entries to get Salt to add OSDs to my
existing nodes.

I would have expected that I can simply add the new disks and Salt will
notice that some disks already exist and just set up the new ones.

Now I am wondering what will happen if I integrate a new identical OSD-
Node. policy.cfg would need to have both disk profiles.  pillar.items
would show duplicate disk entries for the existing nodes again and
stage.3 would fail.

Robert

On Thu, 2017-03-16 at 10:14 -0400, Eric Jackson wrote:
> On Thursday, March 16, 2017 12:55:49 PM Robert Grosschopff wrote:
> > 
> > *,
> > 
> > I added OSDs using Salt the following way:
> > 
> > - Add disks to system
> > - Run stage.1
> > - Modify policy.cfg
> > o add profile-NEWDISK/cluster/OSD*.sls
> > o add profile-NEWDISK/stack/default/ceph/minions/OSD*.yml
> > o REMOVE old profile-OLDDISK/cluster/OSD*.sls
> > o REMOVE old profile-OLDDISK/stack/default/ceph/minions /OSD*.yml
> > - Run stage.2
> > - Run stage.3
> > 
> > If the old profiles are not removed  'salt \* pillar.items' will
> > have add
> > the old OSD profiles again.
> > 
> > Is this the way it is supposed to be done ?
> 
> Since you modify policy.cfg to use the profile-NEWDISK, you do not
> need to 
> remove the old profiles.  However, if you have no machines that will
> ever match 
> them again and want to clean up, there's no harm.
> 
> Does the new profile contain all the disks as OSDs in the way you
> wanted?  If 
> so, do exactly what you did.  Stage 3 will see that the existing OSDs
> are 
> already done and move on to adding the blank drives as additional
> OSDs.
> 
> If the new profile is not a simple addition of the existing disks
> (maybe you 
> replaced smaller disks and added additional disks), then removing the
> node is 
> the simpler alternative.  That is,
> 
> 1) Remove/comment out the node from policy.cfg
> 2) Run Stages 2-5 
> 3) Add the node back with new profile
> 
> Depending on your situation, you can take that as fast or as slow as 
> necessary.  That is, do all the storage nodes you physically changed
> or do 
> them one at a time. 
> > 
> > 
> > Robert
> > 
> > _______________________________________________
> > Deepsea-users mailing list
> > Deepsea-users at lists.suse.com
> > http://lists.suse.com/mailman/listinfo/deepsea-users
> _______________________________________________
> Deepsea-users mailing list
> Deepsea-users at lists.suse.com
> http://lists.suse.com/mailman/listinfo/deepsea-users


More information about the Deepsea-users mailing list