[Deepsea-users] filestore to bluestore migration

[Thomas Sundell]

Thank's Eric for your response, see inline comments.

On Fri, Dec 8, 2017 at 3:10 PM, Eric Jackson <ejackson at suse.com> wrote:
> Hi Thomas,
>   Were you running Stage 2 after making changes to the policy.cfg?  The reason
> I ask is that the behavior you describe (i.e. migration runs, but nothing
> changed) gives the impression that the migration "thinks" that you are already
> configured.
>
>   Try running
>
> salt -I roles:storage osd.report
>
> That will compare the existing configuration in the pillar against the
> configuration of the storage node.  If you see messages like
>
> All configured OSDs are active

salt -I roles:storage osd.report
osd1.ceph.mydomain.fi:
All configured OSDS are active
osd2.ceph.mydomain.fi:
All configured OSDS are active
osd3.ceph.mydomain.fi:
All configured OSDS are active
osd4.ceph.mydomain.fi:
All configured OSDS are active

>
> and a query of the pillar
>
> salt -l roles:storage pillar.get ceph
>

salt -I roles:storage pillar.get ceph
osd1.ceph.mydomain.fi:
osd2.ceph.mydomain.fi:
osd3.ceph.mydomain.fi:
osd4.ceph.mydomain.fi:

> reflects the policy.cfg correctly, then you should be migrated to bluestore.
> To manually verify, pick a storage node and run
>
> cat /var/lib/ceph/osd/ceph-*/type
>

cat /var/lib/ceph/osd/ceph-*/type
filestore
filestore
filestore
filestore
filestore
filestore
filestore
filestore

We only have 4 storage nodes in this test cluster and each of them have 8 OSDs.

> ***
> With respect to the migration, the commands to migrate one node are in
> /srv/salt/ceph/mgirate/osds/default.sls.   Effectively,
>
> salt 'data1*' state.apply ceph.redeploy.osds
> salt 'admin*' state.apply ceph.remove.migrated
>
> The rest of the state file is waiting for a healthy cluster.  Notice that the
> cleanup is batched (i.e. the 'old' OSDs are not removed until the end).
>
> ***
> With respect to the profile-default and the one generated by the
> ceph.migrate.policy orchestration, either should be fine given some conditions.
>
> Here's a quick history: DeepSea originally tried to encode a useful name for a
> profile for a range of defaults.  The general issue is that most sites fell
> outside of the guessed defaults and admins were left to hand crafting their
> hardware profiles.
>
> The strategy now is to create 'profile-default' with a hardcoded 1 to 5 ratio
> or standalone OSDs depending on the hardware available.
>
> If you prefer the configuration provided by the profile-default, then feel free
> to use it.  The only caveat is to verify that devices have not been left off
> or used in ways not originally intended.  The purpose of the
> ceph.migrate.policy was to take keep any manual modifications and only change
> the type.  The journal media would be used for the wal and db for bluestore.
>
> Feel free to experiment with `salt-run proposal.help` and `salt-run
> proposal.peek`.  Once you decide on what you really want, the migration will
> carry it out.
>

Another thing struck me, if the "old" profile was not correctly
formatted when they deployed the cluster could the
"ceph.migrate.policy" have created a "faulty" one? Here is a snip of
the premigrated yml:

storage:
  data+journals: []
  osds:
  - /dev/disk/by-id/ata-ST4000VN0001-1SF178_Z4F0PS2P

Because when I create a new proposal (salt-run proposal.populate
name=my7to1profile ratio=7) I see there are several attributes that
are not in the migrated profile like db, db_size, wal and wal_size.

One last thing, if I were to set the new my7to1profile in policy.cfg
will "salt-run state.orch ceph.migrate.osds" magically migrate the old
setup to the new without destroying our testdata :) ?

Thomas