[Deepsea-users] [ses-users] Trying out migrations

Stephan Müller smueller at suse.com
Thu Jul 6 03:40:58 MDT 2017


Hi I tried to migrate from filestore to bluestore, according to this
mail, but it was without success. After that I tried step 1 from this
mail http://mailman.suse.de/mlarch/SuSE/ses-users/2017/ses-users.2017.0
5/msg00201.html and successfully put my cluster into a error state it
hasn't recovered for over half an hour (no change at all in output of
ceph -s).

I will describe where the problems started in your mail.

Am Donnerstag, den 29.06.2017, 10:41 -0400 schrieb Eric Jackson:
> Hello all,
>   For everyone wanting to try out the migration functionality with
> DeepSea 
> 0.7.15 ( and soon node by node with 0.7.16), here is a primitive
> workflow.  I 
> would suggest using the new proposal runner to create multiple
> configurations.
> 
> 1) Start with a fresh cluster
My cluster wasn't completely fresh as I have used it for 1 or 2 weeks,
but to only to create pools and RBDs with no files on them.

> 2) Run "salt '*' osd.report"
>      The result should agree that your current configuration is
> active
They were.

> 3) Create a new hardware profile
>      e.g. salt-run proposal.populate nvme-ssd=True ratio=3 
> name=bluestore+waldb
>      I used this for servers with 6 SSDs and 2 NVMe devices.
> 
I did it returned {} when I ran it the first time.
Later I have run it again and got a config as the return but also this
warning message:

"[WARNING ] not overwriting existing proposal stargazer-sle-
1.oa.suse.de"

> 4) Edit your policy.cfg, comment out the existing profile and add the
> new 
> profile.  I commented out these lines: 
> 
> profile-2Intel745GB-6INTEL372GB-1/cluster/*.sls
> profile-2Disk2GB-1/stack/default/ceph/minions/data*.yml
> 
> and added
> 
> profile-bluestore+waldb/cluster/blueshark[4-8]*.sls
> profile-bluestore+waldb/stack/default/ceph/minions/blueshark[4-
> 8]*.yml
> 
I did the same.
> 5) Refresh the pillar (e.g. salt-run state.orch ceph.stage.2)
> 
I got the following error:

"profiles_populated       : ['There are no files under the profiles
directory. Probably an issue with the discovery stage.']
[ERROR   ] No highstate or sls specified, no execution made"

Than I started running step 3 again, with the warning seen under step 3
but continued with step 5 getting the same error.

Than I started to copy the missing files from the old comment out
profile. That worked but didn't change anything (If I had really worked
with deepSea before I would have known that)

> 6) Run "salt '*' osd.report"
>      Depending on the configuration changes between the existing and
> new 
> profile, expect to see some list of devices
> 
> 7) Run the migration
>     salt-run state.orch ceph.migrate.osds
>     Unfortunately, no progress is returned.  Running 'ceph osd tree'
> will give 
> an indication of which server and OSD is currently reconfiguring.

To run this you first have to run:
salt-run disengage.safety

After I have run this after I die step 1 from the other mentioned mail
because nothing had changed, my cluster ended up in a unrecoverable
error state (at least for me).

> 
> 8) Run "salt '*' osd.report" when complete.  The expectation is that 
> everything is converted.  If any devices did not succeed, check the 
> /var/log/salt/minion for commands related to that device.  All
> commands (e.g. 
> sgdisk, prepare, activate, etc.) are logged.
> 
> With our working wip-osd branch, Stage 3 will be able to correct
> broken OSDs 
> under some conditions.  I expect this to be available with 0.7.16.
> 
> ----
> Another comment about the current strategy: Stage 3 will only add an
> OSD and 
> the migrate will only reconfigure an OSD.  I can see the argument
> that Stage 3 
> should just make the cluster the way the admin wants it.
> 
> Any thoughts on this either way would be appreciated.
> 

My final question is why did Step 3 not succeed leaving me clueless
whats wrong :/


-- 
Stephan Müller

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://lists.suse.com/pipermail/deepsea-users/attachments/20170706/c8a550a8/attachment.sig>


More information about the Deepsea-users mailing list