From jschmid at suse.de Wed Mar 1 09:15:24 2017 From: jschmid at suse.de (Joshua Schmid) Date: Wed, 1 Mar 2017 17:15:24 +0100 Subject: [Deepsea-users] Alternate Stage 0 In-Reply-To: <1729099.29A3CHdfbB@ruby> References: <1729099.29A3CHdfbB@ruby> Message-ID: <838631fc-fc55-d1ea-342c-a8df1bc90a08@suse.de> Hey, On 03/01/2017 03:41 PM, Eric Jackson wrote: > Hello everyone, > The prep stage is a bit difficult to get right for different environments. One > solution is to create alternative defaults. I believe we have support to > create > > /srv/salt/stage/prep/default-openstack.sls > > The corresponding master and minion orchestrations need to be created. > Steps can be removed that are not necessary for an OpenStack environment or > added to help in the preparation. > > Other discussions have me thinking that a > > /srv/salt/stage/prep/default-minimal.sls > > might be useful to some. I believe that the default behavior should always > have the latest updates including the kernel to give the best chance of a > successful deployment. However, some environments manage updates in their own > way and having a predefined selection is easy to do and seems reasonable. > > Does anyone think they would opt for the minimal setup (salt sync, mine > configuration, dependent packages) that is not in an OpenStack environment? Or > is this necessary? I have no prove/quote for this, but I can imagine that SE's want the quickest possible way to create a running cluster. Taking into consideration that one might re-deploy a system over and over again to demo something. Skipping the updates will save you some time I guess.. I actually think that splitting stage.0 in 1) mandatory calls ( like ceph.sync ceph.packages.common .. ) 2) update/maintenance 3) calls for a special env(actually _optional_) could make sense here. the split does not necessarily need to result in two new stages but could also be managed via a ENV_VAR or something of that nature. > > Eric > > > > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users > -- Freundliche Gr??e - Kind regards, Joshua Schmid SUSE Enterprise Storage SUSE Linux GmbH - Maxfeldstr. 5 - 90409 N?rnberg -------------------------------------------------------------------------------------------------------------------- SUSE Linux GmbH, GF: Felix Imend?rffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG N?rnberg) -------------------------------------------------------------------------------------------------------------------- From cgardner at suse.com Wed Mar 1 10:35:30 2017 From: cgardner at suse.com (Craig Gardner) Date: Wed, 1 Mar 2017 10:35:30 -0700 Subject: [Deepsea-users] Alternate Stage 0 In-Reply-To: <838631fc-fc55-d1ea-342c-a8df1bc90a08@suse.de> References: <1729099.29A3CHdfbB@ruby> <838631fc-fc55-d1ea-342c-a8df1bc90a08@suse.de> Message-ID: On 03/01/2017 09:15 AM, Joshua Schmid wrote: > On 03/01/2017 03:41 PM, Eric Jackson wrote: >> Does anyone think they would opt for the minimal setup (salt sync, mine >> configuration, dependent packages) that is not in an OpenStack environment? Or >> is this necessary? > > I have no prove/quote for this, but I can imagine that SE's want the > quickest possible way to create a running cluster. Taking into > consideration that one might re-deploy a system over and over again to > demo something. > Skipping the updates will save you some time I guess.. My $0.02 response here: Yes, I'm certain SE's want this. In fact, I've heard them request this in various ways. I definitely want to be supportive of the SE's, and they deserve our attention here. But I want to make sure that we understand that a "quick demo deployment" may not be a real or useful scenario for our customers. Yes, customers also want to save time and money, so anything that gets them up and running quickly is meaningful. But if the "quick deployment" is not a realistic, useful deployment, then I worry about having it as an available option for customers. Of course, tools that help SE's to sell the product are always welcome and serve their and our best interests. But maybe a "quick demo deployment" option is something that we should leave out of the mainstream product if it gets in the way of a productive experience for the customers. From ncutler at suse.cz Thu Mar 2 01:21:46 2017 From: ncutler at suse.cz (Nathan Cutler) Date: Thu, 2 Mar 2017 09:21:46 +0100 Subject: [Deepsea-users] Heads-up: need to remove OSDs from CRUSH map before deleting them Message-ID: <91677583-6035-a736-9e35-164a3431c66a@suse.cz> tl;dr Do not "ceph osd rm" an OSD before "ceph osd crush rm"ing it Sage's explanation: https://github.com/ceph/ceph/pull/13732/commits/1f4feb861876e8d997401088c285b70aa2f1ee1a -- Nathan Cutler Software Engineer Distributed Storage SUSE LINUX, s.r.o. Tel.: +420 284 084 037 From joao at suse.de Thu Mar 2 11:01:20 2017 From: joao at suse.de (Joao Eduardo Luis) Date: Thu, 2 Mar 2017 18:01:20 +0000 Subject: [Deepsea-users] Heads-up: need to remove OSDs from CRUSH map before deleting them In-Reply-To: <91677583-6035-a736-9e35-164a3431c66a@suse.cz> References: <91677583-6035-a736-9e35-164a3431c66a@suse.cz> Message-ID: <0dc37863-95f9-461f-d0d0-112707d17cb6@suse.de> On 03/02/2017 08:21 AM, Nathan Cutler wrote: > tl;dr Do not "ceph osd rm" an OSD before "ceph osd crush rm"ing it > > Sage's explanation: > https://github.com/ceph/ceph/pull/13732/commits/1f4feb861876e8d997401088c285b70aa2f1ee1a While still at a proposal phase, this whole thing ties up with http://pad.ceph.com/p/osd-replacement Following up on said proposal, there will be a 'ceph osd purge' command that will perform both those commands, along with a (proposed) 'osd destroy'. This will hit luminous, and therefore SES5. If you are reading this and think you may be interested in how we handle at-rest encryption keys, or replacing osds, you should read the pad above. Keep in mind it's still a proposal and changes are likely to happen. -Joao From Robert.Grosschopff at suse.com Wed Mar 15 06:28:02 2017 From: Robert.Grosschopff at suse.com (Robert Grosschopff) Date: Wed, 15 Mar 2017 12:28:02 +0000 Subject: [Deepsea-users] "no ceph clusters available" Message-ID: I recently installed a cluster using Salt. Everything was fine and I could see a graphical representation of the Crushmap in openATTIC. After a reboot of the cluster I keep getting a ?Sorra folks ? no ceph clusters available" . Any ideas ? From lmb at suse.com Wed Mar 15 06:30:44 2017 From: lmb at suse.com (Lars Marowsky-Bree) Date: Wed, 15 Mar 2017 13:30:44 +0100 Subject: [Deepsea-users] "no ceph clusters available" In-Reply-To: References: Message-ID: <20170315123044.qsww5tswnvqhu755@suse.com> On 2017-03-15T12:28:02, Robert Grosschopff wrote: > I recently installed a cluster using Salt. Everything was fine and I could see a graphical representation of the Crushmap in openATTIC. > > After a reboot of the cluster I keep getting a ?Sorra folks ? no ceph clusters available" . Is the cluster itself running? From Robert.Grosschopff at suse.com Wed Mar 15 06:33:37 2017 From: Robert.Grosschopff at suse.com (Robert Grosschopff) Date: Wed, 15 Mar 2017 12:33:37 +0000 Subject: [Deepsea-users] "no ceph clusters available" In-Reply-To: <20170315123044.qsww5tswnvqhu755@suse.com> References: <20170315123044.qsww5tswnvqhu755@suse.com> Message-ID: <76DB9AC7-CBAE-4ACC-A3D9-01A60C679003@suse.com> Yepp, apart from the usual ?too many PGs per OSD? everything is fine on the command line. openATTIC also shows all OSDs, Pools, Nodes, etc. On 15/03/2017, 13:30, "deepsea-users-bounces at lists.suse.com on behalf of Lars Marowsky-Bree" wrote: On 2017-03-15T12:28:02, Robert Grosschopff wrote: > I recently installed a cluster using Salt. Everything was fine and I could see a graphical representation of the Crushmap in openATTIC. > > After a reboot of the cluster I keep getting a ?Sorra folks ? no ceph clusters available" . Is the cluster itself running? _______________________________________________ Deepsea-users mailing list Deepsea-users at lists.suse.com http://lists.suse.com/mailman/listinfo/deepsea-users From abonilla at suse.com Wed Mar 15 08:10:23 2017 From: abonilla at suse.com (Alejandro Bonilla) Date: Wed, 15 Mar 2017 14:10:23 +0000 Subject: [Deepsea-users] "no ceph clusters available" In-Reply-To: <76DB9AC7-CBAE-4ACC-A3D9-01A60C679003@suse.com> References: <20170315123044.qsww5tswnvqhu755@suse.com> <76DB9AC7-CBAE-4ACC-A3D9-01A60C679003@suse.com> Message-ID: <60608E90-EEDF-4176-9BB5-D162625780CB@suse.com> Robert, I think that ?error? is normal. Just try adding widgets and see if the cluster shows up on the field as ?ceph?? Also, fix the WARNS too? If that doesn?t do it make sure of: ls -l /etc/ceph/ total 16 -rw------- 1 root root 129 Jan 24 23:15 ceph.client.admin.keyring -rw-rw---- 1 openattic openattic 111 Jan 24 23:17 ceph.client.openattic.keyring -rw-r--r-- 1 root root 362 Jan 24 23:15 ceph.conf DeepSea should have done that. > On Mar 15, 2017, at 8:33 AM, Robert Grosschopff wrote: > > Yepp, apart from the usual ?too many PGs per OSD? everything is fine on the command line. > > openATTIC also shows all OSDs, Pools, Nodes, etc. > > > > On 15/03/2017, 13:30, "deepsea-users-bounces at lists.suse.com on behalf of Lars Marowsky-Bree" wrote: > > On 2017-03-15T12:28:02, Robert Grosschopff wrote: > >> I recently installed a cluster using Salt. Everything was fine and I could see a graphical representation of the Crushmap in openATTIC. >> >> After a reboot of the cluster I keep getting a ?Sorra folks ? no ceph clusters available" . > > Is the cluster itself running? > > > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users > > > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users From Robert.Grosschopff at suse.com Wed Mar 15 08:40:46 2017 From: Robert.Grosschopff at suse.com (Robert Grosschopff) Date: Wed, 15 Mar 2017 14:40:46 +0000 Subject: [Deepsea-users] "no ceph clusters available" In-Reply-To: <60608E90-EEDF-4176-9BB5-D162625780CB@suse.com> References: <20170315123044.qsww5tswnvqhu755@suse.com> <76DB9AC7-CBAE-4ACC-A3D9-01A60C679003@suse.com> <60608E90-EEDF-4176-9BB5-D162625780CB@suse.com> Message-ID: <59B7C066-2264-42A6-951A-6A5B6F87C475@suse.com> Hi Alejandro, adding widgets works like a charm ( All other files in /etc/ceph are present as well. I can also extract the crushmap using ceph osd getcrushmap ?o cmap.bin Weird, somehow Robert On 15/03/2017, 15:10, "deepsea-users-bounces at lists.suse.com on behalf of Alejandro Bonilla" wrote: Robert, I think that ?error? is normal. Just try adding widgets and see if the cluster shows up on the field as ?ceph?? Also, fix the WARNS too? If that doesn?t do it make sure of: ls -l /etc/ceph/ total 16 -rw------- 1 root root 129 Jan 24 23:15 ceph.client.admin.keyring -rw-rw---- 1 openattic openattic 111 Jan 24 23:17 ceph.client.openattic.keyring -rw-r--r-- 1 root root 362 Jan 24 23:15 ceph.conf DeepSea should have done that. > On Mar 15, 2017, at 8:33 AM, Robert Grosschopff wrote: > > Yepp, apart from the usual ?too many PGs per OSD? everything is fine on the command line. > > openATTIC also shows all OSDs, Pools, Nodes, etc. > > > > On 15/03/2017, 13:30, "deepsea-users-bounces at lists.suse.com on behalf of Lars Marowsky-Bree" wrote: > > On 2017-03-15T12:28:02, Robert Grosschopff wrote: > >> I recently installed a cluster using Salt. Everything was fine and I could see a graphical representation of the Crushmap in openATTIC. >> >> After a reboot of the cluster I keep getting a ?Sorra folks ? no ceph clusters available" . > > Is the cluster itself running? > > > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users > > > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users _______________________________________________ Deepsea-users mailing list Deepsea-users at lists.suse.com http://lists.suse.com/mailman/listinfo/deepsea-users From abonilla at suse.com Wed Mar 15 09:41:04 2017 From: abonilla at suse.com (Alejandro Bonilla) Date: Wed, 15 Mar 2017 15:41:04 +0000 Subject: [Deepsea-users] "no ceph clusters available" In-Reply-To: <59B7C066-2264-42A6-951A-6A5B6F87C475@suse.com> References: <20170315123044.qsww5tswnvqhu755@suse.com> <76DB9AC7-CBAE-4ACC-A3D9-01A60C679003@suse.com> <60608E90-EEDF-4176-9BB5-D162625780CB@suse.com> <59B7C066-2264-42A6-951A-6A5B6F87C475@suse.com> Message-ID: <5BC4EAF6-F2F0-496C-89AC-5A0E88D9F6F0@suse.com> Robert, You need to fix the Warning and then it won?t give out the confusing error msg... > On Mar 15, 2017, at 10:40 AM, Robert Grosschopff wrote: > > Hi Alejandro, > > adding widgets works like a charm ( > > All other files in /etc/ceph are present as well. > > I can also extract the crushmap using ceph osd getcrushmap ?o cmap.bin > > Weird, somehow > > Robert > > > On 15/03/2017, 15:10, "deepsea-users-bounces at lists.suse.com on behalf of Alejandro Bonilla" wrote: > > Robert, > > I think that ?error? is normal. Just try adding widgets and see if the cluster shows up on the field as ?ceph?? > > Also, fix the WARNS too? If that doesn?t do it make sure of: > > ls -l /etc/ceph/ > total 16 > -rw------- 1 root root 129 Jan 24 23:15 ceph.client.admin.keyring > -rw-rw---- 1 openattic openattic 111 Jan 24 23:17 ceph.client.openattic.keyring > -rw-r--r-- 1 root root 362 Jan 24 23:15 ceph.conf > > DeepSea should have done that. > >> On Mar 15, 2017, at 8:33 AM, Robert Grosschopff wrote: >> >> Yepp, apart from the usual ?too many PGs per OSD? everything is fine on the command line. >> >> openATTIC also shows all OSDs, Pools, Nodes, etc. >> >> >> >> On 15/03/2017, 13:30, "deepsea-users-bounces at lists.suse.com on behalf of Lars Marowsky-Bree" wrote: >> >> On 2017-03-15T12:28:02, Robert Grosschopff wrote: >> >>> I recently installed a cluster using Salt. Everything was fine and I could see a graphical representation of the Crushmap in openATTIC. >>> >>> After a reboot of the cluster I keep getting a ?Sorra folks ? no ceph clusters available" . >> >> Is the cluster itself running? >> >> >> _______________________________________________ >> Deepsea-users mailing list >> Deepsea-users at lists.suse.com >> http://lists.suse.com/mailman/listinfo/deepsea-users >> >> >> _______________________________________________ >> Deepsea-users mailing list >> Deepsea-users at lists.suse.com >> http://lists.suse.com/mailman/listinfo/deepsea-users > > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users > > > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users From Robert.Grosschopff at suse.com Thu Mar 16 05:49:15 2017 From: Robert.Grosschopff at suse.com (Robert Grosschopff) Date: Thu, 16 Mar 2017 11:49:15 +0000 Subject: [Deepsea-users] "no ceph clusters available" In-Reply-To: <5BC4EAF6-F2F0-496C-89AC-5A0E88D9F6F0@suse.com> References: <20170315123044.qsww5tswnvqhu755@suse.com> <76DB9AC7-CBAE-4ACC-A3D9-01A60C679003@suse.com> <60608E90-EEDF-4176-9BB5-D162625780CB@suse.com> <59B7C066-2264-42A6-951A-6A5B6F87C475@suse.com> <5BC4EAF6-F2F0-496C-89AC-5A0E88D9F6F0@suse.com> Message-ID: Alejandro, the Crushmap still doesn't show up after fixing the warning about having too many PGs. That warning appeared right after SES was installed because some initial pools are created with a standard number of PGs unrelated to the number of OSDs. It worked at that time. After rebooting the cluster the Nodes tab stays empty (apart from the title ?Ceph Hosts?) for whatever reason. Robert On 15/03/2017, 16:41, "deepsea-users-bounces at lists.suse.com on behalf of Alejandro Bonilla" wrote: Robert, You need to fix the Warning and then it won?t give out the confusing error msg... > On Mar 15, 2017, at 10:40 AM, Robert Grosschopff wrote: > > Hi Alejandro, > > adding widgets works like a charm ( > > All other files in /etc/ceph are present as well. > > I can also extract the crushmap using ceph osd getcrushmap ?o cmap.bin > > Weird, somehow > > Robert > > > On 15/03/2017, 15:10, "deepsea-users-bounces at lists.suse.com on behalf of Alejandro Bonilla" wrote: > > Robert, > > I think that ?error? is normal. Just try adding widgets and see if the cluster shows up on the field as ?ceph?? > > Also, fix the WARNS too? If that doesn?t do it make sure of: > > ls -l /etc/ceph/ > total 16 > -rw------- 1 root root 129 Jan 24 23:15 ceph.client.admin.keyring > -rw-rw---- 1 openattic openattic 111 Jan 24 23:17 ceph.client.openattic.keyring > -rw-r--r-- 1 root root 362 Jan 24 23:15 ceph.conf > > DeepSea should have done that. > >> On Mar 15, 2017, at 8:33 AM, Robert Grosschopff wrote: >> >> Yepp, apart from the usual ?too many PGs per OSD? everything is fine on the command line. >> >> openATTIC also shows all OSDs, Pools, Nodes, etc. >> >> >> >> On 15/03/2017, 13:30, "deepsea-users-bounces at lists.suse.com on behalf of Lars Marowsky-Bree" wrote: >> >> On 2017-03-15T12:28:02, Robert Grosschopff wrote: >> >>> I recently installed a cluster using Salt. Everything was fine and I could see a graphical representation of the Crushmap in openATTIC. >>> >>> After a reboot of the cluster I keep getting a ?Sorra folks ? no ceph clusters available" . >> >> Is the cluster itself running? >> >> >> _______________________________________________ >> Deepsea-users mailing list >> Deepsea-users at lists.suse.com >> http://lists.suse.com/mailman/listinfo/deepsea-users >> >> >> _______________________________________________ >> Deepsea-users mailing list >> Deepsea-users at lists.suse.com >> http://lists.suse.com/mailman/listinfo/deepsea-users > > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users > > > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users _______________________________________________ Deepsea-users mailing list Deepsea-users at lists.suse.com http://lists.suse.com/mailman/listinfo/deepsea-users From Robert.Grosschopff at suse.com Thu Mar 16 06:55:49 2017 From: Robert.Grosschopff at suse.com (Robert Grosschopff) Date: Thu, 16 Mar 2017 12:55:49 +0000 Subject: [Deepsea-users] Adding OSDs using Salt Message-ID: <4E7AB04A-74A0-4E5A-9C78-7182C6AAA8A9@suse.com> *, I added OSDs using Salt the following way: - Add disks to system - Run stage.1 - Modify policy.cfg o add profile-NEWDISK/cluster/OSD*.sls o add profile-NEWDISK/stack/default/ceph/minions/OSD*.yml o REMOVE old profile-OLDDISK/cluster/OSD*.sls o REMOVE old profile-OLDDISK/stack/default/ceph/minions /OSD*.yml - Run stage.2 - Run stage.3 If the old profiles are not removed 'salt \* pillar.items' will have add the old OSD profiles again. Is this the way it is supposed to be done ? Robert From swamireddy at gmail.com Thu Mar 16 07:52:27 2017 From: swamireddy at gmail.com (M Ranga Swami Reddy) Date: Thu, 16 Mar 2017 19:22:27 +0530 Subject: [Deepsea-users] Fwd: Use the deepsea on ubuntu In-Reply-To: References: Message-ID: Hello, Can I use the deepsea on ubuntu OS? Thanks Swami -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Grosschopff at suse.com Thu Mar 16 09:13:39 2017 From: Robert.Grosschopff at suse.com (Robert Grosschopff) Date: Thu, 16 Mar 2017 15:13:39 +0000 Subject: [Deepsea-users] Adding OSDs using Salt In-Reply-To: <5425463.x1tF534UTD@ruby> References: <4E7AB04A-74A0-4E5A-9C78-7182C6AAA8A9@suse.com> <5425463.x1tF534UTD@ruby> Message-ID: <1489677217.5543.13.camel@suse.com> What I did at first was to simply add the new profile (..2Disk50GB...) in profile.cfg : profile-1Disk50GB-1/cluster/ses4-[1234]*.sls profile-1Disk50GB-1/stack/default/ceph/minions/ses4-[1234]*.yml profile-2Disk50GB-1/cluster/ses4-[1234]*.sls profile-2Disk50GB-1/stack/default/ceph/minions/ses4-[1234]*.yml Then I ran stage.2 : Succeeded: 12 (changed=4) Failed:?????0 salt '*' pillar.items shows : ? ? storage: ????????---------- ????????data+journals: ????????osds: ????????????- /dev/vdb ????????????- /dev/vdb ????????????- /dev/vdc So, vdb was added again. stage.3 throws a lot of failure messages since it cannot add vdb again: cephadm at salt:~> sudo salt-run state.orch ceph.stage.3 firewall?????????????????: disabled fsid?????????????????????: valid public_network???????????: valid public_interface?????????: valid cluster_network??????????: valid cluster_interface????????: valid monitors?????????????????: valid storage??????????????????: valid master_role??????????????: valid mon_host?????????????????: valid mon_initial_members??????: valid time_server??????????????: disabled fqdn?????????????????????: valid [WARNING ] Could not write out jid file for job 20170316122026087326. Retrying. [WARNING ] Could not write out jid file for job 20170316122026087326. Retrying. [WARNING ] Could not write out jid file for job 20170316122026087326. Retrying. [WARNING ] Could not write out jid file for job 20170316122026087326. Retrying. [WARNING ] Could not write out jid file for job 20170316122026087326. Retrying. [ERROR???] prep_jid could not store a jid after 5 tries. [ERROR???] Could not store job cache info. Job details for this run may be unavailable. [ERROR???] Run failed on minions: ses4-3.local.site, ses4-4.local.site, ses4-1.local.site, ses4-2.local.site Failures: ????ses4-3.local.site: ????????Data failed to compile: ????---------- ????????Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID 'prepare /dev/vdb' ????ses4-4.local.site: ????????Data failed to compile: ????---------- ????????Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID 'prepare /dev/vdb' ????ses4-1.local.site: ????????Data failed to compile: ????---------- ????????Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID 'prepare /dev/vdb' ????ses4-2.local.site: ????????Data failed to compile: ????---------- ????????Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID 'prepare /dev/vdb' salt.local.site_master: ? Name: packages - Function: salt.state - Result: Clean Started: - 12:20:27.760260 Duration: 1259.428 ms ? Name: configuration check - Function: salt.state - Result: Clean Started: - 12:20:29.019844 Duration: 177.061 ms ? Name: configuration - Function: salt.state - Result: Clean Started: - 12:20:29.197064 Duration: 598.674 ms ? Name: admin - Function: salt.state - Result: Clean Started: - 12:20:29.795890 Duration: 190.272 ms ? Name: monitors - Function: salt.state - Result: Changed Started: - 12:20:29.986315 Duration: 454.657 ms ? Name: osd auth - Function: salt.state - Result: Changed Started: - 12:20:30.441126 Duration: 332.438 ms ---------- ??????????ID: storage ????Function: salt.state ??????Result: False ?????Comment: Run failed on minions: ses4-3.local.site, ses4- 4.local.site, ses4-1.local.site, ses4-2.local.site ??????????????Failures: ??????????????????ses4-3.local.site: ??????????????????????Data failed to compile: ??????????????????---------- ??????????????????????Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID 'prepare /dev/vdb' ??????????????????ses4-4.local.site: ??????????????????????Data failed to compile: ??????????????????---------- ??????????????????????Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID 'prepare /dev/vdb' ??????????????????ses4-1.local.site: ??????????????????????Data failed to compile: ??????????????????---------- ??????????????????????Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID 'prepare /dev/vdb' ??????????????????ses4-2.local.site: ??????????????????????Data failed to compile: ??????????????????---------- ??????????????????????Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID 'prepare /dev/vdb' ?????Started: 12:20:30.773721 ????Duration: 396.396 ms ?????Changes: Summary for salt.local.site_master ------------ Succeeded: 6 (changed=2) Failed:????1 ------------ Total states run:?????7 Total run time:???3.409 s So I had to remove the old entries to get Salt to add OSDs to my existing nodes. I would have expected that I can simply add the new disks and Salt will notice that some disks already exist and just set up the new ones. Now I am wondering what will happen if I integrate a new identical OSD- Node. policy.cfg would need to have both disk profiles. ?pillar.items would show duplicate disk entries for the existing nodes again and stage.3 would fail. Robert On Thu, 2017-03-16 at 10:14 -0400, Eric Jackson wrote: > On Thursday, March 16, 2017 12:55:49 PM Robert Grosschopff wrote: > > > > *, > > > > I added OSDs using Salt the following way: > > > > - Add disks to system > > - Run stage.1 > > - Modify policy.cfg > > o add profile-NEWDISK/cluster/OSD*.sls > > o add profile-NEWDISK/stack/default/ceph/minions/OSD*.yml > > o REMOVE old profile-OLDDISK/cluster/OSD*.sls > > o REMOVE old profile-OLDDISK/stack/default/ceph/minions /OSD*.yml > > - Run stage.2 > > - Run stage.3 > > > > If the old profiles are not removed??'salt \* pillar.items' will > > have add > > the old OSD profiles again. > > > > Is this the way it is supposed to be done ? > > Since you modify policy.cfg to use the profile-NEWDISK, you do not > need to? > remove the old profiles.??However, if you have no machines that will > ever match? > them again and want to clean up, there's no harm. > > Does the new profile contain all the disks as OSDs in the way you > wanted???If? > so, do exactly what you did.??Stage 3 will see that the existing OSDs > are? > already done and move on to adding the blank drives as additional > OSDs. > > If the new profile is not a simple addition of the existing disks > (maybe you? > replaced smaller disks and added additional disks), then removing the > node is? > the simpler alternative.??That is, > > 1) Remove/comment out the node from policy.cfg > 2) Run Stages 2-5? > 3) Add the node back with new profile > > Depending on your situation, you can take that as fast or as slow as? > necessary.??That is, do all the storage nodes you physically changed > or do? > them one at a time.? > > > > > > Robert > > > > _______________________________________________ > > Deepsea-users mailing list > > Deepsea-users at lists.suse.com > > http://lists.suse.com/mailman/listinfo/deepsea-users > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users From Robert.Grosschopff at suse.com Thu Mar 16 09:41:12 2017 From: Robert.Grosschopff at suse.com (Robert Grosschopff) Date: Thu, 16 Mar 2017 15:41:12 +0000 Subject: [Deepsea-users] Adding OSDs using Salt In-Reply-To: <2006419.BZ5kqsY0mz@ruby> References: <4E7AB04A-74A0-4E5A-9C78-7182C6AAA8A9@suse.com> <5425463.x1tF534UTD@ruby> <1489677217.5543.13.camel@suse.com> <2006419.BZ5kqsY0mz@ruby> Message-ID: OK. User Brain Error. Works as expected. Thanks for clarifying :) On 16/03/2017, 16:36, "deepsea-users-bounces at lists.suse.com on behalf of Eric Jackson" wrote: On Thursday, March 16, 2017 03:13:39 PM Robert Grosschopff wrote: > What I did at first was to simply add the new profile (..2Disk50GB...) > in profile.cfg : > > profile-1Disk50GB-1/cluster/ses4-[1234]*.sls > profile-1Disk50GB-1/stack/default/ceph/minions/ses4-[1234]*.yml > profile-2Disk50GB-1/cluster/ses4-[1234]*.sls > profile-2Disk50GB-1/stack/default/ceph/minions/ses4-[1234]*.yml I misunderstood. Do not include two profiles for the same host. Remove the profile-1Disk50GB lines from your policy.cfg. Your pillar.items will be correct then. > > Then I ran stage.2 : > > Succeeded: 12 (changed=4) > Failed: 0 > > salt '*' pillar.items shows : > > storage: > ---------- > data+journals: > osds: > - /dev/vdb > - /dev/vdb > - /dev/vdc > > So, vdb was added again. > > stage.3 throws a lot of failure messages since it cannot add vdb again: > > cephadm at salt:~> sudo salt-run state.orch ceph.stage.3 > firewall : disabled > fsid : valid > public_network : valid > public_interface : valid > cluster_network : valid > cluster_interface : valid > monitors : valid > storage : valid > master_role : valid > mon_host : valid > mon_initial_members : valid > time_server : disabled > fqdn : valid > [WARNING ] Could not write out jid file for job 20170316122026087326. > Retrying. > [WARNING ] Could not write out jid file for job 20170316122026087326. > Retrying. > [WARNING ] Could not write out jid file for job 20170316122026087326. > Retrying. > [WARNING ] Could not write out jid file for job 20170316122026087326. > Retrying. > [WARNING ] Could not write out jid file for job 20170316122026087326. > Retrying. > [ERROR ] prep_jid could not store a jid after 5 tries. > [ERROR ] Could not store job cache info. Job details for this run may > be unavailable. > [ERROR ] Run failed on minions: ses4-3.local.site, ses4-4.local.site, > ses4-1.local.site, ses4-2.local.site > Failures: > ses4-3.local.site: > Data failed to compile: > ---------- > Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID > 'prepare /dev/vdb' > ses4-4.local.site: > Data failed to compile: > ---------- > Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID > 'prepare /dev/vdb' > ses4-1.local.site: > Data failed to compile: > ---------- > Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID > 'prepare /dev/vdb' > ses4-2.local.site: > Data failed to compile: > ---------- > Rendering SLS 'base:ceph.osd.default' failed: Conflicting ID > 'prepare /dev/vdb' > > salt.local.site_master: > Name: packages - Function: salt.state - Result: Clean Started: - > 12:20:27.760260 Duration: 1259.428 ms > Name: configuration check - Function: salt.state - Result: Clean > Started: - 12:20:29.019844 Duration: 177.061 ms > Name: configuration - Function: salt.state - Result: Clean Started: - > 12:20:29.197064 Duration: 598.674 ms > Name: admin - Function: salt.state - Result: Clean Started: - > 12:20:29.795890 Duration: 190.272 ms > Name: monitors - Function: salt.state - Result: Changed Started: - > 12:20:29.986315 Duration: 454.657 ms > Name: osd auth - Function: salt.state - Result: Changed Started: - > 12:20:30.441126 Duration: 332.438 ms > ---------- > ID: storage > Function: salt.state > Result: False > Comment: Run failed on minions: ses4-3.local.site, ses4- > 4.local.site, ses4-1.local.site, ses4-2.local.site > Failures: > ses4-3.local.site: > Data failed to compile: > ---------- > Rendering SLS 'base:ceph.osd.default' failed: > Conflicting ID 'prepare /dev/vdb' > ses4-4.local.site: > Data failed to compile: > ---------- > Rendering SLS 'base:ceph.osd.default' failed: > Conflicting ID 'prepare /dev/vdb' > ses4-1.local.site: > Data failed to compile: > ---------- > Rendering SLS 'base:ceph.osd.default' failed: > Conflicting ID 'prepare /dev/vdb' > ses4-2.local.site: > Data failed to compile: > ---------- > Rendering SLS 'base:ceph.osd.default' failed: > Conflicting ID 'prepare /dev/vdb' > Started: 12:20:30.773721 > Duration: 396.396 ms > Changes: > > Summary for salt.local.site_master > ------------ > Succeeded: 6 (changed=2) > Failed: 1 > ------------ > Total states run: 7 > Total run time: 3.409 s > > So I had to remove the old entries to get Salt to add OSDs to my > existing nodes. > > I would have expected that I can simply add the new disks and Salt will > notice that some disks already exist and just set up the new ones. > > Now I am wondering what will happen if I integrate a new identical OSD- > Node. policy.cfg would need to have both disk profiles. pillar.items > would show duplicate disk entries for the existing nodes again and > stage.3 would fail. > > Robert > > On Thu, 2017-03-16 at 10:14 -0400, Eric Jackson wrote: > > > On Thursday, March 16, 2017 12:55:49 PM Robert Grosschopff wrote: > > > > > > > > *, > > > > > > I added OSDs using Salt the following way: > > > > > > - Add disks to system > > > - Run stage.1 > > > - Modify policy.cfg > > > o add profile-NEWDISK/cluster/OSD*.sls > > > o add profile-NEWDISK/stack/default/ceph/minions/OSD*.yml > > > o REMOVE old profile-OLDDISK/cluster/OSD*.sls > > > o REMOVE old profile-OLDDISK/stack/default/ceph/minions /OSD*.yml > > > - Run stage.2 > > > - Run stage.3 > > > > > > If the old profiles are not removed 'salt \* pillar.items' will > > > have add > > > the old OSD profiles again. > > > > > > Is this the way it is supposed to be done ? > > > > > > Since you modify policy.cfg to use the profile-NEWDISK, you do not > > need to > > remove the old profiles. However, if you have no machines that will > > ever match > > them again and want to clean up, there's no harm. > > > > Does the new profile contain all the disks as OSDs in the way you > > wanted? If > > so, do exactly what you did. Stage 3 will see that the existing OSDs > > are > > already done and move on to adding the blank drives as additional > > OSDs. > > > > If the new profile is not a simple addition of the existing disks > > (maybe you > > replaced smaller disks and added additional disks), then removing the > > node is > > the simpler alternative. That is, > > > > 1) Remove/comment out the node from policy.cfg > > 2) Run Stages 2-5 > > 3) Add the node back with new profile > > > > Depending on your situation, you can take that as fast or as slow as > > necessary. That is, do all the storage nodes you physically changed > > or do > > them one at a time. > > > > > > > > > > > Robert > > > > > > _______________________________________________ > > > Deepsea-users mailing list > > > Deepsea-users at lists.suse.com > > > http://lists.suse.com/mailman/listinfo/deepsea-users > > > > _______________________________________________ > > Deepsea-users mailing list > > Deepsea-users at lists.suse.com > > http://lists.suse.com/mailman/listinfo/deepsea-users > > _______________________________________________ > Deepsea-users mailing list > Deepsea-users at lists.suse.com > http://lists.suse.com/mailman/listinfo/deepsea-users From dbyte at suse.com Mon Mar 20 07:47:50 2017 From: dbyte at suse.com (David Byte) Date: Mon, 20 Mar 2017 13:47:50 +0000 Subject: [Deepsea-users] detection of public & cluster networks In-Reply-To: <58CFDE8E0200001C002D77AD@prv-mh.provo.novell.com> References: <58CFDE8E0200001C002D77AD@prv-mh.provo.novell.com> <58CFDE8E0200001C002D77AD@prv-mh.provo.novell.com> Message-ID: Just curious if there is a default gateway defined on any of the networks? David Byte Sr. Technology Strategist Alliances and SUSE Embedded dbyte at suse.com 918.528.4422 From: on behalf of Martin Weiss Reply-To: Discussions about the DeepSea management framework for Ceph Date: Monday, March 20, 2017 at 7:52 AM To: "deepsea-users at lists.suse.com" Subject: [Deepsea-users] detection of public & cluster networks Hi *, it seems that deepSea does not detect the networks as required and we have to adjust them in /srv/pillar/ceph/proposals/config/stack/default/ceph/cluster.yml. However - once adjusted them, there I can see that also the public network interfaces in /srv/pillar/ceph/proposals/role-mon/stack/default/ceph/minions are wrong (it seems it just selects the alphabetical first IP). So do I get it right that after stage.2 all the IPs for the minions yml and the /srv/pillar/ceph/proposals/config/stack/default/ceph/cluster.yml need to be adjusted manually or are there further changes required before executing stage.3? Some more background: each node has three networks: 172.17.2.0/24 = public, 172.17.1.0/24 = cluster and 172.17.3.0/24 = management. - deepSea automatically selects 172.17.1.0/24 as public where it should use 172.17.2.0/24 as public. Thanks Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdias at suse.com Tue Mar 21 05:14:36 2017 From: rdias at suse.com (Ricardo Dias) Date: Tue, 21 Mar 2017 11:14:36 +0000 Subject: [Deepsea-users] Single configuration approach for DeepSea Message-ID: <66c96180-24b8-c1b4-b01e-7ef9c9ba112b@suse.com> Hi, I've been developing an alternative approach to the current "policy.cfg" to configure the deployment of Ceph with DeepSea. The implementation resides in my own fork of DeepSea in branch wip-single-config: https://github.com/rjfd/DeepSea/tree/wip-single-config The main idea is to use a single configuration file (single source of truth) in the pillar, written in yaml, that can specify several deployment properties and customize specific options for the several Ceph components, such as OSDs, MONs, etc... This configuration file can be created before any stage 0 of DeepSea because it does not rely on any information generated during stages 0..5. An example of this configuration file can be seen in: https://github.com/rjfd/DeepSea/blob/wip-single-config/srv/pillar/ceph/config.sls The structure in the above file is a proposal and can be (and should be) subject to changes. The include/exclude sets that appear in several section of the config file contain minion names and support gobling. These sets only take effect from stage 1 onwards. Stage 0 is still not supported. Inside the "osds" section, we can specify options that affect the discovery algorithm of stage 1, and we can also specify OSD properties, such as "journal_size" or the use of "dmcrypt". You can specify the OSD properties in three different scopes: - global: it applies to all OSDs in the cluster - minion: it applies to all OSDs in a minion - drive: it applies only to the OSD using that drive The properties specified in the inner scope will override the properties from the outer scope. Regarding the way we can identify the drive/device we can use the device name (with support of globs), or use any other way that can identify each drive, such as vendor, or model. The current implementation only supports the device name. iscsi, rgw, mds, cephfs sections are not supported yet, since this is just a prototype of an idea to check if it's worth to pursue this approach. Another section that isn't still supported is the specification of the public/cluster networks. To use this single config file approach, just checkout the code from my fork, run "make install" to install DeepSea. Create the config file based on the example that is already in the repository (notice that "make install" will not copy the config file sample) in /srv/pillar/ceph/config.sls. Then run stages 0..3 as usual. When running stage 1, it will output the information of OSDs, MONs, networks that will be deployed in the following stages. If you find that the configuration does not fit your needs, you can make changes to the config file and run stage 1 again. This implementation only supports running stages up to stage 3, therefore it only supports deploying a Ceph cluster. It doesn't allow you to deploy iSCSI gateways, nor CephFS for instance. Please note that this is just an idea that I'm exploring, and it is not even planned to be included in DeepSea yet. Moreover, the implementation was only tested in a virtual environment, and may contain a lot of bugs. I'm looking for people to experiment with it and make suggestions on how to improve it. Thanks, Ricardo Dias From Robert.Grosschopff at suse.com Wed Mar 29 04:50:04 2017 From: Robert.Grosschopff at suse.com (Robert Grosschopff) Date: Wed, 29 Mar 2017 10:50:04 +0000 Subject: [Deepsea-users] Setting up RGW using Salt Message-ID: <8B7851D2-88FD-4F5D-AFB6-83D06E0AF133@suse.com> I assigned the rgw role to one of my systems and I can create S3 users and Swift subusers. However, no HTTP server is started or installed. Shouldn't there also be a ceph.client.rgw.keyring be present as well ? Robert