[Deepsea-users] Antw: Re: Fwd: Detecting disks by their properties rather than by their path

David Byte dbyte at suse.com
Fri Jan 6 13:10:35 MST 2017

I completed a version and then failed to upload it to my github (github.com/dmbyte/SES-scripts).  You are looking for the do-osds.sh.  I’ll keep looking and see if I can find the finished version and upload it to my github.  You can take a look at what’s there and maybe figure out the continuation in the meantime.

David Byte
Sr. Technology Strategist
Alliances and SUSE Embedded
dbyte at suse.com

From: <deepsea-users-bounces at lists.suse.com> on behalf of Martin Weiss <Martin.Weiss at suse.com>
Reply-To: Discussions about the DeepSea management framework for Ceph <deepsea-users at lists.suse.com>
Date: Thursday, December 22, 2016 at 6:26 AM
To: "deepsea-users at lists.suse.com" <deepsea-users at lists.suse.com>
Subject: [Deepsea-users] Antw: Re: Fwd: Detecting disks by their properties rather than by their path

Cool thing - would be great if you could share this script so that we can give that a try in different deployments ;-)

Just one additional point - in case the servers have multiple HBAs - building the OSDs and also later on the crush map might also require the info which disk is connected to which HBA on which port..


I wrote a script about 6 months ago that uses lsblk output to compare the configuration of storage nodes. This includes device manufacturer, model, rotational speed and maybe a few other attributes. This is enough on most systems to get you pretty close.  I've been doing some additional work on a script that also picks up the connection type and speed (12G SAS for example).

With all of this data, it would make it easy to group the disks by type and if the ratios are within bounds, you could make it easy to say use group B as journals for group A. Group C should be in its own bucket, etc.

David Byte
Sr. Technical Strategist
IHV Alliances and Embedded

Sent from my iPhone. Typos are Apple's fault.

> On Dec 21, 2016, at 6:31 AM, Eric Jackson <ejackson at suse.com> wrote:
> Hi Lenz,
>  As Lars mentioned, this has been an ongoing conversation.  Currently,
> DeepSea is using the long list, but at least it's generated by Salt.  At some
> point, the list does need to exist on the minion during the creation of OSDs.
> Now, that could be a Salt module instead of the Salt pillar, effectively
> generating the list dynamically.
>  I only have a couple of concerns with no solution at the time.  How do I
> trivially know what models I have?  I most likely cannot say simply Samsung or
> Intel, but will need to collect all the alpha-numeric numbers of every drive
> on all nodes.  Ideally, I'd like to believe that it's only a few at a given
> time in the life of a Ceph cluster, but over the course of a couple of years,
> I could see that list being rather lengthy as well with replacements and
> upgrades.
>  Another concern is debugging.  While a hardcoded list of devices in the Salt
> pillar is not elegant, there's no moving parts.  If a device in the list is
> not an OSD, the process is currently how did ceph-disk fail.  Using a Salt
> module that accepts various filters to return a list may not be seen as an
> improvement over a static file when trying to determine why a drive is not an
> OSD.
>  The last issue I haven't had enough time to think about entirely, but I
> wonder if temporary failure conditions will create any unintended side effects.
> Currently, the Salt pillar is authoritative.  If the list becomes ephemeral
> and changes between runs (not due to actual intended changes, but more like
> flaky hardware), does that make any other operations more difficult?
>  What would you think of a filter based solution that generates the list as a
> static file for each storage node?  This would mimic the current behavior of
> keyrings in DeepSea.  It's a two step process.  The keyrings are generated and
> stored on the Salt master.  The second step includes adding the keyring to
> Ceph and distributing it to the correct roles.  That keeps debugging simple
> and avoids regenerating keyrings unnecessarily.  I think this addresses the
> second and third issue above.  Additionally, the static file may provide a
> history via version control.  That could prove useful for sysadmin teams
> knowing the previously detected hardware (e.g. Did somebody change the filter
> for this node or did they change the hardware?)
>  These lists would still feed into the generation of hardware profiles unless
> the goal is not only to include a filter, but have the admin provide a desired
> profile.  Maybe only adding the filter would be a sufficient first step?  I am
> believing that the default filter would be all available devices as it is now.
> Eric
>> On Wednesday, December 21, 2016 09:46:48 AM Lenz Grimmer wrote:
>> Hi all,
>> stumbled over this idea in the ceph-ansible mailing list and found it
>> quite interesting. Is there some merit in being able to select disks
>> based on this kind information? How does DeepSea handle this?
>> Lenz
>> -------- Forwarded Message --------
>> Subject: [Ceph-ansible] Detecting disks by their properties rather than
>> by their path
>> Date: Thu, 8 Dec 2016 16:51:26 -0500 (EST)
>> From: Erwan Velu <evelu at redhat.com>
>> To: Ceph-ansible at lists.ceph.com, lgrimmer at suse.de
>> Hi list,
>> This is my first contribution to ceph-ansible and wanted to share my
>> idea with you and get preliminary feedbacks on it.
>> First time I saw ceph-ansible, I was surprised on the way disks are
>> defined by naming disks per node with their path like "/dev/sdx"
>> To my low-level background this is a big issue as :
>> - if you have 20-30 disks per node that make a serious list to maintain
>> - doesn't garantee what device you select : does /dev/sdx is a usb key,
>> an SSD or a rotational drive ?
>> - name can change over time : inserting a new disk or rebooting can lead
>> to a different name
>> My first approach was about using /dev/disk/by-id/ paths but that tigger
>> the following issues :
>> - still need a serious list of devices, which is even longer ...
>> - name integrate serial number or hexa strings making difficult to
>> maintain it, every device is different
>> I eneded up with the following idea:
>> - Why should'nt we choose the disks by their features and let a module
>> find the associated path.
>> The syntax looks like :
>> "{'samsung_journal_disks': {'model': 'SAMSUNG MZ7LN512', 'rotational':
>> '0', 'count': 2, 'ceph_type': 'journal' }}"
>> With this syntax, you can select disk by the model, vendor, size, SSD or
>> not.
>> Then you can specify to get more of that type by specifing the count'
>> number : having 10 disks of the same type doesn't make the definition longer
>> It is then possible to define what disks are "journals" or "data" by
>> defining the ceph_type attribute.
>> If that definition match the actual system, the module returns the
>> associated /dev/disk/by-id/ path like :
>> samsung_journal_disks_000 :
>> /dev/disk/by-id/scsi-36848f690e68a50001e428e511e4a6c20
>> samsung_journal_disks_000 :
>> /dev/disk/by-id/scsi-36848f690e68a50001e428e521e55c62b
>> The real benefit of that is the disks path become the result of a search
>> and not a starting point. The disk path have very few value then.
>> Ceph-ansible will be able to select what disks should be used for what :
>> this part is under work.
>> I wrote a documentation about it, you can have an overview here :
>> https://gist.github.com/ErwanAliasr1/b19c62fac061f4f924f587b1973cf0ea

>> All this work can be found in https://github.com/ceph/ceph-ansible/pull/1128

>> I didn't detailled everything in this first mail to avoid being too verbose.
>> I'd love to get your feedbacks on that idea :
>> - does it solve issues you already had ?
>> - could it be useful for you ?
>> Cheers,
>> Erwan
>> _______________________________________________
>> Ceph-ansible mailing list
>> Ceph-ansible at lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-ansible-ceph.com

> _______________________________________________
> Deepsea-users mailing list
> Deepsea-users at lists.suse.com
> http://lists.suse.com/mailman/listinfo/deepsea-users

Deepsea-users mailing list
Deepsea-users at lists.suse.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.suse.com/pipermail/deepsea-users/attachments/20170106/14132f54/attachment.htm>

More information about the Deepsea-users mailing list