[caasp-beta] BTRFS space and quota]

Ludovic Cavajani ludovic.cavajani at suse.com
Thu Nov 2 09:46:53 MDT 2017


Hello Stéphane,

Can you provide us the output of :

# du -csh /*

Regards,

On 11/02/2017 11:54 AM, Paul Gonin wrote:
> -------- Message transféré --------
>
> *Date*: Thu, 2 Nov 2017 10:35:13 +0000
> *Objet*: Re: [caasp-beta] BTRFS space and quota
> *À*: Paul Gonin <paul.gonin at suse.com
> <mailto:Paul%20Gonin%20%3cpaul.gonin at suse.com%3e>>,
> caasp-beta at lists.suse.com <caasp-beta at lists.suse.com
> <mailto:%22caasp-beta at lists.suse.com%22%20%3ccaasp-beta at lists.suse.com%3e>>
> *De*: Le Bihan Stéphane (AMUNDI-ITS) <stephane.lebihan at amundi.com
> <mailto:Le%20Bihan%20%3d%3fISO-8859-1%3fQ%3fSt%3dE9phane%3f%3d%20%22%28AMUNDI-ITS%29%22%20%3cstephane.lebihan at amundi.com%3e>>
>
> Hi Paul,
>
>  
>
> The result of command snapper ls.
>
>  
>
> # snapper ls
>
> Type   | # | Pre # | Date                            | User | Cleanup
> | Description           | Userdata
>
> -------+---+-------+---------------------------------+------+---------+-----------------------+---------
>
> single | 0 |       |                                 | root |        
> | current               |
>
> single | 1 |       | Fri 06 Oct 2017 08:47:14 AM UTC | root |        
> | first root filesystem |
>
>  
>
> I delete quota on /var/lb/etcd, and test balance but it’s not ok.
>
> I recreate quota and rescan and value is same before deletion.
>
>  
>
> For information I launch du –sh on / and result is 7.8Go.
>
>  
>
> # du -sh /
>
> du: cannot access '/proc/7982/task/7982/fd/4': No such file or directory
>
> du: cannot access '/proc/7982/task/7982/fdinfo/4': No such file or
> directory
>
> du: cannot access '/proc/7982/fd/3': No such file or directory
>
> du: cannot access '/proc/7982/fdinfo/3': No such file or directory
>
> 7.8G    /
>
>  
>
> Regards,
>
>  
>
>  
>
> *Stéphane Le Bihan *
>
> SDE/DSI/IPR/SSD/UNX
>
> 90, Boulevard Pasteur - 75015 Paris
>
> *Web: http://www.amundi.com <http://www.amundi.com/>*
>
> Tél: +33 1 76 32 32 08
>
> Equipe Unix : +33 1 76 32 02 30
>
> @: stephane.lebihan at amundi.com <mailto:stephane.lebihan at amundi.com>
>
> @ : sits.unix at amundi.com <mailto:sits.unix at amundi.com>
>
>  
>
>  
>
> *De :*Paul Gonin [mailto:paul.gonin at suse.com]
> *Envoyé :* jeudi 2 novembre 2017 10:55
> *À :* Le Bihan Stéphane (AMUNDI-ITS); caasp-beta at lists.suse.com
> *Objet :* Re: [caasp-beta] BTRFS space and quota
>
>  
>
> Hi Stephane,
>
>  
>
> What is the output of
>
> # snapper ls
>
> ?
>
>  
>
> I assume that since you there were no updates yet it should look like
>
>  
>
> Type   | # | Pre # | Date                     | User | Cleanup |
> Description           | Userdata     
>
> -------+---+-------+--------------------------+------+---------+-----------------------+--------------
>
> single | 0 |       |                          | root |         |
> current               |              
>
> single | 1 |       | Tue Oct 31 09:07:13 2017 | root |         | first
> root filesystem |              
>
> single | 2 |       | Tue Oct 31 09:10:42 2017 | root | number  | after
> installation    | important=yes
>
>  
>
> rgds
>
> Paul
>
>  
>
> Le mardi 31 octobre 2017 à 13:38 +0000, Le Bihan Stéphane (AMUNDI-ITS)
> a écrit :
>
>> Hi Paul,
>>
>>  
>>
>> We work with CaaSP2.
>>
>>  
>>
>> Regards,
>>
>>  
>>
>> *Stéphane Le Bihan *
>>
>> SDE/DSI/IPR/SSD/UNX
>>
>> 90, Boulevard Pasteur - 75015 Paris
>>
>> *Web: http://www.amundi.com <http://www.amundi.com/>*
>>
>> Tél: +33 1 76 32 32 08
>>
>> Equipe Unix : +33 1 76 32 02 30
>>
>> @: stephane.lebihan at amundi.com <mailto:stephane.lebihan at amundi.com>
>>
>> @ : sits.unix at amundi.com <mailto:sits.unix at amundi.com>
>>
>>  
>>
>>  
>>
>> *De :*Paul Gonin [mailto:paul.gonin at suse.com]
>> *Envoyé :* mardi 31 octobre 2017 14:34
>> *À :* Le Bihan Stéphane (AMUNDI-ITS); caasp-beta at lists.suse.com
>> <mailto:caasp-beta at lists.suse.com>
>> *Objet :* Re: [caasp-beta] BTRFS space and quota
>>
>>  
>>
>> Hi Stéphane,
>>
>>  
>>
>> Not that it should make a difference for the issue described, what
>> version of CaaSP the cluster is running ?
>>
>> Is it CaaSP2 ? RC1 ?
>>
>>  
>>
>> thanks
>>
>> Paul
>>
>>  
>>
>> Le mardi 31 octobre 2017 à 08:35 +0000, Le Bihan Stéphane
>> (AMUNDI-ITS) a écrit :
>>
>>> Hello,
>>>
>>>  
>>>
>>> We have a strange case on CAASP plateform with btrfs quota.
>>>
>>>  
>>>
>>> For history, I was out of office since 3 weeks, but others colleague
>>> test kubernetes plateform.
>>>
>>> When I return, we ask me because FS is full on master and worker nodes.
>>>
>>> I don’t have cause, but I think with a bad config, subvolume
>>> /var/lib/etcd grown and  after correction reduce, though quota
>>> reserved all space.
>>>
>>>  
>>>
>>> When I check, I see btrfs usage and it’s really full, but balance as
>>> no effect.
>>>
>>> After search I see quota is activate, and subvolumes /var/lib/etcd
>>> reserved 90% of space. But I don’t succeed to release this space.
>>>
>>>  
>>>
>>> Can you help me for release space disk ?
>>>
>>>  
>>>
>>> ·         On master :
>>>
>>>  
>>>
>>> *# btrfs filesystem usage /*
>>>
>>> Overall:
>>>
>>>     Device size:                  30.00GiB
>>>
>>>     Device allocated:             29.99GiB
>>>
>>>     Device unallocated:           17.00MiB
>>>
>>>     Device missing:                  0.00B
>>>
>>>     Used:                         27.56GiB
>>>
>>>     Free (estimated):            504.93MiB      (min: 496.43MiB)
>>>
>>>     Data ratio:                       1.00
>>>
>>>     Metadata ratio:                   2.00
>>>
>>>     Global reserve:               16.00MiB      (used: 0.00B)
>>>
>>>  
>>>
>>> Data,single: Size:27.97GiB, Used:27.49GiB
>>>
>>>    /dev/vda6      27.97GiB
>>>
>>>  
>>>
>>> Metadata,DUP: Size:1.00GiB, Used:32.64MiB
>>>
>>>    /dev/vda6       2.00GiB
>>>
>>>  
>>>
>>> System,DUP: Size:9.50MiB, Used:16.00KiB
>>>
>>>    /dev/vda6      19.00MiB
>>>
>>>  
>>>
>>> Unallocated:
>>>
>>>    /dev/vda6      17.00MiB
>>>
>>>  
>>>
>>> *# btrfs fi df /*
>>>
>>> Data, single: total=27.97GiB, used=27.50GiB
>>>
>>> System, DUP: total=9.50MiB, used=16.00KiB
>>>
>>> Metadata, DUP: total=1.00GiB, used=32.66MiB
>>>
>>> GlobalReserve, single: total=16.00MiB, used=0.00B
>>>
>>>  
>>>
>>> *# btrfs fi show /*
>>>
>>> Label: none  uuid: 1b0614eb-fc59-4841-bbc5-5318087f6432
>>>
>>>         Total devices 1 FS bytes used 27.53GiB
>>>
>>>         devid    1 size 30.00GiB used 29.99GiB path /dev/vda6
>>>
>>>  
>>>
>>> *# btrfs subvolume list /*
>>>
>>> ID 257 gen 40 top level 5 path @
>>>
>>> ID 258 gen 194820 top level 257 path @/.snapshots
>>>
>>> ID 259 gen 197128 top level 258 path @/.snapshots/1/snapshot
>>>
>>> ID 260 gen 194810 top level 257 path @/boot/grub2/i386-pc
>>>
>>> ID 261 gen 194810 top level 257 path @/boot/grub2/x86_64-efi
>>>
>>> ID 262 gen 194810 top level 257 path @/cloud-init-config
>>>
>>> ID 263 gen 194810 top level 257 path @/home
>>>
>>> ID 264 gen 197081 top level 257 path @/root
>>>
>>> ID 265 gen 197111 top level 257 path @/tmp
>>>
>>> ID 266 gen 194809 top level 257 path @/var/cache
>>>
>>> ID 267 gen 194809 top level 257 path @/var/crash
>>>
>>> ID 268 gen 195783 top level 257 path @/var/lib/ca-certificates
>>>
>>> ID 269 gen 195783 top level 257 path @/var/lib/cloud
>>>
>>> ID 270 gen 24 top level 257 path @/var/lib/docker
>>>
>>> ID 271 gen 194810 top level 257 path @/var/lib/dockershim
>>>
>>> ID 272 gen 195719 top level 257 path @/var/lib/etcd
>>>
>>> ID 273 gen 194810 top level 257 path @/var/lib/kubelet
>>>
>>> ID 274 gen 194810 top level 257 path @/var/lib/machines
>>>
>>> ID 275 gen 196430 top level 257 path @/var/lib/misc
>>>
>>> ID 276 gen 194810 top level 257 path @/var/lib/mysql
>>>
>>> ID 277 gen 194810 top level 257 path @/var/lib/nfs
>>>
>>> ID 278 gen 194810 top level 257 path @/var/lib/ntp
>>>
>>> ID 279 gen 196428 top level 257 path @/var/lib/overlay
>>>
>>> ID 280 gen 194810 top level 257 path @/var/lib/rollback
>>>
>>> ID 281 gen 196427 top level 257 path @/var/lib/systemd
>>>
>>> ID 282 gen 194810 top level 257 path @/var/lib/vmware
>>>
>>> ID 283 gen 194810 top level 257 path @/var/lib/wicked
>>>
>>> ID 284 gen 197128 top level 257 path @/var/log
>>>
>>> ID 285 gen 197111 top level 257 path @/var/spool
>>>
>>> ID 286 gen 196428 top level 257 path @/var/tmp
>>>
>>>  
>>>
>>> *# btrfs qgroup show -pcreFf /var/lib/etcd*
>>>
>>> qgroupid         rfer         excl     max_rfer     max_excl parent 
>>> child
>>>
>>> --------         ----         ----     --------     -------- ------ 
>>> -----
>>>
>>> 0/272        25.14GiB     25.14GiB         none         none ---     ---
>>>
>>>  
>>>
>>> *# du -sh /var/lib/etcd/*
>>>
>>> 417M    /var/lib/etcd/
>>>
>>>  
>>>
>>> ·         On one worker
>>>
>>>  
>>>
>>> *# btrfs fi usage /*
>>>
>>> Overall:
>>>
>>>     Device size:                  30.00GiB
>>>
>>>     Device allocated:             30.00GiB
>>>
>>>     Device unallocated:            1.00MiB
>>>
>>>     Device missing:                  0.00B
>>>
>>>     Used:                         27.94GiB
>>>
>>>     Free (estimated):            135.28MiB      (min: 135.28MiB)
>>>
>>>     Data ratio:                       1.00
>>>
>>>     Metadata ratio:                   2.00
>>>
>>>     Global reserve:               16.00MiB      (used: 0.00B)
>>>
>>>  
>>>
>>> Data,single: Size:27.99GiB, Used:27.86GiB
>>>
>>>    /dev/vda6      27.99GiB
>>>
>>>  
>>>
>>> Metadata,DUP: Size:1.00GiB, Used:43.44MiB
>>>
>>>    /dev/vda6       2.00GiB
>>>
>>>  
>>>
>>> System,DUP: Size:8.00MiB, Used:16.00KiB
>>>
>>>    /dev/vda6      16.00MiB
>>>
>>>  
>>>
>>> Unallocated:
>>>
>>>    /dev/vda6       1.00MiB
>>>
>>>  
>>>
>>> *# btrfs fi df /*
>>>
>>> Data, single: total=27.99GiB, used=27.86GiB
>>>
>>> System, DUP: total=8.00MiB, used=16.00KiB
>>>
>>> Metadata, DUP: total=1.00GiB, used=43.44MiB
>>>
>>> GlobalReserve, single: total=16.00MiB, used=0.00B
>>>
>>>  
>>>
>>> *# btrfs fi show /*
>>>
>>> Label: none  uuid: 1d7b76f8-f91c-47e0-8be2-a3f02f90ac96
>>>
>>>         Total devices 1 FS bytes used 27.90GiB
>>>
>>>         devid    1 size 30.00GiB used 30.00GiB path /dev/vda6
>>>
>>>  
>>>
>>> *# btrfs qgroup show -pcreFf /var/lib/etcd*
>>>
>>> qgroupid         rfer         excl     max_rfer     max_excl parent 
>>> child
>>>
>>> --------         ----         ----     --------     -------- ------ 
>>> -----
>>>
>>> 0/272        20.99GiB     20.99GiB         none         none ---     ---
>>>
>>>  
>>>
>>> *# du -sh /var/lib/etcd/*
>>>
>>> 452M    /var/lib/etcd/
>>>
>>>  
>>>
>>> Regards,
>>>
>>>  
>>>
>>> *Stéphane Le Bihan *
>>>
>>> SDE/DSI/IPR/SSD/UNX
>>>
>>> 90, Boulevard Pasteur - 75015 Paris
>>>
>>> *Web: http://www.amundi.com <http://www.amundi.com/>*
>>>
>>> Tél: +33 1 76 32 32 08
>>>
>>> Equipe Unix : +33 1 76 32 02 30
>>>
>>> @: stephane.lebihan at amundi.com <mailto:stephane.lebihan at amundi.com>
>>>
>>> @ : sits.unix at amundi.com <mailto:sits.unix at amundi.com>
>>>
>>>  
>>>
>>>  
>>>
>>> _______________________________________________
>>> caasp-beta mailing list
>>> caasp-beta at lists.suse.com <mailto:caasp-beta at lists.suse.com>
>>> http://lists.suse.com/mailman/listinfo/caasp-beta
>> _______________________________________________
>> caasp-beta mailing list
>> caasp-beta at lists.suse.com <mailto:caasp-beta at lists.suse.com>
>> http://lists.suse.com/mailman/listinfo/caasp-beta
> _______________________________________________
> caasp-beta mailing list
> caasp-beta at lists.suse.com <mailto:caasp-beta at lists.suse.com>
> http://lists.suse.com/mailman/listinfo/caasp-beta
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.suse.com/pipermail/caasp-beta/attachments/20171102/77831381/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 2430 bytes
Desc: not available
URL: <http://lists.suse.com/pipermail/caasp-beta/attachments/20171102/77831381/attachment.gif>


More information about the caasp-beta mailing list