<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@PMingLiU";
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \, serif";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"Préformaté HTML Car";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Texte de bulles Car";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
span.PrformatHTMLCar
{mso-style-name:"Préformaté HTML Car";
mso-style-priority:99;
mso-style-link:"Préformaté HTML";
font-family:Consolas;}
span.TextedebullesCar
{mso-style-name:"Texte de bulles Car";
mso-style-priority:99;
mso-style-link:"Texte de bulles";
font-family:"Tahoma","sans-serif";}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle26
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:347564974;
mso-list-type:hybrid;
mso-list-template-ids:225879624 -142866436 67895299 67895301 67895297 67895299 67895301 67895297 67895299 67895301;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:PMingLiU;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="FR" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hello Ludovic,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">I can provide us result now, but we success to restore free space yesterday. And I think we find cause.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">For restore free space we have stop etcd.service, remove all file in /var/lib/etcd, and restart etcd.service.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># systemctl stop etcd<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"># rm –rf /etc/sysconfig/etcd/member<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"># systemctl start etcd<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="color:#1F497D"># du -csh /*<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="color:#1F497D">4.6M /bin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">44M /boot<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">0 /cloud-init-config<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">8.0K /dev<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">12M /etc<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">0 /home<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">318M /lib<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">14M /lib64<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">0 /mnt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">0 /opt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">du: cannot access '/proc/24205/task/24205/fd/4': No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">du: cannot access '/proc/24205/task/24205/fdinfo/4': No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">du: cannot access '/proc/24205/fd/4': No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">du: cannot access '/proc/24205/fdinfo/4': No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">0 /proc<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">3.4M /root<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">218M /run<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">5.7M /sbin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">0 /selinux<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">0 /srv<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">0 /sys<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">48K /tmp<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">1.8G /usr<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">5.4G /var<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">7.8G total<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="color:#1F497D"># btrfs fi usage /<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Overall:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> Device size: 30.00GiB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> Device allocated: 5.02GiB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> Device unallocated: 24.99GiB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> Device missing: 0.00B<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> Used: 2.55GiB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> Free (estimated): 25.50GiB (min: 13.00GiB)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> Data ratio: 1.00<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> Metadata ratio: 2.00<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> Global reserve: 16.00MiB (used: 0.00B)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Data,single: Size:3.00GiB, Used:2.49GiB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> /dev/vda6 3.00GiB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Metadata,DUP: Size:1.00GiB, Used:32.59MiB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> /dev/vda6 2.00GiB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">System,DUP: Size:9.50MiB, Used:16.00KiB<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><span style="color:#1F497D">/dev/vda6 19.00MiB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Unallocated:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> /dev/vda6 24.99GiB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Etcd seems ok, but flannel is KO.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">After search I discover we can’t ping all other server (in or not in CAASP) from master and worker.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">I connect to admin node and it’s same.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">So I search in history, and I found my team patch OS of hypervisor on 22-October.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">My architecture is based on KVM, on one physical server SLES12 SP2, but I think after upgrade of hypervisor on SLES12 SP3, virtio card of KVM don’t work correctly…<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="color:#1F497D"># cat /etc/hosts<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">#<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># hosts This file describes a number of hostname-to-address<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># mappings for the TCP/IP subsystem. It is mostly<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># used at boot time, when no name servers are running.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># On small systems, this file can be used instead of a<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># "named" name server.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># Syntax:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">#<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># IP-Address Full-Qualified-Hostname Short-Hostname<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">#<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">127.0.0.1 localhost<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># special IPv6 addresses<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">::1 localhost ipv6-localhost ipv6-loopback<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">fe00::0 ipv6-localnet<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">ff00::0 ipv6-mcastprefix<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">ff02::1 ipv6-allnodes<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">ff02::2 ipv6-allrouters<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">ff02::3 ipv6-allhosts<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">#-- start Salt-CaaSP managed hosts - DO NOT MODIFY --<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">### service names ###<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">127.0.0.1 api api.infra.caasp.local dev-kubm01.unix.sits.credit-agricole.fr<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">### admin nodes ###<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">10.198.47.219 admin admin.infra.caasp.local<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">### kubernetes masters ###<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">10.198.47.220 f74967034d3743f1b843d227df61c7ad f74967034d3743f1b843d227df61c7ad.infra.caasp.local<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">### kubernetes workers ###<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">10.198.47.224 82c1065b62f84a508a9e1ffeb45a5cf2 82c1065b62f84a508a9e1ffeb45a5cf2.infra.caasp.local<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">10.198.47.223 afbe67218e5b4807a16e84997de79c6f afbe67218e5b4807a16e84997de79c6f.infra.caasp.local<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">10.198.47.221 12b79838fd734263830ffeb74dbb35bb 12b79838fd734263830ffeb74dbb35bb.infra.caasp.local<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">10.198.47.222 d246e0d7ff5b49c0996ea10c7bb8ca43 d246e0d7ff5b49c0996ea10c7bb8ca43.infra.caasp.local<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">#-- end Salt-CaaSP managed hosts --<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="color:#1F497D"># ip a<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> inet 127.0.0.1/8 scope host lo<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> valid_lft forever preferred_lft forever<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> inet6 ::1/128 scope host<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> valid_lft forever preferred_lft forever<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> link/ether 52:54:00:49:ee:13 brd ff:ff:ff:ff:ff:ff<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> inet 10.198.47.220/24 brd 10.198.47.255 scope global eth0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> valid_lft forever preferred_lft forever<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> inet6 fe80::5054:ff:fe49:ee13/64 scope link<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> valid_lft forever preferred_lft forever<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="color:#1F497D"># route -n<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Kernel IP routing table<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Destination Gateway Genmask Flags Metric Ref Use Iface<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">0.0.0.0 10.198.47.253 0.0.0.0 UG 0 0 0 eth0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">10.198.47.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="color:#1F497D"># ping 10.198.47.219<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">PING 10.198.47.219 (10.198.47.219) 56(84) bytes of data.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">^C<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">--- 10.198.47.219 ping statistics ---<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">5 packets transmitted, 0 received, 100% packet loss, time 3999ms<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="color:#1F497D"># ping 10.198.47.221<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">PING 10.198.47.221 (10.198.47.221) 56(84) bytes of data.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">^C<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">--- 10.198.47.221 ping statistics ---<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">3 packets transmitted, 0 received, 100% packet loss, time 2014ms<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="color:#1F497D"># ping 10.198.47.253<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">PING 10.198.47.253 (10.198.47.253) 56(84) bytes of data.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">^C<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">--- 10.198.47.253 ping statistics ---<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">5 packets transmitted, 0 received, 100% packet loss, time 4002ms<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#002060"><img width="132" height="50" id="_x0000_i1028" src="cid:image001.gif@01D3547B.A5D70880"></span><span style="font-size:12.0pt;color:#1F497D"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0A0D3D">Stéphane Le Bihan <o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0A0D3D">SDE/DSI/IPR/SSD/UNX<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">90, Boulevard Pasteur - 75015 Paris<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#1F497D">Web:
<a href="http://www.amundi.com/">http://www.amundi.com</a><o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">Tél: +33 1 76 32 32 08<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">Equipe Unix : +33 1 76 32 02 30<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">@:
<a href="mailto:stephane.lebihan@amundi.com">stephane.lebihan@amundi.com</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">@ :
<a href="mailto:sits.unix@amundi.com">sits.unix@amundi.com</a><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">De :</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Ludovic Cavajani [mailto:ludovic.cavajani@suse.com]
<br>
<b>Envoyé :</b> jeudi 2 novembre 2017 16:47<br>
<b>À :</b> Paul Gonin; caasp-beta@lists.suse.com; Le Bihan Stéphane (AMUNDI-ITS)<br>
<b>Objet :</b> Re: [caasp-beta] BTRFS space and quota]<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Hello Stéphane,<o:p></o:p></p>
<p>Can you provide us the output of :<o:p></o:p></p>
<p># du -csh /*<o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Regards,<o:p></o:p></p>
<div>
<p class="MsoNormal">On 11/02/2017 11:54 AM, Paul Gonin wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">-------- Message transféré --------<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><b>Date</b>: Thu, 2 Nov 2017 10:35:13 +0000<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><b>Objet</b>: Re: [caasp-beta] BTRFS space and quota<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><b>À</b>: Paul Gonin <<a href="mailto:Paul%20Gonin%20%3cpaul.gonin@suse.com%3e">paul.gonin@suse.com</a>>,
<a href="mailto:caasp-beta@lists.suse.com">caasp-beta@lists.suse.com</a> <<a href="mailto:%22caasp-beta@lists.suse.com%22%20%3ccaasp-beta@lists.suse.com%3e">caasp-beta@lists.suse.com</a>><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><b>De</b>: Le Bihan Stéphane (AMUNDI-ITS) <<a href="mailto:Le%20Bihan%20%3d%3fISO-8859-1%3fQ%3fSt%3dE9phane%3f%3d%20%22%28AMUNDI-ITS%29%22%20%3cstephane.lebihan@amundi.com%3e">stephane.lebihan@amundi.com</a>><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D">Hi Paul,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">The result of command snapper ls.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># snapper ls</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Type | # | Pre # | Date | User | Cleanup | Description | Userdata</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">-------+---+-------+---------------------------------+------+---------+-----------------------+---------</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">single | 0 | | | root | | current |</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">single | 1 | | Fri 06 Oct 2017 08:47:14 AM UTC | root | | first root filesystem |</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">I delete quota on /var/lb/etcd, and test balance but it’s not ok.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">I recreate quota and rescan and value is same before deletion.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">For information I launch du –sh on / and result is 7.8Go.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"># du -sh /</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">du: cannot access '/proc/7982/task/7982/fd/4': No such file or directory</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">du: cannot access '/proc/7982/task/7982/fdinfo/4': No such file or directory</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">du: cannot access '/proc/7982/fd/3': No such file or directory</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">du: cannot access '/proc/7982/fdinfo/3': No such file or directory</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">7.8G /</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#002060"><img border="0" width="132" height="50" id="_x0000_i1025" src="cid:image001.gif@01D3547B.A5D70880"></span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0A0D3D">Stéphane Le Bihan </span></b><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0A0D3D">SDE/DSI/IPR/SSD/UNX</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">90, Boulevard Pasteur - 75015 Paris</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#1F497D">Web:
<a href="http://www.amundi.com/">http://www.amundi.com</a></span></b><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">Tél: +33 1 76 32 32 08</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">Equipe Unix : +33 1 76 32 02 30</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">@:
<a href="mailto:stephane.lebihan@amundi.com">stephane.lebihan@amundi.com</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">@ :
<a href="mailto:sits.unix@amundi.com">sits.unix@amundi.com</a></span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De :</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Paul Gonin [<a href="mailto:paul.gonin@suse.com">mailto:paul.gonin@suse.com</a>]
<br>
<b>Envoyé :</b> jeudi 2 novembre 2017 10:55<br>
<b>À :</b> Le Bihan Stéphane (AMUNDI-ITS); <a href="mailto:caasp-beta@lists.suse.com">
caasp-beta@lists.suse.com</a><br>
<b>Objet :</b> Re: [caasp-beta] BTRFS space and quota</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">Hi Stephane,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">What is the output of
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""># snapper ls</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">I assume that since you there were no updates yet it should look like</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">Type | # | Pre # | Date | User | Cleanup | Description | Userdata </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">-------+---+-------+--------------------------+------+---------+-----------------------+--------------</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">single | 0 | | | root | | current | </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">single | 1 | | Tue Oct 31 09:07:13 2017 | root | | first root filesystem | </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">single | 2 | | Tue Oct 31 09:10:42 2017 | root | number | after installation | important=yes</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">rgds</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">Paul</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">Le mardi 31 octobre 2017 à 13:38 +0000, Le Bihan Stéphane (AMUNDI-ITS) a écrit :</span><o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #729FCF 1.5pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span style="color:#1F497D">Hi Paul,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">We work with CaaSP2.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#002060"><img border="0" width="132" height="50" id="_x0000_i1026" src="cid:image001.gif@01D3547B.A5D70880"></span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0A0D3D">Stéphane Le Bihan </span></b><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0A0D3D">SDE/DSI/IPR/SSD/UNX</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">90, Boulevard Pasteur - 75015 Paris</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#1F497D">Web:
<a href="http://www.amundi.com/">http://www.amundi.com</a></span></b><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">Tél: +33 1 76 32 32 08</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">Equipe Unix : +33 1 76 32 02 30</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">@:
<a href="mailto:stephane.lebihan@amundi.com">stephane.lebihan@amundi.com</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">@ :
<a href="mailto:sits.unix@amundi.com">sits.unix@amundi.com</a></span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De :</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Paul Gonin [<a href="mailto:paul.gonin@suse.com">mailto:paul.gonin@suse.com</a>]
<br>
<b>Envoyé :</b> mardi 31 octobre 2017 14:34<br>
<b>À :</b> Le Bihan Stéphane (AMUNDI-ITS); <a href="mailto:caasp-beta@lists.suse.com">
caasp-beta@lists.suse.com</a><br>
<b>Objet :</b> Re: [caasp-beta] BTRFS space and quota</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">Hi Stéphane,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">Not that it should make a difference for the issue described, what version of CaaSP the cluster is running ?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">Is it CaaSP2 ? RC1 ?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">thanks</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">Paul</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif""> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman , serif","serif"">Le mardi 31 octobre 2017 à 08:35 +0000, Le Bihan Stéphane (AMUNDI-ITS) a écrit :</span><o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #729FCF 1.5pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span lang="EN-US">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">We have a strange case on CAASP plateform with btrfs quota.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">For history, I was out of office since 3 weeks, but others colleague test kubernetes plateform.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">When I return, we ask me because FS is full on master and worker nodes.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">I don’t have cause, but I think with a bad config, subvolume /var/lib/etcd grown and after correction reduce, though quota reserved all space.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">When I check, I see btrfs usage and it’s really full, but balance as no effect.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">After search I see quota is activate, and subvolumes /var/lib/etcd reserved 90% of space. But I don’t succeed to release this space.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Can you help me for release space disk ?</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">On master :</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><b><span lang="EN-US"># btrfs filesystem usage /</span></b><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">Overall:</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Device size: 30.00GiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Device allocated: 29.99GiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Device unallocated: 17.00MiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Device missing: 0.00B</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Used: 27.56GiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Free (estimated): 504.93MiB (min: 496.43MiB)</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Data ratio: 1.00</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Metadata ratio: 2.00</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Global reserve: 16.00MiB (used: 0.00B)</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">Data,single: Size:27.97GiB, Used:27.49GiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> /dev/vda6 27.97GiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">Metadata,DUP: Size:1.00GiB, Used:32.64MiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> /dev/vda6 2.00GiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">System,DUP: Size:9.50MiB, Used:16.00KiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> /dev/vda6 19.00MiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">Unallocated:</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> /dev/vda6 17.00MiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><b><span lang="EN-US"># btrfs fi df /</span></b><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">Data, single: total=27.97GiB, used=27.50GiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">System, DUP: total=9.50MiB, used=16.00KiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">Metadata, DUP: total=1.00GiB, used=32.66MiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">GlobalReserve, single: total=16.00MiB, used=0.00B</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><b><span lang="EN-US"># btrfs fi show /</span></b><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">Label: none uuid: 1b0614eb-fc59-4841-bbc5-5318087f6432</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Total devices 1 FS bytes used 27.53GiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> devid 1 size 30.00GiB used 29.99GiB path /dev/vda6</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><b><span lang="EN-US"># btrfs subvolume list /</span></b><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 257 gen 40 top level 5 path @</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 258 gen 194820 top level 257 path @/.snapshots</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 259 gen 197128 top level 258 path @/.snapshots/1/snapshot</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 260 gen 194810 top level 257 path @/boot/grub2/i386-pc</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 261 gen 194810 top level 257 path @/boot/grub2/x86_64-efi</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 262 gen 194810 top level 257 path @/cloud-init-config</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 263 gen 194810 top level 257 path @/home</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 264 gen 197081 top level 257 path @/root</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 265 gen 197111 top level 257 path @/tmp</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 266 gen 194809 top level 257 path @/var/cache</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 267 gen 194809 top level 257 path @/var/crash</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 268 gen 195783 top level 257 path @/var/lib/ca-certificates</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 269 gen 195783 top level 257 path @/var/lib/cloud</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 270 gen 24 top level 257 path @/var/lib/docker</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 271 gen 194810 top level 257 path @/var/lib/dockershim</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 272 gen 195719 top level 257 path @/var/lib/etcd</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 273 gen 194810 top level 257 path @/var/lib/kubelet</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 274 gen 194810 top level 257 path @/var/lib/machines</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 275 gen 196430 top level 257 path @/var/lib/misc</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 276 gen 194810 top level 257 path @/var/lib/mysql</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 277 gen 194810 top level 257 path @/var/lib/nfs</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 278 gen 194810 top level 257 path @/var/lib/ntp</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 279 gen 196428 top level 257 path @/var/lib/overlay</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 280 gen 194810 top level 257 path @/var/lib/rollback</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 281 gen 196427 top level 257 path @/var/lib/systemd</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 282 gen 194810 top level 257 path @/var/lib/vmware</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 283 gen 194810 top level 257 path @/var/lib/wicked</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 284 gen 197128 top level 257 path @/var/log</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 285 gen 197111 top level 257 path @/var/spool</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">ID 286 gen 196428 top level 257 path @/var/tmp</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><b><span lang="EN-US"># btrfs qgroup show -pcreFf /var/lib/etcd</span></b><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">qgroupid rfer excl max_rfer max_excl parent child</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">-------- ---- ---- -------- -------- ------ -----</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">0/272 25.14GiB 25.14GiB none none --- ---</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><b># du -sh /var/lib/etcd/</b><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">417M /var/lib/etcd/</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">On one worker</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><b><span lang="EN-US"># btrfs fi usage /</span></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">Overall:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> Device size: 30.00GiB</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> Device allocated: 30.00GiB</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> Device unallocated: 1.00MiB</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> Device missing: 0.00B</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> Used: 27.94GiB</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> Free (estimated): 135.28MiB (min: 135.28MiB)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> Data ratio: 1.00</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> Metadata ratio: 2.00</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> Global reserve: 16.00MiB (used: 0.00B)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">Data,single: Size:27.99GiB, Used:27.86GiB</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> /dev/vda6 27.99GiB</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">Metadata,DUP: Size:1.00GiB, Used:43.44MiB</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> /dev/vda6 2.00GiB</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">System,DUP: Size:8.00MiB, Used:16.00KiB</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> /dev/vda6 16.00MiB</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">Unallocated:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"> /dev/vda6 1.00MiB</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><b><span lang="EN-US"># btrfs fi df /</span></b><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">Data, single: total=27.99GiB, used=27.86GiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">System, DUP: total=8.00MiB, used=16.00KiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">Metadata, DUP: total=1.00GiB, used=43.44MiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">GlobalReserve, single: total=16.00MiB, used=0.00B</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><b><span lang="EN-US"># btrfs fi show /</span></b><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">Label: none uuid: 1d7b76f8-f91c-47e0-8be2-a3f02f90ac96</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> Total devices 1 FS bytes used 27.90GiB</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> devid 1 size 30.00GiB used 30.00GiB path /dev/vda6</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><b><span lang="EN-US"># btrfs qgroup show -pcreFf /var/lib/etcd</span></b><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">qgroupid rfer excl max_rfer max_excl parent child</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">-------- ---- ---- -------- -------- ------ -----</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">0/272 20.99GiB 20.99GiB none none --- ---</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoListParagraph"><b># du -sh /var/lib/etcd/</b><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US">452M /var/lib/etcd/</span><o:p></o:p></p>
<p class="MsoListParagraph"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#002060"><img border="0" width="132" height="50" id="_x0000_i1027" src="cid:image001.gif@01D3547B.A5D70880"></span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0A0D3D">Stéphane Le Bihan </span></b><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0A0D3D">SDE/DSI/IPR/SSD/UNX</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">90, Boulevard Pasteur - 75015 Paris</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Arial","sans-serif"">Web:
<a href="http://www.amundi.com/">http://www.amundi.com</a></span></b><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">Tél: +33 1 76 32 32 08</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">Equipe Unix : +33 1 76 32 02 30</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">@:
<a href="mailto:stephane.lebihan@amundi.com">stephane.lebihan@amundi.com</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#0A0D3D">@ :
<a href="mailto:sits.unix@amundi.com">sits.unix@amundi.com</a></span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>caasp-beta mailing list<o:p></o:p></pre>
<pre><a href="mailto:caasp-beta@lists.suse.com">caasp-beta@lists.suse.com</a><o:p></o:p></pre>
<pre><a href="http://lists.suse.com/mailman/listinfo/caasp-beta">http://lists.suse.com/mailman/listinfo/caasp-beta</a><o:p></o:p></pre>
</blockquote>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>caasp-beta mailing list<o:p></o:p></pre>
<pre><a href="mailto:caasp-beta@lists.suse.com">caasp-beta@lists.suse.com</a><o:p></o:p></pre>
<pre><a href="http://lists.suse.com/mailman/listinfo/caasp-beta">http://lists.suse.com/mailman/listinfo/caasp-beta</a><o:p></o:p></pre>
</blockquote>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>caasp-beta mailing list<o:p></o:p></pre>
<pre><a href="mailto:caasp-beta@lists.suse.com">caasp-beta@lists.suse.com</a><o:p></o:p></pre>
<pre><a href="http://lists.suse.com/mailman/listinfo/caasp-beta">http://lists.suse.com/mailman/listinfo/caasp-beta</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></p>
</div>
</body>
</html>