[suse-sles-e] Very high MD latencies with SLES10 SP1.

From: Norbert Paschedag (noe_at_physik.unizh.ch)
Date: Wed Dec 05 2007 - 16:13:18 CET

• Next message: Kim Johansen: "[suse-sles-e] Updates to sles10"
• Previous message: carlopmart: "Re: [suse-sles-e] (off. Should we move it out of the list?) Re: [suse-sles-e] Some questions about recntly released SLES Real time"
• Next in thread: Norbert Paschedag: "Re: [suse-sles-e] Very high MD latencies with SLES10 SP1."
• Maybe reply: Norbert Paschedag: "Re: [suse-sles-e] Very high MD latencies with SLES10 SP1."
• Maybe reply: heupink, mourik jan: "RE: [suse-sles-e] Very high MD latencies with SLES10 SP1."
• Reply: Manfred Hollstein: "Re: [suse-sles-e] Very high MD latencies with SLES10 SP1."
• Maybe reply: Norbert Paschedag: "Re: [suse-sles-e] Very high MD latencies with SLES10 SP1."
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Date: Wed, 5 Dec 2007 16:13:18 +0100 (CET)
From: Norbert Paschedag <noe@physik.unizh.ch>
Message-ID: <Pine.LNX.4.64.0712051518290.6336@hesse.math.tu-berlin.de>
Subject: [suse-sles-e] Very high MD latencies with SLES10 SP1.

Hi,

we're having a bit of a performance problem with our MD RAID1 setup after
having switched to SLES10 SP1 (from SLES9 SP3). For random-I/O in particular,
we're seeing maximum read and write latencies up to _several seconds_ (with averages
around 30ms) where SLES9 showed maxima around 500ms with avg around 20ms.

Typical outputs (randomio-1.3, using O_DIRECT):

sles9sp3:
  total | read: latency (ms) | write: latency (ms)
   iops | iops min avg max sdev | iops min avg max sdev
--------+-----------------------------------+----------------------------------
  636.8 | 476.7 0.2 20.6 501.6 39.1 | 160.1 0.4 1.0 25.0 1.4

sles10sp1:
  total | read: latency (ms) | write: latency (ms)
   iops | iops min avg max sdev | iops min avg max sdev
--------+-----------------------------------+----------------------------------
  350.1 | 262.5 0.2 29.9 7114.5 284.6 | 87.6 0.5 22.0 6511.6 253.6

The setup:
 - Two 500GB RAID5 slices, each from an Infortrend SATA-FC-Raid, connected
   to the fabrics with two FC lines each. Failover is
   enabled in the QLA2432 (ql2xfailover=1 ConfigRequired=0
   ql2xfdmienable=1 ql2xmaxqdepth=64), driver version v8.01.07.15-2.
   OS: SLES10sp1, kernel 2.6.16.54-0.2.3-smp.
   HW: dual opteron 2216 with 8G ram.
 - The two sd devices are then mirrored :
   mdadm -C /dev/md3 -l 1 -N test -n 2 /dev/sdc /dev/sdd
   mkfs.xfs -f -d agcount=32 -l version=2 /dev/md3
   mount -o nobarrier,logbufs=8,logbsize=64k /dev/md3 /mnt

The sd devices alone show normal timings, it seems the MD device that's
causing these latencies.
Any idea which parameter to change here ? Any caveats regarding MD RAID1
that I missed ?

We would really like to avoid going back to sles9.

Thanks for any help!

  N. Paschedag

---------------------------------------------------------------------
To unsubscribe, e-mail: suse-sles-e-unsubscribe@suse.com
For additional commands, e-mail: suse-sles-e-help@suse.com

• Next message: Kim Johansen: "[suse-sles-e] Updates to sles10"
• Previous message: carlopmart: "Re: [suse-sles-e] (off. Should we move it out of the list?) Re: [suse-sles-e] Some questions about recntly released SLES Real time"
• Next in thread: Norbert Paschedag: "Re: [suse-sles-e] Very high MD latencies with SLES10 SP1."
• Maybe reply: Norbert Paschedag: "Re: [suse-sles-e] Very high MD latencies with SLES10 SP1."
• Maybe reply: heupink, mourik jan: "RE: [suse-sles-e] Very high MD latencies with SLES10 SP1."
• Reply: Manfred Hollstein: "Re: [suse-sles-e] Very high MD latencies with SLES10 SP1."
• Maybe reply: Norbert Paschedag: "Re: [suse-sles-e] Very high MD latencies with SLES10 SP1."
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]

This archive was generated by hypermail 2.1.7 : Wed Dec 05 2007 - 05:13:51 CET