|
Home > Archive > Red Hat Topics > January 2006 > Linux file copy slowness ??
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Linux file copy slowness ??
|
|
| parvinderb@gmail.com 2006-01-06, 5:49 pm |
| Hi,
We are trying to copy a 15Gig file from one directory to another
(inside same partition) for example:
cp /data/BigFile /data/data/ (Bigfile - being file that is greater
than 15 gig).
It takes about 32 minutes to finish copying. When we do the same
operation on a windows 2003 server it takes almost HALF the time.
I am wondering if this behaviour is normal or something wrong with
system?kernel? etc.
The system is a hefty: Dell 1850 server with Dual Xeons 3.2 ghz (hyper
threaded) with 2 Gig ram, SCSI hard drives 10k rpm.
Also, while the system is copying the file, I don't see any kernel
errors or IO errros and all the time while the sytem is copying this
drive the cpu usage is at 50 Any help is highly appreciated.
Forgot to mention on another post: The file system is EXT3 and OS
-Redhat Enterprise 3
-Parvinder Bhasin
| |
| Scott Lurndal 2006-01-06, 8:46 pm |
| parvinderb@gmail.com writes:
>Hi,
>
>We are trying to copy a 15Gig file from one directory to another
>(inside same partition) for example:
>
>cp /data/BigFile /data/data/ (Bigfile - being file that is greater
>than 15 gig).
So, both files are on the same disk, and you're getting 7.8Mb/second.
For reference, this is the output from a 1gb random transfer on
a 36MB 10krpm seagate cheetah:
$ t2 -r -f /dev/sday -b 131072 -i 1000
Device /dev/sday has 70022843 512-byte blocks
131072000 bytes (1000 131072-byte blocks to 1 devs) in 14.50 seconds (9039.45 Kbytes/sec)
(-r accesses the device randomly).
Drop the block size, and:
$ t2 -r -f /dev/sday -b 4096 -i 1000
Device /dev/sday has 70022843 512-byte blocks
4096000 bytes (1000 4096-byte blocks to 1 devs) in 9.00 seconds (455.11 Kbytes/sec)
So, your 7.8MB/second seems pretty good. You may get better
performance if you mk2efs your filesystem with -b 8196 instead
of 4096.
$ stat -f /
File: "/"
ID: 0 Namelen: 255 Type: ext2/ext3
Blocks: Total: 10078499 Free: 9590416 Available: 9078445 Size: 4096
Inodes: Total: 5128192 Free: 5065884
(the block size is the "Size:" field).
-----------Just for reference:
Pure sequential access on a Cheetah:
$ t2 -f /dev/sday -b 131072 -i 1000
Device /dev/sday has 70022843 512-byte blocks
131072000 bytes (1000 131072-byte blocks to 1 devs) in 3.50 seconds (37449.14 Kbytes/sec)
And, the advantages of fibrechannel and lots of JBOD (and a 2Gbit switch):
$ t2 -f /dev/sdag -f /dev/sdah -f /dev/sdam -f /dev/sdan -f /dev/sdaq -f /dev/sdap -f /dev/sdaa -f /dev/sdab -b 131072 -i 10000
Device /dev/sdag has 70022843 512-byte blocks
Device /dev/sdah has 70022843 512-byte blocks
Device /dev/sdam has 70022843 512-byte blocks
Device /dev/sdan has 70022843 512-byte blocks
Device /dev/sdaq has 163397632 512-byte blocks
Device /dev/sdap has 163397632 512-byte blocks
Device /dev/sdaa has 163397632 512-byte blocks
Device /dev/sdab has 163397632 512-byte blocks
10485760000 bytes (10000 131072-byte blocks to 8 devs) in 52.00 seconds (201649.23 Kbytes/sec)
Note that the test program, t2, uses O_DIRECT to bypass the operating system
file cache, and uses io_submit(2) fpr asynchronous I/O.
scott
| |
| Jean-David Beyer 2006-01-06, 8:46 pm |
| parvinderb@gmail.com wrote:
> Hi,
>
> We are trying to copy a 15Gig file from one directory to another (inside
> same partition) for example:
>
> cp /data/BigFile /data/data/ (Bigfile - being file that is greater than
> 15 gig).
>
> It takes about 32 minutes to finish copying. When we do the same
> operation on a windows 2003 server it takes almost HALF the time.
>
> I am wondering if this behaviour is normal or something wrong with
> system?kernel? etc.
>
> The system is a hefty: Dell 1850 server with Dual Xeons 3.2 ghz (hyper
> threaded) with 2 Gig ram, SCSI hard drives 10k rpm.
>
> Also, while the system is copying the file, I don't see any kernel errors
> or IO errros and all the time while the sytem is copying this drive the
> cpu usage is at 50 Any help is highly appreciated.
>
> Forgot to mention on another post: The file system is EXT3 and OS
> -Redhat Enterprise 3
>
Do you have another scsi drive (preferably, but not necessarily on a
different SCSI controller) on the system? If so, it might be faster to copy
to a spare spot on a different drive and then copy it back. This would cut
the seek time dramatically if that is the bottleneck, as it very well might be.
But I have 10,000 rpm hard drives on an Ultra/320 SCSI controller and I have
seen 50MBytes/second transfer rate on working system (also RHEL3) for a
sustained interval (many seconds), but with a full load of other processes
running, messing up the L3 cache on the dual 3.06GHz hyperthreades Xeons,
the disk IO rate can drop to under 15 MBytes/second.
One trick is to ask how you do the copying. If you use _cpio --pass-through_
it might run faster than with the _cp_ command.
--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key: 9A2FC99A Registered Machine 241939.
/( )\ Shrewsbury, New Jersey http://counter.li.org
^^-^^ 20:50:00 up 8 days, 11:36, 4 users, load average: 4.38, 4.27, 4.14
| |
| decrepit 2006-01-08, 7:46 am |
| parvinderb@gmail.com wrote:
>
>
> It takes about 32 minutes to finish copying. When we do the same
> operation on a windows 2003 server it takes almost HALF the time.
>
> I am wondering if this behaviour is normal or something wrong with
> system?kernel? etc.
>
>
I'm not an expert so don't know if this is relevant, but I'll throw it
in just in case.
A while ago I was playing with multibooting XP and core 2 each on their
own hardrive, I was selecting OS with bios, somehow I managed to turn
the CPU cache off. This crippled XP slowed it down by a factor of about
1,000, took 30minutes just to boot!!, but core 2 seemed to be unafected.
Obviously XP makes a great use of cpu cache and fedora doesn't, would
this affect file juggling speeds???
| |
| Scott Lurndal 2006-01-09, 10:24 pm |
| decrepit <""mikelayzell\"@netscape .net"> writes:
>parvinderb@gmail.com wrote:
>I'm not an expert so don't know if this is relevant, but I'll throw it
>in just in case.
>
>A while ago I was playing with multibooting XP and core 2 each on their
>own hardrive, I was selecting OS with bios, somehow I managed to turn
>the CPU cache off. This crippled XP slowed it down by a factor of about
>1,000, took 30minutes just to boot!!, but core 2 seemed to be unafected.
>Obviously XP makes a great use of cpu cache and fedora doesn't, would
>this affect file juggling speeds???
"Obviously?" Hardly. Fedora core 2 (or any linux for that matter)
will use the processor cache just as much as Windows. The lack of
the cache will slow both down quite significantly. If your installation
of FC2 ran fast, then the cache was enabled.
It's possible that Windows obeyed the BIOS setting, and linux didn't, but
that's a different issue.
scott
| |
| decrepit 2006-01-13, 9:22 pm |
| Scott Lurndal wrote:
> "Obviously?" Hardly. Fedora core 2 (or any linux for that matter)
> will use the processor cache just as much as Windows. The lack of
> the cache will slow both down quite significantly. If your installation
> of FC2 ran fast, then the cache was enabled.
>
> It's possible that Windows obeyed the BIOS setting, and linux didn't, but
> that's a different issue.
>
> scott
Thanks Scott for putting me right, I guess I didn't consider that linux
wouldn't take any notice of bios.
So my point doesn't help this thread, but I've learnt something!
Mike
| |
| Jean-David Beyer 2006-01-13, 9:22 pm |
| Scott Lurndal wrote:
> "Obviously?" Hardly. Fedora core 2 (or any linux for that matter) will
> use the processor cache just as much as Windows. The lack of the cache
> will slow both down quite significantly. If your installation of FC2
> ran fast, then the cache was enabled.
>
> It's possible that Windows obeyed the BIOS setting, and linux didn't, but
> that's a different issue.
>
How do I tell if Linux obeys my BIOS setting? In the BIOS, I get to decide
if it is to use the L3 cache or not. I got the idea that the kernel got no
choice in the matter. On my machine, I have two hyperthreaded 3.06 GHz Xeon
processors. These have
/proc/cpuinfo says, in part (there are four of these, but you get the idea).
$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 3.06GHz
stepping : 5
cpu MHz : 3056.546
cache size : 512 KB
physical id : 0
siblings : 2
runqueue : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 6094.84
Now in these processors, there are "three" caches, as shown by /var/log/dmesg:
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: L3 cache: 1024K
It seems to me that the kernel is aware of the L3 cache, but it does not
appear in /proc/cpuinfo. So does the kernel use it or not? I always assumed
it did.
--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key: 9A2FC99A Registered Machine 241939.
/( )\ Shrewsbury, New Jersey http://counter.li.org
^^-^^ 07:40:00 up 11 days, 22:26, 3 users, load average: 4.38, 4.26, 4.20
| |
| Scott Lurndal 2006-01-13, 9:22 pm |
| Jean-David Beyer <jdbeyer@exit109.com> writes:
>Scott Lurndal wrote:
>
>How do I tell if Linux obeys my BIOS setting? In the BIOS, I get to decide
>if it is to use the L3 cache or not. I got the idea that the kernel got no
>choice in the matter. On my machine, I have two hyperthreaded 3.06 GHz Xeon
>processors. These have
>
>/proc/cpuinfo says, in part (there are four of these, but you get the idea).
>
>$ cat /proc/cpuinfo
>processor : 0
>vendor_id : GenuineIntel
>cpu family : 15
>model : 2
>model name : Intel(R) Xeon(TM) CPU 3.06GHz
>stepping : 5
>cpu MHz : 3056.546
>cache size : 512 KB
>physical id : 0
>siblings : 2
>runqueue : 0
>fdiv_bug : no
>hlt_bug : no
>f00f_bug : no
>coma_bug : no
>fpu : yes
>fpu_exception : yes
>cpuid level : 2
>wp : yes
>flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
>cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
>bogomips : 6094.84
>
>Now in these processors, there are "three" caches, as shown by /var/log/dmesg:
>
>CPU: Trace cache: 12K uops, L1 D cache: 8K
>CPU: L2 cache: 512K
>CPU: L3 cache: 1024K
>
>It seems to me that the kernel is aware of the L3 cache, but it does not
>appear in /proc/cpuinfo. So does the kernel use it or not? I always assumed
>it did.
it really has no choice. Weather or not the L3 cache
is used is governed by a set of bits in a configuration register associated
with the memory controller/northbridge chipset.
The BIOS is generally the only code which manipulates these configuration
registers, as they are highly chipset dependent.
L2 cache settings are somewhat under the control of the operating system
via the architecturally defined MTRR registers and attribute bits in the page tables.
scott
>
>--
> .~. Jean-David Beyer Registered Linux User 85642.
> /V\ PGP-Key: 9A2FC99A Registered Machine 241939.
> /( )\ Shrewsbury, New Jersey http://counter.li.org
> ^^-^^ 07:40:00 up 11 days, 22:26, 3 users, load average: 4.38, 4.26, 4.20
|
|
|
|
|