|
Home > Archive > Data Storage > June 2007 > time to read meta data on 300TB
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
time to read meta data on 300TB
|
|
| byaarov@yahoo.com 2007-06-06, 1:14 am |
| Hi,
How long would it take (on a decent NAS filer) to read the meta data
for 300TB of data via NFS?
I know the answer varys for different vendors, but im just looking for
a ballpark... Assuming average file size of 1MB, which is the case in
my environment, I estimate I have about 300 million files.
Assuming average meta data packet sizes are about 100bytes, thats
about 30GB of data. Network activity alone would take about 5
minutes, but I dont know what impact the filers ability will have on
this... so any ball park estimates are appreciated.
B
| |
| Faeandar 2007-06-08, 1:14 am |
| On Tue, 05 Jun 2007 19:23:46 -0700, byaarov@yahoo.com wrote:
>Hi,
>How long would it take (on a decent NAS filer) to read the meta data
>for 300TB of data via NFS?
>
>I know the answer varys for different vendors, but im just looking for
>a ballpark... Assuming average file size of 1MB, which is the case in
>my environment, I estimate I have about 300 million files.
>
>Assuming average meta data packet sizes are about 100bytes, thats
>about 30GB of data. Network activity alone would take about 5
>minutes, but I dont know what impact the filers ability will have on
>this... so any ball park estimates are appreciated.
>
>B
*IF* the metadata were readily available in memory or some similar
blazing fast medium on the NAS server it could take 30 minutes or
more. The 5 minute time you list would only be if that blazing fast
medium with the metadata in it were direct attaced to the client or
connected via FC. Essentially a local file system.
Realistically, 300TB of data on a NAS filer with it's metadata being
read by an NFS client will take days, not hours.
An NFS client cannot just access the metadata directly on traditional
NAS systems, it has to traverse the file system and access each
file/dir's inode.
That means requesting access to a file, having access granted,
requesting a read of the file (or lookup), having the NAS server seek
to the inode, read it from disk, hand it back to the client.
That's alot of work and not fast in the doing. And that's for every
single file in that 300TB.
You don't mention what application is doing the metadata reads, that
can also have bearing.
In my experience with SRM tools they generally scan at no more than 30
files/sec. Which is roughly equivalent to 2.5M files a day.
~F
|
|
|
|
|