|
Home > Archive > Data Storage > October 2007 > Metadata Server ?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
|
|
| Sebastien Campion 2007-10-19, 1:16 pm |
| Hi,
does anybody know if it's possible to have a kind of metadata server ?
I explain, with 25Tb of data and many directories of 100 000 files,
when i try "ls -all"
it's too long .....
I'd enough time to take a coffee 
So i would like to find a kind of metadata server synchronize with the
nas file system.
I think use a virtual distributed file system like Lustre or Gfarm.
thanks in advance,
Séb.
| |
| Faeandar 2007-10-20, 1:15 pm |
| On Fri, 19 Oct 2007 16:19:20 +0200, Sebastien Campion
<sebastien.campion@irisa.fr> wrote:
>Hi,
>does anybody know if it's possible to have a kind of metadata server ?
>
>I explain, with 25Tb of data and many directories of 100 000 files,
>when i try "ls -all"
>it's too long .....
>I'd enough time to take a coffee
>
>So i would like to find a kind of metadata server synchronize with the
>nas file system.
>
>I think use a virtual distributed file system like Lustre or Gfarm.
>
>
>thanks in advance,
>
>Séb.
If the file system supports a seperation of data and metadata, like
SAMFs or Lustre, then you actually *have* to have a metadata server.
If the fs does not support it then you can't have one, at least not
without doing some very unsupported things.
~F
| |
| Sebastien Campion 2007-10-22, 7:14 am |
| Thanks, i learned more about the filesystem, it's WAFL by NetApp.
In documentation, i see "The WAFL file system store metadata in files,
three files :
- inode file
- block-map file
- inode map"
But i don't know where i can find it.
I'd only NFS (or CIFS) protocol enable (it's a NAS NetApp) so i don't
know if it's feasible to connect Lustre on NFS.
If it's ok, i will install Lustre.
Séb.
But i don't know where
Faeandar wrote:
> On Fri, 19 Oct 2007 16:19:20 +0200, Sebastien Campion
> <sebastien.campion@irisa.fr> wrote:
>
>
>
> If the file system supports a seperation of data and metadata, like
> SAMFs or Lustre, then you actually *have* to have a metadata server.
> If the fs does not support it then you can't have one, at least not
> without doing some very unsupported things.
>
> ~F
>
| |
| Jan-Frode Myklebust 2007-10-22, 1:14 pm |
| On 2007-10-22, Sebastien Campion <sebastien.campion@irisa.fr> wrote:
>
> I'd only NFS (or CIFS) protocol enable (it's a NAS NetApp) so i don't
> know if it's feasible to connect Lustre on NFS.
> If it's ok, i will install Lustre.
No, lustre is a completely different file system. It can't help your NetApp.
100.000 files in each directory is going to be painfull to list,
no matter what file system you're using. Things that might help:
- run a name service cache daemon (nscd) to cache uid/gid lookups
- don't resolve uid/gid by using "ls -ln"
- use "locate" as your "metadata server", if it contains enough
metadata for your needs.
- see if an fscache/cachefs can help
http://people.redhat.com/~dhowells/fscache/FS-Cache.pdf
- tune your NFS client http://www.netapp.com/library/tr/3183.pdf
-jf
|
|
|
|
|