|
Home > Archive > Data Storage > August 2006 > At what point is ILM needed?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
At what point is ILM needed?
|
|
| jondelac@gmail.com 2006-08-07, 7:13 pm |
| Hi,
I am trying to figure out at what point I should be looking into some
sort of ILM technology. In reading vendor case studies such as
(http://www.netapp.com/library/cs/mustang.pdf), I dont see why I cant
just keep adding more disks to my primary storage right now as demand
increases... when should I be looking into ILM (at what storage
capacity)? How many GigaBytes or TeraBytes before I should consider an
ILM strategy?
Jon
| |
| Faeandar 2006-08-07, 7:13 pm |
| On 7 Aug 2006 13:38:41 -0700, jondelac@gmail.com wrote:
>Hi,
>I am trying to figure out at what point I should be looking into some
>sort of ILM technology. In reading vendor case studies such as
>(http://www.netapp.com/library/cs/mustang.pdf), I dont see why I cant
>just keep adding more disks to my primary storage right now as demand
>increases... when should I be looking into ILM (at what storage
>capacity)? How many GigaBytes or TeraBytes before I should consider an
>ILM strategy?
>
>Jon
Generally speaking ILM is used for legal purposes. Retention for
things like Intellectual Property and SOX compliance are usually the
driving factor. In some cases pure volume is the driver but those are
less frequent these days for just the reason you stated.
If I can get 500GB SATA drives, move data to those and stop backups,
why invest in ILM insfrastructure? You wouldn't.
But for pure volume I don't think you can beat tape as the backend
medium for cost. Footprint, capacity, heat, power, etc., all are won
by tape. Longevity and ease of migration are won by disk.
There is no magic dial that says "invest in ILM here" so you've got to
make judegement calls.
If you want to provide more information there may be people here who
can help.
~F
| |
| jondelac@gmail.com 2006-08-07, 7:13 pm |
|
Faeandar wrote:
> On 7 Aug 2006 13:38:41 -0700, jondelac@gmail.com wrote:
>
>
> Generally speaking ILM is used for legal purposes. Retention for
> things like Intellectual Property and SOX compliance are usually the
> driving factor. In some cases pure volume is the driver but those are
> less frequent these days for just the reason you stated.
>
> If I can get 500GB SATA drives, move data to those and stop backups,
> why invest in ILM insfrastructure? You wouldn't.
>
> But for pure volume I don't think you can beat tape as the backend
> medium for cost. Footprint, capacity, heat, power, etc., all are won
> by tape. Longevity and ease of migration are won by disk.
>
> There is no magic dial that says "invest in ILM here" so you've got to
> make judegement calls.
>
> If you want to provide more information there may be people here who
> can help.
>
> ~F
That helps somewhat... is there some specific growth rate at which ILM
for pure volume makes sense? Should I be worried about 4 TB of data in
my primary storage? I keep hearing about so many case studies etc
where poeple have applied ILM to help primary storage volume and my own
management says I should be doing the same... So I cant understand the
disconnect...
Jon.
| |
| Faeandar 2006-08-07, 7:13 pm |
| On 7 Aug 2006 15:13:18 -0700, jondelac@gmail.com wrote:
>
>Faeandar wrote:
>
>That helps somewhat... is there some specific growth rate at which ILM
>for pure volume makes sense? Should I be worried about 4 TB of data in
>my primary storage? I keep hearing about so many case studies etc
>where poeple have applied ILM to help primary storage volume and my own
>management says I should be doing the same... So I cant understand the
>disconnect...
>
>Jon.
Well, the disconnect is usually management that has heard about this
new thing that will "save them a ton of money". 4TB of data is huge
to some, and a pitance to others. The pure volume is hard to quantify
for ILM.
What storage type is your 4TB on? How long did it take you to get
there? What is your anticipated growth over the next 6-12 months?
What does your backup infrastructure consist of (drive classes, number
of tapes, backup servers, etc)? What type of data is it (structured,
unstructured, IP, e-mail, home directories)?
All these things will be used in a formula that will only work for
your environment.
~F
| |
| jondelac@gmail.com 2006-08-07, 7:13 pm |
|
Faeandar wrote:
> On 7 Aug 2006 15:13:18 -0700, jondelac@gmail.com wrote:
>
>
> Well, the disconnect is usually management that has heard about this
> new thing that will "save them a ton of money". 4TB of data is huge
> to some, and a pitance to others. The pure volume is hard to quantify
> for ILM.
>
> What storage type is your 4TB on? How long did it take you to get
> there? What is your anticipated growth over the next 6-12 months?
> What does your backup infrastructure consist of (drive classes, number
> of tapes, backup servers, etc)? What type of data is it (structured,
> unstructured, IP, e-mail, home directories)?
>
> All these things will be used in a formula that will only work for
> your environment.
>
> ~F
Our 4TB of data are on two FAS960Cs... we are expected to grow 50% this
year (year end we will have 6TB) and next year we expect to add another
4TB. We just started growing rapidly due to the nature of construction
projects being taken on, and they mostly consist of engineering and
design documents... all unstructured... actually, depending on how you
look at it, 70% unstructured and 30% email.
We have backup (tape) hanging off of the netapp directly and use
veritas to move stuff into backup (bi weekly I beleive, but I dont
manage that).
I spend about 15K per TB of data and our current solution cost about
$450K.
PS: One area where I get dinged all the time is the backup restore
times... management says I should do it faster and tell me I should be
looking into ILM for that. Also, they cringe whenever I go and put a
PO in for another TB of data.
| |
| Faeandar 2006-08-07, 7:13 pm |
| On 7 Aug 2006 15:51:40 -0700, jondelac@gmail.com wrote:
>
>Our 4TB of data are on two FAS960Cs... we are expected to grow 50% this
>year (year end we will have 6TB) and next year we expect to add another
>4TB. We just started growing rapidly due to the nature of construction
>projects being taken on, and they mostly consist of engineering and
>design documents... all unstructured... actually, depending on how you
>look at it, 70% unstructured and 30% email.
>
>We have backup (tape) hanging off of the netapp directly and use
>veritas to move stuff into backup (bi weekly I beleive, but I dont
>manage that).
>
>I spend about 15K per TB of data and our current solution cost about
>$450K.
>
>PS: One area where I get dinged all the time is the backup restore
>times... management says I should do it faster and tell me I should be
>looking into ILM for that. Also, they cringe whenever I go and put a
>PO in for another TB of data.
Well, since the 960's will expand to 64TB you've got plenty of grow
room.
For backup performance, if you don;t manage it why do you get dinged?
And how is it being done today? Volume level backups? Qtree level?
Any replication involved (snapmirror or snapvault)?
ILM is not a fix for any problem, period. It is a conglomerate of
products/programs/scripts that automate a business process. The
business process is the fix. Never lose sight of that and your battle
is assured.
Does your business have a process for long term retention? Data
destruction? E-mail compliance? Whatever it happens to be, ILM can
wrap some automation around it but the process and requirements have
to exist first. Your management needs an education and you are going
to have to do it. But first you need to educate yourself, and this
was a good start.
Take a look at some of the ILM procucts out there. See what they do.
You'll find ILM is just HSM rebranded with a few more bells and
whistles attached. Information Lifecycle Management, the name alone
should fire off some bells and sirens for you to deal with your
management.
The flip side is ILM is a great resume builder. However, I'd bet that
if you did due diligence, set your requirements, selected a vendor,
then got a quote, management would be happy to sign PO's for TB's of
storage for the next 2-3 years.
Don;t forget all the hidden costs of ILM; it's not just the software.
Tape, disk, switch ports, FC ports, whatever. Make sure you add it
all in.
~F
| |
| jondelac@gmail.com 2006-08-07, 7:13 pm |
|
Faeandar wrote:
> On 7 Aug 2006 15:51:40 -0700, jondelac@gmail.com wrote:
>
>
>
> Well, since the 960's will expand to 64TB you've got plenty of grow
> room.
>
> For backup performance, if you don;t manage it why do you get dinged?
> And how is it being done today? Volume level backups? Qtree level?
> Any replication involved (snapmirror or snapvault)?
>
> ILM is not a fix for any problem, period. It is a conglomerate of
> products/programs/scripts that automate a business process. The
> business process is the fix. Never lose sight of that and your battle
> is assured.
>
> Does your business have a process for long term retention? Data
> destruction? E-mail compliance? Whatever it happens to be, ILM can
> wrap some automation around it but the process and requirements have
> to exist first. Your management needs an education and you are going
> to have to do it. But first you need to educate yourself, and this
> was a good start.
>
> Take a look at some of the ILM procucts out there. See what they do.
> You'll find ILM is just HSM rebranded with a few more bells and
> whistles attached. Information Lifecycle Management, the name alone
> should fire off some bells and sirens for you to deal with your
> management.
>
> The flip side is ILM is a great resume builder. However, I'd bet that
> if you did due diligence, set your requirements, selected a vendor,
> then got a quote, management would be happy to sign PO's for TB's of
> storage for the next 2-3 years.
>
> Don;t forget all the hidden costs of ILM; it's not just the software.
> Tape, disk, switch ports, FC ports, whatever. Make sure you add it
> all in.
>
> ~F
Thanks... well I get dinged because its my data thats increasing the
backup restore window times... and the backup guys say I have too much
data to be restored...
In any event, I understand your arguments.
One other question... IN our company, overall we must have over 100TB
of data (I just manage 4TB of that)... I cant help but imagine that
someone should be consolidating all of this... I will raise this to
management, but before I do so, do folks here think this is a big
problem just in our company or are others out there like this... ?
How many others have a similar situation?
Thanks
Jon.
| |
| mapicella 2006-08-08, 1:15 am |
| jondelac@gmail.com wrote:
> Faeandar wrote:
>
> Thanks... well I get dinged because its my data thats increasing the
> backup restore window times... and the backup guys say I have too much
> data to be restored...
>
> In any event, I understand your arguments.
>
> One other question... IN our company, overall we must have over 100TB
> of data (I just manage 4TB of that)... I cant help but imagine that
> someone should be consolidating all of this... I will raise this to
> management, but before I do so, do folks here think this is a big
> problem just in our company or are others out there like this... ?
>
> How many others have a similar situation?
>
> Thanks
> Jon.
>
Not sure I understand. Your business unit (you) creates only 4% of the
company's data and YOU are under the gun to create an ILM strategy? I
don't think you'll find many others in that situation.
Does the remaining 96% have an ILM strategy?
Also I can't help noticing a disconnect. In addition to ILM, your
management is asking for faster backups but you think consolidation
(that would probably lengthen backup times) is needed. What gives?
If I may make a suggestion try to learn more about ILM (BTW is NOT a
technology). The EMC site is a good place where to start, not because
they are necessarily the best, but they have just about every peace to
implement an ILM strategy, and a rich Web site.
| |
| jondelac@gmail.com 2006-08-08, 1:15 am |
|
mapicella wrote:
> jondelac@gmail.com wrote:
> Not sure I understand. Your business unit (you) creates only 4% of the
> company's data and YOU are under the gun to create an ILM strategy? I
> don't think you'll find many others in that situation.
>
> Does the remaining 96% have an ILM strategy?
>
> Also I can't help noticing a disconnect. In addition to ILM, your
> management is asking for faster backups but you think consolidation
> (that would probably lengthen backup times) is needed. What gives?
>
> If I may make a suggestion try to learn more about ILM (BTW is NOT a
> technology). The EMC site is a good place where to start, not because
> they are necessarily the best, but they have just about every peace to
> implement an ILM strategy, and a rich Web site.
We have grown quite chaotic and our growth has been ad-hoc with many
regional offices (typically near the major customers)... we acquired a
lot of small consulting shops - so thats why we have so many
distributed sites... Our dept is however the main IT shop in the
company... no the others do not have one yet as well...
we are learning as we go - but we are paying a lot of money on our
data... and I dont think the remote distrbuted data is helping... I am
sure I should first bring their data in centrally and I think I need to
suggest that first...
Regarding faster backups - this is where I am led to beleive ILM
couldve helped (yes as you say, it is not a product but best
practises)... I guess to refine my original question, should I propose
something like this: 1. consolidate and 2. buy a product that lets us
move data from the netapp filers to some cheaper storage for older data
Jon
| |
|
|
jondelac@gmail.com wrote:
> mapicella wrote:
>
> We have grown quite chaotic and our growth has been ad-hoc with many
> regional offices (typically near the major customers)... we acquired a
> lot of small consulting shops - so thats why we have so many
> distributed sites... Our dept is however the main IT shop in the
> company... no the others do not have one yet as well...
>
> we are learning as we go - but we are paying a lot of money on our
> data... and I dont think the remote distrbuted data is helping... I am
> sure I should first bring their data in centrally and I think I need to
> suggest that first...
>
> Regarding faster backups - this is where I am led to beleive ILM
> couldve helped (yes as you say, it is not a product but best
> practises)... I guess to refine my original question, should I propose
> something like this: 1. consolidate and 2. buy a product that lets us
> move data from the netapp filers to some cheaper storage for older data
>
> Jon
| |
|
| In article <1154983121.889986.178130@i3g2000cwc.googlegroups.com>,
jondelac@gmail.com says...
>
>Hi,
>I am trying to figure out at what point I should be looking into some
>sort of ILM technology. In reading vendor case studies such as
>(http://www.netapp.com/library/cs/mustang.pdf), I dont see why I cant
>just keep adding more disks to my primary storage right now as demand
>increases... when should I be looking into ILM (at what storage
>capacity)? How many GigaBytes or TeraBytes before I should consider an
>ILM strategy?
>
i think that the 1st step in going to ILM is getting involved with
one of these new class of products that do "data classification".
Companies like Kazeon & Abrevity (the one we use) will discover
your data and present it to you in an understandable manner. They
will also / usually have the ability to let you move the data
(that you wish to move) and leave markers / pointers to the new
location. These solutions are also being used for finding legally
required data (within the files) and compliance issues.
_____ . .
' \\ . . |>>
O// . . |
\_\ . . |
| | . . . |
/ | . www.EvenEnterprises.com . . . |
/ .| info@EvenEnterprises.com . . . |
/ . | 310-544-9439 / 310-544-9309 fax . . . o
----------------------------------------------------------------------
Authorized - DIRECT VAR/VAD/Distributor for new mid-high end storage
iSCSI/NAS/SAN/RAID from EMC, HP, Equallogic, Quantum, ADIC, Abrevity
| |
| jondelac@gmail.com 2006-08-08, 1:14 pm |
|
Andy wrote:
> In article <1154983121.889986.178130@i3g2000cwc.googlegroups.com>,
> jondelac@gmail.com says...
>
> i think that the 1st step in going to ILM is getting involved with
> one of these new class of products that do "data classification".
> Companies like Kazeon & Abrevity (the one we use) will discover
> your data and present it to you in an understandable manner. They
> will also / usually have the ability to let you move the data
> (that you wish to move) and leave markers / pointers to the new
> location. These solutions are also being used for finding legally
> required data (within the files) and compliance issues.
>
> _____ . .
> ' \\ . . |>>
> O// . . |
> \_\ . . |
> | | . . . |
> / | . www.EvenEnterprises.com . . . |
> / .| info@EvenEnterprises.com . . . |
> / . | 310-544-9439 / 310-544-9309 fax . . . o
> ----------------------------------------------------------------------
> Authorized - DIRECT VAR/VAD/Distributor for new mid-high end storage
> iSCSI/NAS/SAN/RAID from EMC, HP, Equallogic, Quantum, ADIC, Abrevity
Great... will look into those products.
Also, has anyone used Enterprise Vault? If I use a product like Vault
to move my data from primary to a less expensive storage, what happens
when someone tries to access the data on the primary storage? Will the
system automatically go find it from the the place that vault moved the
data to?
Jon.
| |
| jondelac@gmail.com 2006-08-08, 1:14 pm |
|
jondelac@gmail.com wrote:
> Andy wrote:
>
> Great... will look into those products.
>
> Also, has anyone used Enterprise Vault? If I use a product like Vault
> to move my data from primary to a less expensive storage, what happens
> when someone tries to access the data on the primary storage? Will the
> system automatically go find it from the the place that vault moved the
> data to?
>
> Jon.
Looking at http://www.kazeon.com/solutions/mig..._archival.html, Can
you tell me how the product actually moves the data from one tier to
the next tier? What happens when the application tries to access the
data from the original location?
Jon
| |
|
| In article <1155060761.618983.164240@i3g2000cwc.googlegroups.com>,
jondelac@gmail.com says...
>
>
>jondelac@gmail.com wrote:
>
>Looking at http://www.kazeon.com/solutions/mig..._archival.html, Can
>you tell me how the product actually moves the data from one tier to
>the next tier? What happens when the application tries to access the
>data from the original location?
>
can't tell you about kazeon but the www.abrevity.com product has a
built in data mover
_____ . .
' \\ . . |>>
O// . . |
\_\ . . |
| | . . . |
/ | . www.EvenEnterprises.com . . . |
/ .| info@EvenEnterprises.com . . . |
/ . | 310-544-9439 / 310-544-9309 fax . . . o
----------------------------------------------------------------------
Authorized - DIRECT VAR/VAD/Distributor for new mid-high end storage
iSCSI/NAS/SAN/RAID from EMC, HP, Equallogic, Quantum, OverLand Storage
| |
| jondelac@gmail.com 2006-08-08, 7:13 pm |
|
Andy wrote:
> In article <1155060761.618983.164240@i3g2000cwc.googlegroups.com>,
> jondelac@gmail.com says...
>
>
> can't tell you about kazeon but the www.abrevity.com product has a
> built in data mover
>
> _____ . .
> ' \\ . . |>>
> O// . . |
> \_\ . . |
> | | . . . |
> / | . www.EvenEnterprises.com . . . |
> / .| info@EvenEnterprises.com . . . |
> / . | 310-544-9439 / 310-544-9309 fax . . . o
> ----------------------------------------------------------------------
> Authorized - DIRECT VAR/VAD/Distributor for new mid-high end storage
> iSCSI/NAS/SAN/RAID from EMC, HP, Equallogic, Quantum, OverLand Storage
Does it make the move seemless to the clients or do the clients have to
have some software installed on them in order to track the moved files
down?
| |
|
| In article <1155070916.423493.169520@n13g2000cwa.googlegroups.com>,
jondelac@gmail.com says...
>
>
>Andy wrote:
an[vbcol=seagreen]
[vbcol=seagreen]
[vbcol=seagreen]
>
>Does it make the move seemless to the clients or do the clients have to
>have some software installed on them in order to track the moved files
>down?
>
seemless, leaves a "pointer"
email me your real email address & i'll get you set up for a
free demo download
_____ . .
' \\ . . |>>
O// . . |
\_\ . . |
| | . . . |
/ | . www.EvenEnterprises.com . . . |
/ .| info@EvenEnterprises.com . . . |
/ . | 310-544-9439 / 310-544-9309 fax . . . o
----------------------------------------------------------------------
Authorized - DIRECT VAR/VAD/Distributor for new mid-high end storage
iSCSI/NAS/SAN/RAID from EMC, HP, Equallogic, Quantum, OverLand Storage
| |
| Ed Wilts 2006-08-09, 1:14 pm |
| jondelac@gmail.com wrote:
> I am trying to figure out at what point I should be looking into some
> sort of ILM technology. In reading vendor case studies such as
> (http://www.netapp.com/library/cs/mustang.pdf), I dont see why I cant
> just keep adding more disks to my primary storage right now as demand
> increases... when should I be looking into ILM (at what storage
> capacity)? How many GigaBytes or TeraBytes before I should consider an
> ILM strategy?
This isn't measured in GB or TB. It's measured by the dollars saved by
reducing (in theory) active storage and floor space versus the costs of
purchasing and managing the ILM software and hardware.
I've been involved in complicated cost/benefit analysis in the past
where ILM made absolutely no sense - it drove our costs up, not down.
We did this with a vendor that sent their product and marketing people
to our site to crunch the numbers.
There will be cases where ILM makes sense and identically sized
configurations where it doesn't. It depends on how you're using your
data, what you're paying for the storage, software, people, floor
space, etc. You need to factor in the time it takes to get data back,
the risk of losing your only copy of a file (or are you making 2 copies
to your archive?), the complexity of managing an ILM solution, and a
ton of other stuff. You also need to factor in the many hours you'll
be spending trying to see if this is the right way to change your
infrastructure and evaluating products.
.../Ed
| |
| Gunhaver05 2006-08-10, 7:16 am |
| > Well, since the 960's will expand to 64TB you've got plenty of grow
> room.
Just remember that the 64TB is the maximum raw usable. If you are
using WAFL file system on the drives as NAS, this actual number will be
much lower. This may also be lower depending on how many snapshots you
take and if you snapmirror the data off to a remote location. A good
rule of thumb on the NetApp's is to use a third of their raw number for
the maximum usable. Also, if you are not using the maximum size drive
available for the frame, you will never achieve this number. Using the
largest drive also means sacrificing speed for capacity.
| |
| Gunhaver05 2006-08-10, 7:16 am |
| > someone should be consolidating all of this... I will raise this to
> management, but before I do so, do folks here think this is a big
> problem just in our company or are others out there like this... ?
>
> How many others have a similar situation?
>
Lol. Believe me, you are not the only one. Many others out there.
| |
| Gunhaver05 2006-08-10, 7:16 am |
| Yes. I've managed an Exchange environment with Enterprise Vault. It's
not much fun for the users. It creates more work for the Exchange and
storage administrators as well. You will need lower speed, larger
capacity disks for longer retention of e-mails. If you are looking for
governmental compliance of the data, that requires more specific
storage hardware to achieve your goal.
Specifically with the vault, users will notice a difference. Unless
they've improved it since the 5.x version we were using, the user would
have a stub of a message. Once archived, they can click and open it,
the vault will retrieve the contents, and then display the message. At
some point, however, you may not want all the stubs in a users mailbox
because even those small pointers can add up to your storage groups.
After the stubs disappear from the mailbox, a user has to go into a
browser to see the message. The browser can typically be linked from
Outlook via a toolbar button.
Ultimately, there are many benefits to implementing the vault but not
without increased administration.
Hope this helps.
jondelac@gmail.com wrote:
> Andy wrote:
>
> Great... will look into those products.
>
> Also, has anyone used Enterprise Vault? If I use a product like Vault
> to move my data from primary to a less expensive storage, what happens
> when someone tries to access the data on the primary storage? Will the
> system automatically go find it from the the place that vault moved the
> data to?
>
> Jon.
| |
| jondelac@gmail.com 2006-08-10, 7:16 pm |
|
Gunhaver05 wrote:[vbcol=seagreen]
> Yes. I've managed an Exchange environment with Enterprise Vault. It's
> not much fun for the users. It creates more work for the Exchange and
> storage administrators as well. You will need lower speed, larger
> capacity disks for longer retention of e-mails. If you are looking for
> governmental compliance of the data, that requires more specific
> storage hardware to achieve your goal.
>
> Specifically with the vault, users will notice a difference. Unless
> they've improved it since the 5.x version we were using, the user would
> have a stub of a message. Once archived, they can click and open it,
> the vault will retrieve the contents, and then display the message. At
> some point, however, you may not want all the stubs in a users mailbox
> because even those small pointers can add up to your storage groups.
> After the stubs disappear from the mailbox, a user has to go into a
> browser to see the message. The browser can typically be linked from
> Outlook via a toolbar button.
>
> Ultimately, there are many benefits to implementing the vault but not
> without increased administration.
>
> Hope this helps.
>
> jondelac@gmail.com wrote:
Thanks... that was insightful. One question on ILM software such as
those from Kazeon and Abrevity... when they move a file from primary to
secondary, aparently they leave a "pointer" on the primary... A few
questions here:
1. Arent these "pointers" clutter at some point?
2. What exactly are these pointers... are they sym links... are they
shortcuts? Anything that needs to be installed on the client machines
to understand how to interpret these links?
Jon.
| |
| Faeandar 2006-08-10, 7:16 pm |
| On 10 Aug 2006 11:30:52 -0700, jondelac@gmail.com wrote:
>
>
>Thanks... that was insightful. One question on ILM software such as
>those from Kazeon and Abrevity... when they move a file from primary to
>secondary, aparently they leave a "pointer" on the primary... A few
>questions here:
>
>1. Arent these "pointers" clutter at some point?
>2. What exactly are these pointers... are they sym links... are they
>shortcuts? Anything that needs to be installed on the client machines
>to understand how to interpret these links?
>
>Jon.
They leave what is essentially a meta-inode. It's not a symlink and
nothing needs to be done to the client (or the server for that
matter).
They are indeed clutter at some point. Imagine having migrated an
entire volume's worth of data, say 10m files. Now you want to remove
that volume because your upgrading/migrating to a newer faster filer.
You actually have to migrate the meta-inodes and change automount maps
(or DFS entries or whatever you use).
Not to say this isn't worth it, just that there's no cure-all.
~F
| |
| jondelac@gmail.com 2006-08-10, 7:16 pm |
|
Faeandar wrote:
> On 10 Aug 2006 11:30:52 -0700, jondelac@gmail.com wrote:
>
>
>
> They leave what is essentially a meta-inode. It's not a symlink and
> nothing needs to be done to the client (or the server for that
> matter).
>
> They are indeed clutter at some point. Imagine having migrated an
> entire volume's worth of data, say 10m files. Now you want to remove
> that volume because your upgrading/migrating to a newer faster filer.
> You actually have to migrate the meta-inodes and change automount maps
> (or DFS entries or whatever you use).
>
> Not to say this isn't worth it, just that there's no cure-all.
>
> ~F
Is there some place I can read more about these "meta-inodes"? Google
search yeilds cryptic results...
Jon
| |
| Faeandar 2006-08-11, 1:14 am |
| On 10 Aug 2006 16:41:23 -0700, jondelac@gmail.com wrote:
>
>Faeandar wrote:
>
>Is there some place I can read more about these "meta-inodes"? Google
>search yeilds cryptic results...
>
>Jon
Note I said "essentially". I'm not sure that's even a real term but
it gets the concept across.
Try kazeon's web site and product spec sheets. You may actually have
to talk to a Kazeon rep to get the full scoop (or at least as full as
you're gonna get without being a customer).
~F
| |
|
| Another product is from Arkivio. They won NetApp's Innovation Award
last year. It can MOVE files, leaving nothing behind (relocation),
COPY files (for replication) or "migrate" files leaving behind one of
several options: a Windows shortcut for CIFS shares, a symlink for NFS
exports, or in the case of NetApp/Celerra, a host-vendor-specific stub
file which can be used through either CIFS or NFS.
I take issue with the earlier post suggesting that ILM is generally for
legal purposes. A "part" of ILM can include retention criteria, which
can be set for a variety of reasons, including compliance requirements.
ILM is more about how to manage data over its life: lots of data, as
it ages, doesn't need to be treated the same. I'm sure we all have old
files that, while we may not yet want to permanently delete it, it can
move to less expensive and lower performance storage. If it is again
needed, it can be accessed, just maybe not as quickly (a few seconds vs
sub-second). In addition, if I can migrate these "little used" or
"never updated" files to a long-term archive tier of storage, I can
backup that tier much less frequently, say quarterly incrementals (when
I archive newly-eligible files) and annual fulls. If half my data fits
into this catagory (typical for non-DB files), that can eliminate half
my backups on a weekly basis. It's all still available, all still
protected (archive can be RAID), all still backed up (backups from
before it was moved plus less frequent backups after it was moved).
And the archive tier storage can be a fraction of the cost of tier-1.
Plus, if someone wants to access it, the file can either be accessed
directly from its current location, or it can be recalled back to
tier-1 disk.
As for what's left behind being clutter, if you don't need/want them
anymore, just delete them. Just make sure that the ILM solution can go
back and cleanup the original files off the repository tier afterwards.
If you migrate to new disk/storage, it's a lot quicker to move those
small links than the original files.
| |
| jondelac@gmail.com 2006-08-15, 7:14 am |
|
MikeB wrote:
> Another product is from Arkivio. They won NetApp's Innovation Award
> last year. It can MOVE files, leaving nothing behind (relocation),
> COPY files (for replication) or "migrate" files leaving behind one of
> several options: a Windows shortcut for CIFS shares, a symlink for NFS
> exports, or in the case of NetApp/Celerra, a host-vendor-specific stub
> file which can be used through either CIFS or NFS.
>
> I take issue with the earlier post suggesting that ILM is generally for
> legal purposes. A "part" of ILM can include retention criteria, which
> can be set for a variety of reasons, including compliance requirements.
> ILM is more about how to manage data over its life: lots of data, as
> it ages, doesn't need to be treated the same. I'm sure we all have old
> files that, while we may not yet want to permanently delete it, it can
> move to less expensive and lower performance storage. If it is again
> needed, it can be accessed, just maybe not as quickly (a few seconds vs
> sub-second). In addition, if I can migrate these "little used" or
> "never updated" files to a long-term archive tier of storage, I can
> backup that tier much less frequently, say quarterly incrementals (when
> I archive newly-eligible files) and annual fulls. If half my data fits
> into this catagory (typical for non-DB files), that can eliminate half
> my backups on a weekly basis. It's all still available, all still
> protected (archive can be RAID), all still backed up (backups from
> before it was moved plus less frequent backups after it was moved).
> And the archive tier storage can be a fraction of the cost of tier-1.
> Plus, if someone wants to access it, the file can either be accessed
> directly from its current location, or it can be recalled back to
> tier-1 disk.
>
> As for what's left behind being clutter, if you don't need/want them
> anymore, just delete them. Just make sure that the ILM solution can go
> back and cleanup the original files off the repository tier afterwards.
> If you migrate to new disk/storage, it's a lot quicker to move those
> small links than the original files.
Now that answers my questions!
Thanks Much
| |
| Faeandar 2006-08-15, 7:13 pm |
| On 12 Aug 2006 09:12:31 -0700, "MikeB" <michaelb314@gmail.com> wrote:
>Another product is from Arkivio. They won NetApp's Innovation Award
>last year. It can MOVE files, leaving nothing behind (relocation),
>COPY files (for replication) or "migrate" files leaving behind one of
>several options: a Windows shortcut for CIFS shares, a symlink for NFS
>exports, or in the case of NetApp/Celerra, a host-vendor-specific stub
>file which can be used through either CIFS or NFS.
That's the rub in most cases, moving data is easy but making it
transparent or quick for the user community is not. With DFS it can
be a short 5 minute downtime/cutover. With NFS it's alot trickier.
Leaving behind pointers can be a mess and the cutover still has to
happen sometime, unless you want to leave all that muck about.
>
>As for what's left behind being clutter, if you don't need/want them
>anymore, just delete them. Just make sure that the ILM solution can go
>back and cleanup the original files off the repository tier afterwards.
> If you migrate to new disk/storage, it's a lot quicker to move those
>small links than the original files.
Moving data is actually quite simple and very fast, it's the follow-up
that can get you. NetApp in particular has a great tool for
replication and migration, Snapmirror. I use it all the time. So
making data go from A to B really takes no effort. But making data go
from A to B without users knowing it happened or making the cutover
fast is not easy.
~F
| |
|
| Faeandar wrote:
> On 12 Aug 2006 09:12:31 -0700, "MikeB" <michaelb314@gmail.com> wrote:
>
>
> That's the rub in most cases, moving data is easy but making it
> transparent or quick for the user community is not. With DFS it can
> be a short 5 minute downtime/cutover. With NFS it's alot trickier.
>
Who's talking about cutover? You seem to be describing transparent
relocation of data, say from an old filer to a new one, while users are
actually trying to access it. That's a different animal - I'm
referring to moving old, little accessed data from Tier-1 to Tier-n.
ILM is about putting data on the storage that best suits users'
requirements for that data. Need quick access to read/update data -
put it on expensive, fast Tier-1 storage. Have old data that may or
may not be accessed this year - put it on cheaper, slower Tier-2
storage, and don't back it up again every weekend. And since it's
little accessed, it's VERY unlikely someone happens to try to access
one of those files at the PRECISE moment it is being migrated. If a
file you haven't accessed in 6 months moves in the middle of the night,
do you care (assuming a pointer is left in its place)?
> Leaving behind pointers can be a mess and the cutover still has to
> happen sometime, unless you want to leave all that muck about.
>
Not sure what you mean by pointers being a mess - as opposed to what,
leaving the files behind? Pointers are a lot quicker/easier to backup
and relocate (if there's that need) than the original files.
>
> Moving data is actually quite simple and very fast, it's the follow-up
> that can get you. NetApp in particular has a great tool for
> replication and migration, Snapmirror. I use it all the time. So
> making data go from A to B really takes no effort. But making data go
> from A to B without users knowing it happened or making the cutover
> fast is not easy.
>
> ~F
Agreed - Snapmirror works great for replication and migration, but
that's not ILM, it has nothing to do with the changing value of the
data, just the changing LOCATION of it. ILM is a more sophisticated
twist on HSM, and there's little transparent (to users) about HSM
migration/recall.
|
|
|
|
|