I'm trying to evaluate NetApp's SAN functionality, not the iSCSI way but
Fibre Channel.

Network Appliance offers FC-SAN connectivity for real fibre SAN's. Inside
the box the LUN's they offer exist as files on top of their WAFL filesystem.
So the LUN's are pure virtual. To me that sounds like it could be beneficial
as this allows NetApp to do all the clustering, cache, snapshot and remote
replication over Ethernet stuff they do on any of the files stored on their
box.

Has anyone tried their FC-SAN setup at all? How does the performance compare
to, say, an IBM DS4300 or an HDS 9570 for streaming and transactional loads?
What experiences have you had, good or bad?

Rob

[ Post a follow-up to this message ]

On Mon, 14 Feb 2005 22:55:01 +0100, "Rob Turk"
<_wipe_me_r.turk@chello.nl> wrote:

>I'm trying to evaluate NetApp's SAN functionality, not the iSCSI way but
>Fibre Channel.
>
>Network Appliance offers FC-SAN connectivity for real fibre SAN's. Inside
>the box the LUN's they offer exist as files on top of their WAFL filesystem
.
>So the LUN's are pure virtual. To me that sounds like it could be beneficia
l
>as this allows NetApp to do all the clustering, cache, snapshot and remote
>replication over Ethernet stuff they do on any of the files stored on their
>box.
>
>Has anyone tried their FC-SAN setup at all? How does the performance compar
e
>to, say, an IBM DS4300 or an HDS 9570 for streaming and transactional loads
?
>What experiences have you had, good or bad?
>
>Rob
>

No personal experience for reasons I'm about to mention; but alot of
homework on it.

Fiber channel is considered an interrupt to the primary functions of
the filer, namely NFS and Snap*.  It takes precendence over CIFS but
just about everything does already.  Traffic out of FC on a filer is
secondary to Ethernet, so there's one thing to consider.  Of course,
if this will only be SAN then no issue.

Second, clustering good.  Cache good.  Snapshot and remote replication
problematic.
If you're using an application that NetApp officially supports (like
Exchange or SQL Server) then they have nice tools that tie snapshots
and IO quiesence together.  If not, then you need to manually (or
scripted, but your dime) halt IO to the LUN in order to get a good
snapshot.  Under Ethernet circumstances this is not an issue, Ontap
takes care of the traffic.
The same problem holds for remote replication, mainly Snap*.

The you have to factor in OS upgrades and such against your apps.
Over NFS, or even CIFS to a limited degree, apps will handle the
reboot time of an upgrade without barfing.  Not all apps will if they
are making direct access calls (or what they think are direct).  The
OS shields the app from the NFS timeouts in almost all cases.  And
CIFS is stateful so it will reconnect on next use (unless you were
actually transferring data, then you gotta start over).

Since no personal experience I can't speak to performance.  I hear
it's good, but given my first point I would tend to classify it at
best around 9570 performance.  At best.

Don't get me wrong, I love NetApp.  But SAN is not their thing.  They
have the best NAS on the market, bar none.  But SAN?  Not in my
opinion.

~F

[ Post a follow-up to this message ]

On Tue, 15 Feb 2005 02:52:50 GMT, Faeandar wrote:

[excellent appraisal snipped]

>Since no personal experience I can't speak to performance.  I hear
>it's good, but given my first point I would tend to classify it at
>best around 9570 performance.  At best.

I don't think it's that good... :-/

>Don't get me wrong, I love NetApp.  But SAN is not their thing.  They
>have the best NAS on the market, bar none.  But SAN?  Not in my
>opinion.

I agree with you completely, but I wanted to point out that just
because it might not be the best SAN in the world doesn't mean that
it's not suitable for the OP.

The OP didn't specify what he wants the storage for.  I wouldn't
necessarily want to run a huge OLTP system off the FC ports on a
Filer, but if I had lower performance requirements it might just suit
fine.

I think the NetApp approach works exceptionally well from a unified
storage pool perspective.  If I wanted one reasonably priced device
that would do pretty much everything well, NetApp would be a good
choice.

HVB.

[ Post a follow-up to this message ]

On Tue, 15 Feb 2005 11:30:48 +0000, HVB <devnull@127.0.0.1> wrote:


>
>I agree with you completely, but I wanted to point out that just
>because it might not be the best SAN in the world doesn't mean that
>it's not suitable for the OP.
>
>The OP didn't specify what he wants the storage for.  I wouldn't
>necessarily want to run a huge OLTP system off the FC ports on a
>Filer, but if I had lower performance requirements it might just suit
>fine.
>
>I think the NetApp approach works exceptionally well from a unified
>storage pool perspective.  If I wanted one reasonably priced device
>that would do pretty much everything well, NetApp would be a good
>choice.
>
>HVB.

The OP was talking about FC specifically, and in that case I would say
it's not acceptable.  Primarily because of unpredictable application
behavior during a reboot/upgrade/failover as well as the performance
interrupt.  Even if the the latter is not not important the former
still applies.

Now, if the actual intent is simply block level storage then by all
means try it.  But use iSCSI instead of FC.  iSCSI has configurable
timeouts very similar to NFS, and application behavior is much more
predictable.  And ethernet is what the filer was designed for...

~F

[ Post a follow-up to this message ]

On Tue, 15 Feb 2005 02:52:50 GMT, Faeandar <mr_castalot@yahoo.com>
wrote:

>On Mon, 14 Feb 2005 22:55:01 +0100, "Rob Turk"
><_wipe_me_r.turk@chello.nl> wrote:
> 
>
>No personal experience for reasons I'm about to mention; but alot of
>homework on it.
>
>Fiber channel is considered an interrupt to the primary functions of
>the filer, namely NFS and Snap*.  It takes precendence over CIFS but
>just about everything does already.  Traffic out of FC on a filer is
>secondary to Ethernet, so there's one thing to consider.  Of course,
>if this will only be SAN then no issue.
>
>Second, clustering good.  Cache good.  Snapshot and remote replication
>problematic.
>If you're using an application that NetApp officially supports (like
>Exchange or SQL Server) then they have nice tools that tie snapshots
>and IO quiesence together.  If not, then you need to manually (or
>scripted, but your dime) halt IO to the LUN in order to get a good
>snapshot.  Under Ethernet circumstances this is not an issue, Ontap
>takes care of the traffic.
>The same problem holds for remote replication, mainly Snap*.
>
>The you have to factor in OS upgrades and such against your apps.
>Over NFS, or even CIFS to a limited degree, apps will handle the
>reboot time of an upgrade without barfing.  Not all apps will if they
>are making direct access calls (or what they think are direct).  The
>OS shields the app from the NFS timeouts in almost all cases.  And
>CIFS is stateful so it will reconnect on next use (unless you were
>actually transferring data, then you gotta start over).
>
>Since no personal experience I can't speak to performance.  I hear
>it's good, but given my first point I would tend to classify it at
>best around 9570 performance.  At best.
>
>Don't get me wrong, I love NetApp.  But SAN is not their thing.  They
>have the best NAS on the market, bar none.  But SAN?  Not in my
>opinion.
>
>~F

Update to my original post.  I did some checking on the FC priority
thing.  Turns out only *backup* FC traffic is low priority, not data
transfer traffic like SAN or (duh) disk access.  I thought OT new the
difference between it's disk FC controllers and any other FC
controllers.  That is not the case, it only knows the difference
between dump and not-dump, not FC or non-FC.  So even backup traffic
over ethernet is low priority.

So one of my arguments against NetApp SAN is not valid.  However, my
gut still says bad idea.  But your gut may say differently.

~F

[ Post a follow-up to this message ]

"Faeandar" <mr_castalot@yahoo.com> wrote in message
 news:per911lg0v1f049l252t3d6se1294f9vke@
4ax.com...
>
> Update to my original post.  I did some checking on the FC priority
> thing.  Turns out only *backup* FC traffic is low priority, not data
> transfer traffic like SAN or (duh) disk access.  I thought OT new the
> difference between it's disk FC controllers and any other FC
> controllers.  That is not the case, it only knows the difference
> between dump and not-dump, not FC or non-FC.  So even backup traffic
> over ethernet is low priority.
>
> So one of my arguments against NetApp SAN is not valid.  However, my
> gut still says bad idea.  But your gut may say differently.
>
> ~F

Thank you for your in-dept analysis. My gut feeling is exactly like yours,
that's why I asked in the first place. The customer's intent is to hook up a
2-node AIX HACMP cluster to the FC-SAN side of the filer, then use async
replication to replicate to a secondary Filer on a fairly low-bandwidth
(10Mbit) link. The AIX machines are used for database stuff, mostly read.
I've been able to find next to nothing on real production iSCSI experiences
for AIX, and customer wants SAN, not NFS..

They also have a couple of Windows machines that they'd like to use CIFS on
the Filer too, and replicate that as well.

Rob

[ Post a follow-up to this message ]

On Thu, 17 Feb 2005 21:48:27 +0100, "Rob Turk"
<_wipe_me_r.turk@chello.nl> wrote:

>"Faeandar" <mr_castalot@yahoo.com> wrote in message
> news:per911lg0v1f049l252t3d6se1294f9vke@
4ax.com... 
>
>Thank you for your in-dept analysis. My gut feeling is exactly like yours,
>that's why I asked in the first place. The customer's intent is to hook up 
a
>2-node AIX HACMP cluster to the FC-SAN side of the filer, then use async
>replication to replicate to a secondary Filer on a fairly low-bandwidth
>(10Mbit) link. The AIX machines are used for database stuff, mostly read.
>I've been able to find next to nothing on real production iSCSI experiences
>for AIX, and customer wants SAN, not NFS..
>
>They also have a couple of Windows machines that they'd like to use CIFS on
>the Filer too, and replicate that as well.
>
>Rob
>

The "SAN not NFS" aspect is very old world.  We tested Oracle 10g RAC
over NFS on a pair of filers and it rocked.  Absolutely kicked XXX.

But the initial investment was more than the DB group wanted to spend,
so they ended up on an existing SAN.  They are already regretting it
because the 20 line scripts I provided as examples for backup,
replication, and test environment refreshes are in the process of
becoming 7 different scripts and a proprietary binary from the storage
vendor.  All that *after* a 3 day on-site consultancy and 2 weeks of
prep.

No idea how well it works for non-Oracle db's but Oracle over NFS is
fantastic.  Especially RAC.  They actually recommended NFS until they
got a GA release of OCFS.  But OCFS blows goats so NFS is still the
best solution in that case.

On a side note, someone (I believe it was CITI but I could be
mistaken) showed that NFS as a protocol is just as fast as Fiber
Channel (scsi).  It's all in the implementation...
Just food for thought.


We also have production iSCSI in place for SQL Server but no AIX
involved anywhere.  That is just a matter of testing the client
though.

~F

[ Post a follow-up to this message ]

Faeandar wrote:
> On Thu, 17 Feb 2005 21:48:27 +0100, "Rob Turk"
> <_wipe_me_r.turk@chello.nl> wrote:
> 
priority[vbcol=seagreen] 
data[vbcol=seagreen] 
the[vbcol=seagreen] 
traffic[vbcol=seagreen] 
my[vbcol=seagreen] 
yours,[vbcol=seagreen] 
hook up a[vbcol=seagreen] 
async[vbcol=seagreen] 
low-bandwidth[vbcol=seagreen] 
read.[vbcol=seagreen] 
experiences[vbcol=seagreen] 
CIFS on[vbcol=seagreen] 
>
> The "SAN not NFS" aspect is very old world.  We tested Oracle 10g RAC
> over NFS on a pair of filers and it rocked.  Absolutely kicked XXX.
>
> But the initial investment was more than the DB group wanted to
spend,
> so they ended up on an existing SAN.  They are already regretting it
> because the 20 line scripts I provided as examples for backup,
> replication, and test environment refreshes are in the process of
> becoming 7 different scripts and a proprietary binary from the
storage
> vendor.  All that *after* a 3 day on-site consultancy and 2 weeks of
> prep.
>
> No idea how well it works for non-Oracle db's but Oracle over NFS is
> fantastic.  Especially RAC.  They actually recommended NFS until they
> got a GA release of OCFS.  But OCFS blows goats so NFS is still the
> best solution in that case.
>
> On a side note, someone (I believe it was CITI but I could be
> mistaken) showed that NFS as a protocol is just as fast as Fiber
> Channel (scsi).  It's all in the implementation...
> Just food for thought.
>
>
> We also have production iSCSI in place for SQL Server but no AIX
> involved anywhere.  That is just a matter of testing the client
> though.
>
> ~F

[ Post a follow-up to this message ]

I would be cautious of NetApp endorsements for block level
requirements, especially in the absence of any benchmarks,
configuration info or any meaningful comparisons or stats. SAN
technology is not "old world" at all. There are situations where SAN
makes much more sense than NAS and visa-versa. For file services like
nfs and cifs NAS is the obvious choice. For block level requirements
like database or other disk intensive applications SAN is the obvious
choice.

As far as FC SAN with a NetApp I think this only makes sense if you
also need nfs and cifs although I still think that Netapp makes
absolutely the most expensive "file server" on the planet.

An alternative approach for consolidating the storage pool would be a
Hitachi array with a netapp gateway if you really need the netapp.
Otherwise possibly a SAN of some type with a hefty HP/Compaq or Dell
server running samba and nfs may work for a lot less money.

Regards,
Vic

[ Post a follow-up to this message ]

SAN is not an obvious choice for Oracle.
Read, for example, this:

http://www.google.ca/url?sa=U&start...>
2.pdf&e=7629


victor.engle@gmail.com wrote:
> I would be cautious of NetApp endorsements for block level
> requirements, especially in the absence of any benchmarks,
> configuration info or any meaningful comparisons or stats. SAN
> technology is not "old world" at all. There are situations where SAN
> makes much more sense than NAS and visa-versa. For file services like
> nfs and cifs NAS is the obvious choice. For block level requirements
> like database or other disk intensive applications SAN is the obvious
> choice.
>
> As far as FC SAN with a NetApp I think this only makes sense if you
> also need nfs and cifs although I still think that Netapp makes
> absolutely the most expensive "file server" on the planet.
>
> An alternative approach for consolidating the storage pool would be a
> Hitachi array with a netapp gateway if you really need the netapp.
> Otherwise possibly a SAN of some type with a hefty HP/Compaq or Dell
> server running samba and nfs may work for a lot less money.
>
> Regards,
> Vic
>

[ Post a follow-up to this message ]