Data Storage - Real life case: build my own high-availability iSCSI SAN: howto?

Author

Real life case: build my own high-availability iSCSI SAN: howto?

news.xs4all.nl

2004-10-16, 2:45 am

Hi, I am investigating how to create a high-availability iSCSI SAN. I have
investigated Intransa and Equallogic iSCSI SANs and EMC and other Fiber
Channel SANs, but they are all way to expensive. Especially iSCSI which I
expect to undergo many changes and improvements in the coming years making
my investment a risk. I have also investigated low-budget iSCSI SANs, but
they always lack something like a second NIC for a multi-path, second fan,
enough space, S-ATA, enough IOPS etc.

Therefor I would like to know if it is possible to build an iSCSI SAN
myself. I need some help here!

I have several Servers each running instances of MS SQL-Server normal
edition. I expect to give the a HBA each.
Question:
- will it be better to use a HBA or should I add a second 10GB NIC using the
Microsoft iSCSI initiator?
- if the first is true, what is the fastest available in the market?

Now I assume I have to connect the servers with their HBA to an iSCSI
target. How to create such a target? I have read about a Linux initiative.
Do I buy two servers for redundancy and install Linux with this module? Or
is there any other software available?
Other Questions:
- does this solution provide support for redundant servers?
- what type do these servers need to be?
- what will the data throughput and IOPS be of this solution?
- does this target also need something like an HBA or will this be a regular
NIC?
- will both servers have different ip's, or do they need to be load balanced
for a single IP?
- is there any software available to manage the SAN (snap shots, volume
creation etc)

Then this target needs to be connected to a disk array, I assume. Can this
be a plain Serial ATA disk array?
Questions:
- what will be a good and affordable disk array?
- how to connect the array to both target?
- can I use multiple arrays?
- how scalable will this solution be?

I understand that I have a lot of questions, but I hope you guys and girls
will have some time to assist me on this road I have never been before.

Kind regards, Maurice.

JT Johnson

2004-10-16, 2:45 am

"news.xs4all.nl" <iscsi@tijhuis.com> wrote in message news:<416af257$0$559$e4fe514c@news.xs4all.nl>...
> Hi, I am investigating how to create a high-availability iSCSI SAN. I have
> investigated Intransa and Equallogic iSCSI SANs and EMC and other Fiber
> Channel SANs, but they are all way to expensive. Especially iSCSI which I
> expect to undergo many changes and improvements in the coming years making
> my investment a risk. I have also investigated low-budget iSCSI SANs, but
> they always lack something like a second NIC for a multi-path, second fan,
> enough space, S-ATA, enough IOPS etc.
>
> Therefor I would like to know if it is possible to build an iSCSI SAN
> myself. I need some help here!
>
> I have several Servers each running instances of MS SQL-Server normal
> edition. I expect to give the a HBA each.
> Question:
> - will it be better to use a HBA or should I add a second 10GB NIC using the
> Microsoft iSCSI initiator?
> - if the first is true, what is the fastest available in the market?
>
> Now I assume I have to connect the servers with their HBA to an iSCSI
> target. How to create such a target? I have read about a Linux initiative.
> Do I buy two servers for redundancy and install Linux with this module? Or
> is there any other software available?
> Other Questions:
> - does this solution provide support for redundant servers?
> - what type do these servers need to be?
> - what will the data throughput and IOPS be of this solution?
> - does this target also need something like an HBA or will this be a regular
> NIC?
> - will both servers have different ip's, or do they need to be load balanced
> for a single IP?
> - is there any software available to manage the SAN (snap shots, volume
> creation etc)
>
> Then this target needs to be connected to a disk array, I assume. Can this
> be a plain Serial ATA disk array?
> Questions:
> - what will be a good and affordable disk array?
> - how to connect the array to both target?
> - can I use multiple arrays?
> - how scalable will this solution be?
>
> I understand that I have a lot of questions, but I hope you guys and girls
> will have some time to assist me on this road I have never been before.
>
> Kind regards, Maurice.

Maurice,

First, I think you might be a bit confused as to iSCSI vs Fiber
Channel. iSCSI and Fiber Channel are protocols, not technologies.
iSCSI and Fiber Channel use the same SCSI drives. iSCSI is simply
SCSI encapsulated in IP. Translation occurs within whichever OS the
hardware vendor creates. Therefore, an HBA is not required for iSCSI
since the iSCSI hardware will be connected directly into your data
network. You'd then create LUN's via the iSCSI controller (head unit)
and give a server permission to connect to the LUN. If you were to
use HBA's you'd be connecting via Fiber Channel and would require a FC
switch such as McData, QLogic, or Brocade.

If you have many servers (say, more than 30 or so) that you'd like to
connect to this solution, I think that 10Gbit switches will cost you
more than if you were to purchase a descient SAN to begin with. Last
time I looked, a 24 port cisco 10Gbit switch was going for more than
$4500 per port. Factor in the server NIC's and dual attaching and
your talking bucks.

Serial ATA is also different from iSCSI in that SATA is IDE drives
instead of SCSI. Again, in a larger environment, SATA won't give the
performance of SCSI, not by a long shot. But we use it for disk to
disk backups and it's a lot better than tape.

Unfortunatly, I haven't had experience with these other vendors you
mentioned so I can't be of much help. I just thought I'd get you
started on the technologies to help you find what you need.

Thanks,
JT

news.xs4all.nl

2004-10-16, 2:45 am

Thank you JT, this was very helpfull. I agree that I will need a 10GB switch
as well and this may become more expensive than the most expensive iSCSI
array I have seen around. I am not sure about the HBA's, since I have seen
HBA's for iSCSI to be able to handle the traffic without using the server's
processor. Take a look at http://www.qlogic.com/products/iscsi/checklist.asp
for instance.

Well, would a selfmade iSCSI SAN based on SCSI drives be a solution, but
then based on 1GB traffic and switches?

Anyone else? Please?

Maurice.

"JT Johnson" <jtjohnson@faegre.com> wrote in message
news:b09ab426.0410121205.537fccc0@posting.google.com...
> "news.xs4all.nl" <iscsi@tijhuis.com> wrote in message
news:<416af257$0$559$e4fe514c@news.xs4all.nl>...
have[vbcol=seagreen]
I[vbcol=seagreen]
making[vbcol=seagreen]
but[vbcol=seagreen]
fan,[vbcol=seagreen]
the[vbcol=seagreen]
initiative.[vbcol=seagreen]
Or[vbcol=seagreen]
regular[vbcol=seagreen]
balanced[vbcol=seagreen]
this[vbcol=seagreen]
girls[vbcol=seagreen]
>
>
> Maurice,
>
> First, I think you might be a bit confused as to iSCSI vs Fiber
> Channel. iSCSI and Fiber Channel are protocols, not technologies.
> iSCSI and Fiber Channel use the same SCSI drives. iSCSI is simply
> SCSI encapsulated in IP. Translation occurs within whichever OS the
> hardware vendor creates. Therefore, an HBA is not required for iSCSI
> since the iSCSI hardware will be connected directly into your data
> network. You'd then create LUN's via the iSCSI controller (head unit)
> and give a server permission to connect to the LUN. If you were to
> use HBA's you'd be connecting via Fiber Channel and would require a FC
> switch such as McData, QLogic, or Brocade.
>
> If you have many servers (say, more than 30 or so) that you'd like to
> connect to this solution, I think that 10Gbit switches will cost you
> more than if you were to purchase a descient SAN to begin with. Last
> time I looked, a 24 port cisco 10Gbit switch was going for more than
> $4500 per port. Factor in the server NIC's and dual attaching and
> your talking bucks.
>
> Serial ATA is also different from iSCSI in that SATA is IDE drives
> instead of SCSI. Again, in a larger environment, SATA won't give the
> performance of SCSI, not by a long shot. But we use it for disk to
> disk backups and it's a lot better than tape.
>
> Unfortunatly, I haven't had experience with these other vendors you
> mentioned so I can't be of much help. I just thought I'd get you
> started on the technologies to help you find what you need.
>
> Thanks,
> JT

Andrew Gideon

2004-10-16, 2:45 am

JT Johnson wrote:

> If you have many servers (say, more than 30 or so) that you'd like to
> connect to this solution, I think that 10Gbit switches will cost you
> more than if you were to purchase a descient SAN to begin with.

JT:

First, I want to say that I enjoyed your post. It very clearly explained a
lot.

However, I didn't understand the reasoning behind what I've quoted above.
Of course, port costs need to be considered when pricing a solution. But
I'm not sure what you mean by comparing this to "a decent SAN to begin
with". Wouldn't there still be the cost of the network (either IP or FC)
ports (plus HBAs for FC or GigE adaptors for IP) with any SAN solution,
decent or not?

Thanks...

- Andrew

Jochen Kaiser

2004-10-16, 2:45 am

Hello,

> I have several Servers each running instances of MS SQL-Server normal
> edition. I expect to give the a HBA each.
> Question:
> - will it be better to use a HBA or should I add a second 10GB NIC using the
> Microsoft iSCSI initiator?
===
As a previous poster has already pointed out, 10GbE is quite costly,
more costly in fact than FC-based SAN. Apart from this fact most
application won't need a high streaming throughput and most low-price
arrays won't be able to deliver 500MB/sec throughput anayway.I'd say
that normal sized MSSQL servers (5 to 30GB db size) can proabably live
on a single 100Mbit interface, throughput-wise that is. The problem in
databases is latency and the number of IOs which can be achieved by the
hardware per second. iSCSI is not very good at providing fast and many
IOs and SATA drives are unfortunately not good either, but given normal
db servers I'd say latency is ok, but not perfect.

> - if the first is true, what is the fastest available in the market?
===
Alacritech claims that theirs is the fastest, Software initators are
even faster IMHO but do of course consume some CPU cycles (less than the
HBA vendors would like you to think though).

> Now I assume I have to connect the servers with their HBA to an iSCSI
> target. How to create such a target? I have read about a Linux initiative.
> Do I buy two servers for redundancy and install Linux with this module? Or
> is there any other software available?
===
Apart from this you can visit WASABITECH, they have a solution for NETBSD.

> Other Questions:
> - does this solution provide support for redundant servers?
> - what type do these servers need to be?
> - what will the data throughput and IOPS be of this solution?
> - does this target also need something like an HBA or will this be a regular
> NIC?
> - will both servers have different ip's, or do they need to be load balanced
> for a single IP?
> - is there any software available to manage the SAN (snap shots, volume
> creation etc)
===
My budget choice would be a pair of linux servers, cluster config'd to
some scsi trays with 4+ controllers each. Either normal scsi controllers
(adaptec) and software raid or raid controllers by lsilogic. The
solution can probably be configured redundant. The servers should offer
at least half as many GbE ports as there are targets.
From a performance perspective I'd say you'd loose approx one-third over
a DAS solution using the same hardware and would calculate at 20-30 IOPS
and 5-8 MBytes of throughput per 15k scsi drive in a RAID 5 setup. With
SATA driver you'd probabaly end up with 15-20 IOPS and 3-5MB/s per drive.

Now the downside:
A redunant approach is difficult to setup and veen more probably
unstable than a non-redundant setup. A Linux solution requires a large
amount of time for implementation, configuration and testing.
Scalability of the solution will be unpredicatble, I'd guess that no one
can tell you whether you can build an iSCSI active-active NiC
failover-team under Linux if you'd need more than 1 GbE.

On the upside:
You'll probably be the most proficient iSCSI Linux User in 6 months

A server (2x3GHz or similar, 2GB Ram, 3 dual channel scsi controllers,
2-3 dual GbE) will cost only 4k EUR or so, and a storage brick with
redundant paths and 16x 73GB@15k disks can be had for 10k EUR or so.
Three bricks and two servers, under 50k EUR for a redundant config ain't
too bad. If you sacrifice redundancy you'd probably come as cheap as 12k
or so with a big 5U server having 24x SATA, two SATA RAId ctrls and 6x GbE.

HTH,

Jochen

Nik Simpson

2004-10-16, 2:45 am

news.xs4all.nl wrote:
>
> I have several Servers each running instances of MS SQL-Server normal
> edition. I expect to give the a HBA each.

I beleive there are some hardware implementations of the iSCSI initiator
which are to all intents and purposes iSCSI HBAs, I beleive QLogic has one
and maybe LSI. A step down the food chain is a Gbit Enet card with a TCP/IP
Offload engine (also known as a ToE) which offloads much of the IP frame
processing from the host OS/CPU and should be considerably more efficient
than a dumb NIC.

Most cost-effective at this time is going to be ToE approach.

> Question:
> - will it be better to use a HBA or should I add a second 10GB NIC
> using the Microsoft iSCSI initiator?

A software iSCSI initiator like that shipped by Microsoft couldn't fill a
1Gbit pipe without dragging the server to its knees processing all the IP
frames, so no point in even looking at a 10Gbit network for iSCSI traffic to
a single server at this point.

> - if the first is true, what is the fastest available in the market?
>

Sorry, can't help with specific product recommendations.

> Now I assume I have to connect the servers with their HBA to an iSCSI
> target. How to create such a target?

Two ways

1. Buy an array that supports iSCSI connections, major vendors like EMC are
just starting to trickle that sort of support out. There are a number of
startups though that have had product for some time, Lefthand, Intransa to
name but two.

2. Take a software pacakge like DataCore's SAN Melody or FalconStor's IPStor
and load it on a standard x86 server. This turns the server and its drives
into a the equivalent of a iSCSI array.

> I have read about a Linux
> initiative. Do I buy two servers for redundancy and install Linux
> with this module? Or is there any other software available?
> Other Questions:
> - does this solution provide support for redundant servers?
> - what type do these servers need to be?
> - what will the data throughput and IOPS be of this solution?

If the software is well designed, then you can get excellent performance
from a standard x86 server platform, particularly if the software uses
memopry for I/O caching. However your mileage may vary, it would help to
understand what the servers currently use and whether they are I/O bound.

> - does this target also need something like an HBA or will this be a
> regular NIC?
I think we are getting initiators and targets confused, or at least I am:-)

An iSCSI initiator (for LINUX or Windows) is software that runs on the
application server and does the SCSI to iSCSI command set translations and
send/recieves data from a disk somewhere on the net, i.e. it initiates a I/O
request.

An iSCSI target is whatever looks like a disk to the initiator, i.e. the
target is usually implemented inside the array, or in the software that
turned an x86 server into a psuedo array.

> - will both servers have different ip's, or do they need to be load
> balanced for a single IP?
> - is there any software available to manage the SAN (snap shots,
> volume creation etc)

The commercial products like FalconStor & DataCore offer certainly offer
these functions, as do the arrays from folks like LeftHand.

>
> Then this target needs to be connected to a disk array, I assume. Can
> this be a plain Serial ATA disk array?

No, the target is the disk array :-) As to what sort of disks are in the
array it really doesn't matter, none of them take iSCSI natively so there is
a controller in the array that speaks iSCSI to the outside world and
translates that to SATA or SCSI or FC, depending on the native interface of
the drives used in the array. The software in a DataCore type solution
performs the same function as the controller in a conventional array.

> Questions:
> - what will be a good and affordable disk array?

It certainly can be, you could put a 2-3TB array with mirroring and pair of
servers using something SANMelody or IPStor for very little money. The iSCSI
arrays from folks like Lefthand are also pretty reasonable.

> - how to connect the array to both target?

I think we've got into thios target/initiator discussion again. Your hosts
are the initiators, the array is the target. Connection should probably be a
1GBit switch link for each server, amybe two if you can find an iSCSI
initiator that understands trunking.

> - can I use multiple arrays?

Yes.

> - how scalable will this solution be?

How scalable does it need to be? Are you measuring scalability in terms of
capacity, performance, # of hosts connected?

--
Nik Simpson