Data Storage - I need a solution for really fast storage (GByte/sec)

Author

I need a solution for really fast storage (GByte/sec)

Giorgos P.

2004-11-23, 5:48 pm

Does anyone have a solution to the following problem:

requirements:
1)A storage unit-system
2)with a data transfer rate of up to 5 GByte/sec (yes that is Giga)
3)Storage capacity of up to 20 TByte.
4)Preferably an off the self solution (though suggesting an architecture of
eg. a specific raid controller would help)

Thanks in advance guys.

ps. by the way do you have to be like an expert to set up a raid system (the
only custom solution I've found up to now)

Faeandar

2004-11-23, 5:48 pm

On Wed, 24 Nov 2004 00:10:20 +0200, "Giorgos P." <gpouikli@ee.duth.gr>
wrote:

>Does anyone have a solution to the following problem:
>
>requirements:
>1)A storage unit-system
>2)with a data transfer rate of up to 5 GByte/sec (yes that is Giga)
>3)Storage capacity of up to 20 TByte.
>4)Preferably an off the self solution (though suggesting an architecture of
>eg. a specific raid controller would help)
>
>Thanks in advance guys.
>
>ps. by the way do you have to be like an expert to set up a raid system (the
>only custom solution I've found up to now)
>

You're asking for some very fast, very expensive storage. As long as
you are aware of that here's a couple:

HDS TagmaStore
Data Direct Networks S2A8500 (several for aggregate)

There are a couple of other options but they are much more complex,
and the more complex the less likely it will work....

~F

Malcolm Weir

2004-11-24, 2:45 am

On Wed, 24 Nov 2004 00:10:20 +0200, "Giorgos P." <gpouikli@ee.duth.gr>
wrote:

>Does anyone have a solution to the following problem:
>
>requirements:
>1)A storage unit-system
>2)with a data transfer rate of up to 5 GByte/sec (yes that is Giga)
>3)Storage capacity of up to 20 TByte.
>4)Preferably an off the self solution (though suggesting an architecture of
>eg. a specific raid controller would help)

Ummm... do the math: the maximum theoretical payload of 2Gb/s Fibre
Channel is 200MB/sec. So you want a *minimum* of 25 2Gb/s FC pipes.
Since no-one will get that throughput, you'll probably need at least
30 of them.

Then take your 20TB capacity, and you find each pipe fronts a piddling
666GB.

Something seems upside down, here!

[ Snip ]

>ps. by the way do you have to be like an expert to set up a raid system (the
>only custom solution I've found up to now)

No, but you *really* have to be an expert to design a system that will
do those sorts of rates. Think about it: what you want exceeds the
transfer capability of a dozen 64 bit 66MHz PCI busses!

Malc.

_firstname_@lr_dot_los-gatos_dot_ca.us

2004-11-24, 8:45 pm

In article <co0cfv$rbi$1@nic.grnet.gr>, Giorgos P. <gpouikli@ee.duth.gr> wrote:
>Does anyone have a solution to the following problem:
>
>requirements:
>1)A storage unit-system
>2)with a data transfer rate of up to 5 GByte/sec (yes that is Giga)

As Malcolm already said: I don't know any single computer (except
exotic supercomputers) that can feed a storage system at 5GB/sec. It
can easily be done by using a compute cluster with about 100 or 200
machines, each feeding 25 to 50 MB/second.

By the way, it is REALLY hard, using commodity-grade computers, to
write an application that can read or write at more than 25-50
MB/second sustained. The problem is that the application typically
wants to do interesting things with the data, and at those data rates,
there aren't very many CPU cycles around to do interesting things.

>3)Storage capacity of up to 20 TByte.

Any system that can handle the above bandwidth will have no problem
with the capacity. To get 5 GB/s sustained, you really need at least
at the bare minimum 150 to 200 disk drives. To get redundancy (which
is required in a system of this scale), and some safety margin, better
double that. At that point, capacity will be trivially taken care of.

>4)Preferably an off the self solution (though suggesting an architecture of
>eg. a specific raid controller would help)

No solution for this scale is simply off-the-shelf. Instead it can be
built from a large assembly of off-the-shelf components.

Here would be a few ways to do this. First option: Use a small number
(3 or 4) of really large RAID arrays (big Hitachi lightnings, high-end
IBM sharks, and so). Those cost a few million $ apiece. They each
have a few dozen fibre channel ports, each able to handle 200 MB/sec
(the reason you need multiple is that they can't handle the full 5
GB/s sustained internally, even though they have enough external
bandwidth). Building the fibre channel switching fabric for this
sustained bandwidth will be very epensive. Second option: Use many
many (maybe 25 or 50) small RAID arrays, for example 4-U rackmount
units with one FC port each, from second-tier manufacturers if you are
cost-sensitive. The storage solution will be much cheaper, but the
cost of the fibre channel fabric will be much higher. Third option:
Use generic x86 computers as storage devices, each equipped with a few
cheap RAID cards (for example, use 100 x86 computers, each with three
SATA-RAID cards, and each with 12 SATA disks). Connect everything via
gig-E. Use a storage protocol of your choice (iSCSI, Linux NBD,
cluster FS of your choice, if you are completely crazy you could even
use NFS). Problem is that now you need an absolutely massive Ethernet
switching fabric; to get to 5 GB/second you probably need multiple
high-end routers, interconnected with 10gig-E backbones.

As you can see, any system of this scale involves multiple hundreds of
machines, a massive network switching system, and several million $.

>ps. by the way do you have to be like an expert to set up a raid system (the
>only custom solution I've found up to now)

No, with a few hours of practice, a skilled sysadmin can set up a RAID
system. I mean a generic RAID system, with a dozen disks.

Setting up a system that handles 5 GB/sec is a completely different
kettle of fish. I have helped setting up a system that has similar
specs (about 30 TB, about 1 GB/sec sustained, using about 100
computers). It took between 5 and 10 people for about 1/2 to 1 year,
but using early pre-release hardware and software. At supercomputer
conferences, the supercomputer-class filesystems (GPFS, CXFS, Lustre)
can be set up within a few days for systems of this scale. But this
is all done by super-experts and gurus, usually developers of the
storage systems.

My advice would be: Either reduce about your requirements (if you
reduce the 5 GB/sec by an order of magnitude, life would become a lot
easier). Or contact one of the vendors of really large storage
systems and really large computer systems (the usual suspects), and
get them to work out a quote for you. You may decide to not go with
their quotes, and do the system yourself, but to start with, you need
serious professional help.

--
The address in the header is invalid for obvious reasons. Please
reconstruct the address from the information below (look for _).
Ralph Becker-Szendy _firstname_@lr_dot_los-gatos_dot_ca.us

Faeandar

2004-11-24, 8:45 pm

On Wed, 24 Nov 2004 00:25:07 -0800, Malcolm Weir <malc@gelt.org>
wrote:

>On Wed, 24 Nov 2004 00:10:20 +0200, "Giorgos P." <gpouikli@ee.duth.gr>
>wrote:
>
>
>Ummm... do the math: the maximum theoretical payload of 2Gb/s Fibre
>Channel is 200MB/sec. So you want a *minimum* of 25 2Gb/s FC pipes.
>Since no-one will get that throughput, you'll probably need at least
>30 of them.
>
>Then take your 20TB capacity, and you find each pipe fronts a piddling
>666GB.
>
>Something seems upside down, here!

Not necessarily upside down, but very very expensive. An HDS
Lightning has 32 storage ports, which will give you the 5GB/sec
requirement, purely aggregate througput though. There's about zero
chance of getting this throughput for a single system but if what the
OP wants is an aggregate then it's possible, expensive but possible.
The Lightning will also expand to 96TB so they OP can still get their
storage requirements.

But again, to get all those ports you need a fully configured
enterprise array. Something on the order of US$1.5+ million.

~F

Robert Wessel

2004-11-24, 8:45 pm

Malcolm Weir <malc@gelt.org> wrote in message news:<nre8q0dij0216cpmm5cdloi5suilcmp7g0@4ax.com>...
> On Wed, 24 Nov 2004 00:10:20 +0200, "Giorgos P." <gpouikli@ee.duth.gr>
> wrote:
>
>
> Ummm... do the math: the maximum theoretical payload of 2Gb/s Fibre
> Channel is 200MB/sec. So you want a *minimum* of 25 2Gb/s FC pipes.
> Since no-one will get that throughput, you'll probably need at least
> 30 of them.
>
> Then take your 20TB capacity, and you find each pipe fronts a piddling
> 666GB.
>
> Something seems upside down, here!
>
> [ Snip ]
>
>
> No, but you *really* have to be an expert to design a system that will
> do those sorts of rates. Think about it: what you want exceeds the
> transfer capability of a dozen 64 bit 66MHz PCI busses!

While I doubt the OP is really serious, or really understands what
he's asking for, the recent 3.2M tpm-c submission from IBM shows an
example of a configuration on this scale.

p5 595 with 64 1.9GHz POWER 5 CPUs, 2TB of RAM, 240TB of disk (6548
drives!), on *90* 2Gb FC links. Just write IBM a check for $17
million (and the storage is about half of that).

http://www.tpc.org/tpcc/results/tpc...sp?id=104111801

David

2004-11-27, 7:45 am

_firstname_@lr_dot_los-gatos_dot_ca.us wrote in message news:<1101315208.370655@smirk>...
> In article <co0cfv$rbi$1@nic.grnet.gr>, Giorgos P. <gpouikli@ee.duth.gr> wrote:
>
> As Malcolm already said: I don't know any single computer (except
> exotic supercomputers) that can feed a storage system at 5GB/sec. It
> can easily be done by using a compute cluster with about 100 or 200
> machines, each feeding 25 to 50 MB/second.
>

Or a lot less machines doing much large I/O rates.

> By the way, it is REALLY hard, using commodity-grade computers, to
> write an application that can read or write at more than 25-50
> MB/second sustained. The problem is that the application typically
> wants to do interesting things with the data, and at those data rates,
> there aren't very many CPU cycles around to do interesting things.
>

A reasonable dual-proc Windows box with a dual-port F/C card
can do line rate (400MB/s). For example, playing back or capturing
2K film requires about 320MB/s, mind you thats doing 24x12MB direct
I/O reads and writes per second. The challenge is making sure you have
the right storage to do those rates, and the application is written is
such a way that it has multiple I/O operations in progress at the same
time so it has some compute cycles for other things.

>
> Setting up a system that handles 5 GB/sec is a completely different
> kettle of fish. I have helped setting up a system that has similar
> specs (about 30 TB, about 1 GB/sec sustained, using about 100
> computers). It took between 5 and 10 people for about 1/2 to 1 year,
> but using early pre-release hardware and software. At supercomputer
> conferences, the supercomputer-class filesystems (GPFS, CXFS, Lustre)
> can be set up within a few days for systems of this scale. But this
> is all done by super-experts and gurus, usually developers of the
> storage systems.
>

Agreed, 5GB/s is a serious amount of I/O and needs significant storage,
infrastructure and a good filesystem, so vendors like SGI are likely
candidates.

David

seb

2004-11-28, 7:45 am

== Quote from Giorgos P. (gpouikli@ee.duth.gr)'s article


> Does anyone have a solution to the following problem:
> requirements:

Yes, pending validation

> ps. by the way do you have to be like an expert to set up a raid system (the
> only custom solution I've found up to now)

Yes

Here is the proposed solution

5x HP XP128 Disk Array with 4 CHIP controllers where each controller has 2 fibre
channel interfaces (2Gbit). You must make sure you do not format the array in
RAID5, and you will have lots of wasted disk space on the array to max out the
disk performance.

To each XP128 you will connect 2x HP RP2470 machines with each 2x 2Gbit FC HBA's

Oh yes, you will also need an HP storage consultant, and your own storage
consultant to help you set it up, perform benchmarks and testing, maybe re-design
the solution a little,...

Does your budget for next year include at least 500,000 $ for this project ?

Bye
seb

Anton Rang

2004-11-30, 5:45 pm

"Giorgos P." <gpouikli@ee.duth.gr> writes:
> Does anyone have a solution to the following problem:
>
> requirements:
> 1)A storage unit-system
> 2)with a data transfer rate of up to 5 GByte/sec (yes that is Giga)
> 3)Storage capacity of up to 20 TByte.
> 4)Preferably an off the self solution (though suggesting an architecture of
> eg. a specific raid controller would help)

We have SAM-QFS customers sustaining over 8 GB/second on a single file
system (together with 8 GB/second of data coming into the machine from
other sources, to be stored to disk).

However, they're using a Sun F15K and a lot of RAID hardware, probably
a large number of the high-end LSI Logic or StorageTek arrays.

I don't know of any cheap hardware that can sustain 5 GB/second.
Actually, you probably need 10 GB/second to memory? Where is all of
this data coming from?

-- Anton

David Boles

2004-12-02, 5:45 pm

_firstname_@lr_dot_los-gatos_dot_ca.us wrote:
> In article <co0cfv$rbi$1@nic.grnet.gr>, Giorgos P.
<gpouikli@ee.duth.gr> wrote:
>
>
>
> As Malcolm already said: I don't know any single computer (except
> exotic supercomputers) that can feed a storage system at 5GB/sec. It
> can easily be done by using a compute cluster with about 100 or 200
> machines, each feeding 25 to 50 MB/second.

Hmmm, I think this is much more doable than you think. An off-the
shelf 4-way Opteron system (Sun v40z) can read data from a network
using (IIRC) 2 10Gb Ethernet ports and write that data through NTFS
at something like 1.2GBytes/sec sustained. That's shoving the data
up through the TCP/IP stack on one side and writing it through the
filesystem and OS on the other, all on a single 4P system.

I don't think you'll get to 5GByte/sec with that one box, but you'd
get at least 3GByte/sec. IIRC the box has 4 64/133 PCI-X buses,
1 64/100, and one 64/66. Each of the 64/133 slots is an independent
bus, while the other two are on seperate buses as well but may each
have some low-bandwidth internal peripherals on them as well.

Stop using systems with a FSB and a lot of what are perceived as
limitations of commodity systems simply vanish. Such systems can
be purchased for quite modest sums.

Cheers,

- db

imi

2004-12-22, 7:01 am

quote:
Originally posted by Giorgos P.
Does anyone have a solution to the following problem:

requirements:
1)A storage unit-system
2)with a data transfer rate of up to 5 GByte/sec (yes that is Giga)
3)Storage capacity of up to 20 TByte.
4)Preferably an off the self solution (though suggesting an architecture of
eg. a specific raid controller would help)

Thanks in advance guys.

ps. by the way do you have to be like an expert to set up a raid system (the
only custom solution I've found up to now)

HI ya

My name is Imran I work for a company called Workstations UK LTD. I belive we can help you achieve the 5 GByte/sec transfer rate.
my email address is Imran@workstationsuk.co.uk
or call me on 00 44 1494 724498

Regards
Imran