Data Storage - storage solution

Author

storage solution

chivalc@yahoo.com

2005-12-07, 7:52 am

Hello,

I am planning to experiment with an online service technology for file
sharing something similar to sendthisfile.com or putfile.com where
there is a huge storage requirement and bandwidth

Any ideas of a cost effective storage solution (in Linux or MS) which
could be implemented from a home-office. Cost is an important factor
since I am just experimenting with the technology right now.

I would not like to go for shared hosting since I need an hands on
experience.

Precisely,

i) What hardware/architecture is needed for storage (about 2+ TB), web
server with say daily 25 Gb bandwidth of download.

ii) Also, what bandwidth of Internet backbone is needed (in Mbps????).
Leased Line?

"HOW MUCH WOULD ALL THIS COST ?"

Thanks and Regards,
Chival

Nik Simpson

2005-12-07, 5:49 pm

>
> i) What hardware/architecture is needed for storage (about 2+ TB), web
> server with say daily 25 Gb bandwidth of download.

Tha'ts ~300Kbit/s, you'd be hard pressed to find any storage that *couldn't*
sustain that without breaking a sweat. So I'd go for SATA drives as big as
you can get plus perhaps some sort of RAID. Cheapest solution might be to
build yourself a PC with onboard SATA RAID.

>
> ii) Also, what bandwidth of Internet backbone is needed (in Mbps????).
> Leased Line?

Again, you aren't talking about much if the load is evenly balanced
throughout the day, 300Kbit's is something a good DSL line can handle and
certainly wouldn't be a problem for a cable modem. Even if the demand is
"bursty" it's still not likely to need more than a cable modem with
500K-1Mbit of upstream bandwidth.

>
> "HOW MUCH WOULD ALL THIS COST ?"
>

2-2.5K tops for this type of application would be my guess.

--
Nik Simpson

chivalc@yahoo.com

2005-12-08, 2:46 am

Many thanks Nik.

I have come accross many web references which recommend a SAN if
storage is in TB's and likely to exceed
How scalable is SATA RAID?

Thanks again.

Nik Simpson wrote:

>
> Tha'ts ~300Kbit/s, you'd be hard pressed to find any storage that *couldn't*
> sustain that without breaking a sweat. So I'd go for SATA drives as big as
> you can get plus perhaps some sort of RAID. Cheapest solution might be to
> build yourself a PC with onboard SATA RAID.
>
>
> Again, you aren't talking about much if the load is evenly balanced
> throughout the day, 300Kbit's is something a good DSL line can handle and
> certainly wouldn't be a problem for a cable modem. Even if the demand is
> "bursty" it's still not likely to need more than a cable modem with
> 500K-1Mbit of upstream bandwidth.
>
>
> 2-2.5K tops for this type of application would be my guess.
>
>
> --
> Nik Simpson

Miroslav Pragl

2005-12-08, 7:47 am

SATA RAID: i'd be afraid of CPU load as SATA RAIDs are usually software
raids (performed by driver, not hardware). Doublecheck the chipset used!

Miroslav
[vbcol=seagreen]

Nik Simpson

2005-12-08, 7:47 am

"Miroslav Pragl" <lists.subscriber@pragl.cz> wrote in message
news:dn92rd$256r$2@ns.felk.cvut.cz...
> SATA RAID: i'd be afraid of CPU load as SATA RAIDs are usually software
> raids (performed by driver, not hardware). Doublecheck the chipset used!
>

1. Any half decent system will have plenty of CPU power to spare. Think
about it, your typical PCI RAID card with an aging 3-500MHz Strong-Arm CPU
or similar vs a 2+GHz Pentium 4 or Athlon, tell me which is going to have
more CPU cycles free to handle RAID overhead?

2. The sort of bandwidth requirements of the original poster make this a
moot point, average throughput pof 300Kbits/s is nothing

--
Nik Simpson

_firstname_@lr_dot_los-gatos_dot_ca.us

2005-12-08, 5:48 pm

In article <1134019993.342816.156230@g43g2000cwa.googlegroups.com>,
<chivalc@yahoo.com> wrote:
>I have come accross many web references which recommend a SAN if
>storage is in TB's and likely to exceed
>How scalable is SATA RAID?

This rule made a lot of sense when enterprise-class disks were 9GB and
18GB. In those days (which are only a few years ago!) a 1TB storage
system required say 70 or 200 disks (depending for example on the
level of redundancy, the lower end is RAID5 4+1 with 18GB disks
requiring 70 disks, the upper end is RAID1 with 9GB disks requiring
224 disks). A system with that many disks should not be built by
direct-attaching disks to a host, as the administration, management,
cabling etc. are all hard problems. With that many disks, regular
maintenance is required, as you expect a disk to fail every few weeks
or months.

Today, 500GB SATA disks are available easily on the open market. This
means that building a 1TB storage system with redundancy requires
between 3 and 4 disks (3 for 2+1 RAID5, 4 for RAID1). This easily
fits inside any decent server case. With SATA, cabling 4 disks is
trivial (even I could do it before breakfast, and I'm not a hardware
hacker). And setting up and administring a RAID controller (whether
software or hardware) for 3 or 4 disks takes less than 1/2 hour, and
would require no more than a cup of coffee to get going. And with
this few disks, failures are going to be rare events (every few
years), so they can be dealt with by keeping one spare disk in the
cabinet, and doing a manual hardware swap when needed.

I think it is VERY VERY important to express the complexity of a
system not in terms of GB/TB/PB (nor in terms of MIPS and FLOPS), but
in terms of the number of "moving parts": How many disk drives, how
many CPUs (or how many independent hsots with operating systems, as
SMPs and multicore-CPUs are quute similar to single-CPU systems), how
many network switches, how many host names and IP addresses, and so
on. So today a 1TB system has 3-4 spindles, which does not require a
SAN.

--
The address in the header is invalid for obvious reasons. Please
reconstruct the address from the information below (look for _).
Ralph Becker-Szendy _firstname_@lr_dot_los-gatos_dot_ca.us

chivalc@yahoo.com

2005-12-09, 7:48 am

Thanks.

Any ideas how this would work out if a remote backup (outside city
limits) is needed.

- Sid

_firstname_@lr_dot_los-gatos_dot_ca.us wrote:

> In article <1134019993.342816.156230@g43g2000cwa.googlegroups.com>,
> <chivalc@yahoo.com> wrote:
>
> This rule made a lot of sense when enterprise-class disks were 9GB and
> 18GB. In those days (which are only a few years ago!) a 1TB storage
> system required say 70 or 200 disks (depending for example on the
> level of redundancy, the lower end is RAID5 4+1 with 18GB disks
> requiring 70 disks, the upper end is RAID1 with 9GB disks requiring
> 224 disks). A system with that many disks should not be built by
> direct-attaching disks to a host, as the administration, management,
> cabling etc. are all hard problems. With that many disks, regular
> maintenance is required, as you expect a disk to fail every few weeks
> or months.
>
> Today, 500GB SATA disks are available easily on the open market. This
> means that building a 1TB storage system with redundancy requires
> between 3 and 4 disks (3 for 2+1 RAID5, 4 for RAID1). This easily
> fits inside any decent server case. With SATA, cabling 4 disks is
> trivial (even I could do it before breakfast, and I'm not a hardware
> hacker). And setting up and administring a RAID controller (whether
> software or hardware) for 3 or 4 disks takes less than 1/2 hour, and
> would require no more than a cup of coffee to get going. And with
> this few disks, failures are going to be rare events (every few
> years), so they can be dealt with by keeping one spare disk in the
> cabinet, and doing a manual hardware swap when needed.
>
> I think it is VERY VERY important to express the complexity of a
> system not in terms of GB/TB/PB (nor in terms of MIPS and FLOPS), but
> in terms of the number of "moving parts": How many disk drives, how
> many CPUs (or how many independent hsots with operating systems, as
> SMPs and multicore-CPUs are quute similar to single-CPU systems), how
> many network switches, how many host names and IP addresses, and so
> on. So today a 1TB system has 3-4 spindles, which does not require a
> SAN.
>
> --
> The address in the header is invalid for obvious reasons. Please
> reconstruct the address from the information below (look for _).
> Ralph Becker-Szendy _firstname_@lr_dot_los-gatos_dot_ca.us

Arne Joris

2005-12-09, 5:50 pm

Now you're gettng into very murky waters...
Do you need instant replication of your data or a nightly backup ?
Hundreds of kilometers or just one ? Does the data need to be
application coherent so you can restart apps from it in case of a
disaster ?

Whereas the storage can be cheap because it has 24 hours to receive 25
GB and disks can easily sustain that rate, the line you'll need to get
all this data out in a backup window < 24 hours will be very expensive.
Say you need a nightly backup while your webserver is closed. You have
25 GB to move in, say 8 hours; that's slightly under one meg per
second. A T1 line is not going to cut it unless you can apply some
extreme compression to your data.

Doing a tape backup and driving the tape to your destination will be
the cheapest method.