Recently ran into the account of a guy who built his own 1.2TB
RAID50-based storage array for $1600 (see
<URL:http://www.finnie.org/terabyte/> ). I really like the idea and
have been thinking about following suit.

Like Finnie, I want to be able to store huge amounts of DivX/Xvid
files online. In addition to the storage array, I also plan to build a
separate MythTV (<URL:http://www.mythtv.org> ) box, which among other
things will let me play them at will. My 200GB Series 1 TiVo's been
serving me well for more than four years, but I really like the idea
of being able to seamlessly integrate my AVI collection with TV
recordings, and from what I gather MythTV has finally matured enough
to be a realistic TiVo alternative.

I have been 100% Linux at home for almost a decade and am quite
comfortable with most of the technical aspects of the project.

I'm planning on making the following changes to Finnie's build
configuration:

* Instead of 200GB ATA, use 250GB SATA drives for a total of
1.5TB. Outpost.com offers a Western Digital 250GB SATA drive for
$170 (<URL:http://shop1.outpost.com/product/3868597> ). I just missed
the chance to get a $30 rebate off each drive, but I'm sure
Fatwallet will alert me to a similar opportunity sooner or later.
* Accordingly, get a HighPoint SATA RAID card instead of the specified
RocketRAID 454 ATA RAID card. I think the RocketRAID 1640
(<URL:http://www.newegg.com/app/SearchPro...ubmit=Go&DEPA=0> )
is the way to go.
* Instead of ext3, use XFS as the file system.

My questions:
* If I connect the storage array to my Linksys WRT54G router, will
100Mbps Ethernet be fast enough to pump the AVI files to the MythTV
box without dropping frames?
* Conversely, will 100Mbps Ethernet be sufficient to let me use the
storage array as the primary storage medium for MythTV's recordings?
What about HDTV encodings (using the pcHDTV Linux-only card)? Or do
I have to upgrade to a Gigabit Ethernet router? Or would the encoder
card and MythTV software have to run on the storage array itself in
order to achieve acceptable performance?  (Actually, I'm not opposed
to doing so, if one box can simultaneously handle both storage and
MythTV tasks.)
* Anything else that I'm missing or should keep in mind?

--
Read my Deep Thoughts @ <URL:http://www.ylee.org/blog/>     PERTH ----> *
Cpu(s):  8.9% us,  5.3% sy, 85.4% ni,  0.0% id,  0.0% wa,  0.3% hi,  0.0% si
Mem:    516960k total,   514056k used,     2904k free,     9212k buffers
Swap:  2101032k total,   989384k used,  1111648k free,   135456k cached

[ Post a follow-up to this message ]

Yeechang Lee wrote:
> Recently ran into the account of a guy who built his own 1.2TB
> RAID50-based storage array for $1600 (see
> <URL:http://www.finnie.org/terabyte/> ). I really like the idea and
> have been thinking about following suit.
>
> Like Finnie, I want to be able to store huge amounts of DivX/Xvid
> files online. In addition to the storage array, I also plan to build a
> separate MythTV (<URL:http://www.mythtv.org> ) box, which among other
> things will let me play them at will. My 200GB Series 1 TiVo's been
> serving me well for more than four years, but I really like the idea
> of being able to seamlessly integrate my AVI collection with TV
> recordings, and from what I gather MythTV has finally matured enough
> to be a realistic TiVo alternative.
>
> I have been 100% Linux at home for almost a decade and am quite
> comfortable with most of the technical aspects of the project.
>
> I'm planning on making the following changes to Finnie's build
> configuration:
>
> * Instead of 200GB ATA, use 250GB SATA drives for a total of
>   1.5TB. Outpost.com offers a Western Digital 250GB SATA drive for
>   $170 (<URL:http://shop1.outpost.com/product/3868597> ). I just missed
>   the chance to get a $30 rebate off each drive, but I'm sure
>   Fatwallet will alert me to a similar opportunity sooner or later.
> * Accordingly, get a HighPoint SATA RAID card instead of the specified
>   RocketRAID 454 ATA RAID card. I think the RocketRAID 1640
>   (<URL:http://www.newegg.com/app/SearchPro...ubmit=Go&DEPA=0
> )
>   is the way to go.
> * Instead of ext3, use XFS as the file system.
>
> My questions:
> * If I connect the storage array to my Linksys WRT54G router, will
>   100Mbps Ethernet be fast enough to pump the AVI files to the MythTV
>   box without dropping frames?
> * Conversely, will 100Mbps Ethernet be sufficient to let me use the
>   storage array as the primary storage medium for MythTV's recordings?
>   What about HDTV encodings (using the pcHDTV Linux-only card)? Or do
>   I have to upgrade to a Gigabit Ethernet router? Or would the encoder
>   card and MythTV software have to run on the storage array itself in
>   order to achieve acceptable performance?  (Actually, I'm not opposed
>   to doing so, if one box can simultaneously handle both storage and
>   MythTV tasks.)
> * Anything else that I'm missing or should keep in mind?
>

I don't know much about MythTV, but I did some testing with Gigabit
Ethernet.  The results are here:

http://somacon.com/docs/gigabitnas.html

Highlights:------------------------
TEST: Repeated disk-to-disk 150 MB file transfer via FTP.
Connected via Gigabit Switch to Athlon XP 2500+.

P3 @ 500 Mhz, tg3 driver = 27 MB/sec

P3 @ 850 Mhz, tg3 driver = 33 MB/sec

P3 @ 850 Mhz, smc95x2 driver = 36 MB/sec

P3 @ 1000 Mhz, smc95x2 driver = 36 MB/sec

---------------------------------------

With Gigabit Ethernet, maximum transfer speed is affected by the CPU
speed.  The CPU burden is likely introduced by the TCP stack and
network driver.  By comparison, current hardware will easily top out a
100Mbit network, with a rate of 11 MB/sec.

[ Post a follow-up to this message ]

Yeechang Lee wrote:

> Recently ran into the account of a guy who built his own 1.2TB
> RAID50-based storage array for $1600 (see
> <URL:http://www.finnie.org/terabyte/> ). I really like the idea and
> have been thinking about following suit.
>
> Like Finnie, I want to be able to store huge amounts of DivX/Xvid
> files online. In addition to the storage array, I also plan to build a
> separate MythTV (<URL:http://www.mythtv.org> ) box, which among other
> things will let me play them at will. My 200GB Series 1 TiVo's been
> serving me well for more than four years, but I really like the idea
> of being able to seamlessly integrate my AVI collection with TV
> recordings, and from what I gather MythTV has finally matured enough
> to be a realistic TiVo alternative.
>
> I have been 100% Linux at home for almost a decade and am quite
> comfortable with most of the technical aspects of the project.
>
> I'm planning on making the following changes to Finnie's build
> configuration:
>
> * Instead of 200GB ATA, use 250GB SATA drives for a total of
>   1.5TB. Outpost.com offers a Western Digital 250GB SATA drive for
>   $170 (<URL:http://shop1.outpost.com/product/3868597> ). I just missed
>   the chance to get a $30 rebate off each drive, but I'm sure
>   Fatwallet will alert me to a similar opportunity sooner or later.

You might want to check the CompUSA site--they were having a deal for 200
gig drives for about 90 bucks counting a mail-in rebate, but I don't know
if it's still on.

> * Accordingly, get a HighPoint SATA RAID card instead of the specified
>   RocketRAID 454 ATA RAID card. I think the RocketRAID 1640

Personally for a RAID that size I'd go for a 3Ware or LSI Logic.  No
particular reason, just that I'm used to terabytes being mainframe
territory and I get nervous with consumer RAID controllers trying to handle
that much data.

> (<URL:http://www.newegg.com/app/SearchPro...ubmit=Go&DEPA=0> )
>   is the way to go.
> * Instead of ext3, use XFS as the file system.
>
> My questions:
> * If I connect the storage array to my Linksys WRT54G router, will
>   100Mbps Ethernet be fast enough to pump the AVI files to the MythTV
>   box without dropping frames?

Yes.

> * Conversely, will 100Mbps Ethernet be sufficient to let me use the
>   storage array as the primary storage medium for MythTV's recordings?

Yes.

>   What about HDTV encodings (using the pcHDTV Linux-only card)?

Yes.

>   Or do
>   I have to upgrade to a Gigabit Ethernet router?

No.

>   Or would the encoder
>   card and MythTV software have to run on the storage array itself in
>   order to achieve acceptable performance?  (Actually, I'm not opposed
>   to doing so, if one box can simultaneously handle both storage and
>   MythTV tasks.)
> * Anything else that I'm missing or should keep in mind?

HDTV is carried on two adjacent 6 MHz channels, for an aggregate of 12
analog MHz.  I don't recall the actual bit rate but it's not very high.
There's no reason that any other kind of TV would need more than that
unless you're storing files in an uncompressed or lossless-compressed form.

--
--John
Reply to jclarke at ae tee tee global dot net
(was jclarke at eye bee em dot net)

[ Post a follow-up to this message ]

Yeechang Lee wrote:


> * Instead of 200GB ATA, use 250GB SATA drives for a total of
>   1.5TB. Outpost.com offers a Western Digital 250GB SATA drive for
>   $170 (<URL:http://shop1.outpost.com/product/3868597> ). I just missed
>   the chance to get a $30 rebate off each drive, but I'm sure
>   Fatwallet will alert me to a similar opportunity sooner or later.

Generally rebates are limited to one per SKU per household.  Be sure to
check the details.


> * Anything else that I'm missing or should keep in mind?

I'm doing a cheap-o scaled down version of what you describe.  As soon
as my drives come in I'll let you know.   I have a feeling that the
latency will be increased when seeking through a recording, but other
than that might be OK.


-WD

[ Post a follow-up to this message ]

Will Dormann wrote:
> I have a feeling that the latency will be increased when seeking
> through a recording, but other than that might be OK.
>

That's something I hadn't even considered. On my TiVo seeks are
lightining-fast; I can't believe any network-based system will be
quite so quick. (On the other hand, even over a wireless link and
Samba I've found seeks using mplayer to be more or less acceptable, so
a storage array on much newer equipment might not be so bad at all.) I
wonder if MythTV has the ability to locally cache recordings?

--
Read my Deep Thoughts @ <URL:http://www.ylee.org/blog/>     PERTH ----> *
Cpu(s):  8.9% us,  5.3% sy, 85.5% ni,  0.0% id,  0.0% wa,  0.3% hi,  0.0% si
Mem:    516960k total,   513536k used,     3424k free,    12744k buffers
Swap:  2101032k total,   989264k used,  1111768k free,   118932k cached

[ Post a follow-up to this message ]

Yeechang Lee wrote:

> I
> wonder if MythTV has the ability to locally cache recordings?

Not that I'm aware of.   I haven't looked into it too much, but I think
it would be nice to be able to record to and watch from the local disk
but after a certain amount of time the recordings would get offloaded to
the NAS for storage.

I know there is a built-in option to transcode files after an amount of
time, but I wonder if it would be possible to have that same function
just do a simple transfer without re-encoding.

I'm sure once I get my NAS set up, I'll be looking deeper into it!


-WD

[ Post a follow-up to this message ]

On 4 Aug 2004 17:48:15 GMT, Yeechang Lee <ylee@pobox.com> wrote:

>* If I connect the storage array to my Linksys WRT54G router, will
>  100Mbps Ethernet be fast enough to pump the AVI files to the MythTV
>  box without dropping frames?
>* Conversely, will 100Mbps Ethernet be sufficient to let me use the
>  storage array as the primary storage medium for MythTV's recordings?
>  What about HDTV encodings (using the pcHDTV Linux-only card)? Or do
>  I have to upgrade to a Gigabit Ethernet router? Or would the encoder
>  card and MythTV software have to run on the storage array itself in
>  order to achieve acceptable performance?  (Actually, I'm not opposed
>  to doing so, if one box can simultaneously handle both storage and
>  MythTV tasks.)

Seems to me your architecture has just one primary client system (the
MythTV box).  So why not stuff an extra gigabit NIC in the MythTV box
and the storage box and connect them with a cross-over cable?  A quick
peak suggests that gigabit NICs will cost from about $30...

(Keep the regular network for everything else, of course).

When/if you want to expand the gigabit network, just add a switch...

Malc.

[ Post a follow-up to this message ]

"Shailesh Humbad" <noreply@nowhere.com> wrote in message
news:OR9Qc.115875$fv.4318@fe2.columbus.rr.com...
> Yeechang Lee wrote:
>
> I don't know much about MythTV, but I did some testing with Gigabit
> Ethernet.  The results are here:
>
> http://somacon.com/docs/gigabitnas.html
>
> Highlights:------------------------
> TEST: Repeated disk-to-disk 150 MB file transfer via FTP.
> Connected via Gigabit Switch to Athlon XP 2500+.
>
> P3 @ 500 Mhz, tg3 driver = 27 MB/sec
>
> P3 @ 850 Mhz, tg3 driver = 33 MB/sec
>
> P3 @ 850 Mhz, smc95x2 driver = 36 MB/sec
>
> P3 @ 1000 Mhz, smc95x2 driver = 36 MB/sec
>
> ---------------------------------------
>
> With Gigabit Ethernet, maximum transfer speed is affected by the CPU
> speed.  The CPU burden is likely introduced by the TCP stack and
> network driver.  By comparison, current hardware will easily top out a
> 100Mbit network, with a rate of 11 MB/sec.

Depending on your workload, it is affected by so many things: CPU,
memory, backplane, drivers, even the file system and how NFS (in
our tests) is configured.

Using dual 2.4 Xeons to drive 2 3ware cards and two disk arrays
of 8 in a 6-1-1 raid 5 config and raid 1 the two arrays (~2.8TB)
with the 2.6.7 linux kernel using XFS I am up to ~55MB/sec in
our prelim testing.  We got a little better read speed with JFS but
XFS wrote faster.

We have done thinks from changing NFS parameters to
tweaking the intel ethernet drivers:
http://support.intel.com/support/ne...b/CS-009209.htm
to changing memory buffers.  I am doing some research now
on tweaking flush buffers.

You can get rather obsesses with this. 

--
Angel R. Rivera aka Wolf
----------------------------------------------------------------
Please post all reponses to UseNet.  All email cheerfully and automagically
routed to Dave Null

[ Post a follow-up to this message ]

Malcolm Weir wrote:

> On 4 Aug 2004 17:48:15 GMT, Yeechang Lee <ylee@pobox.com> wrote:
> 
>
> Seems to me your architecture has just one primary client system (the
> MythTV box).  So why not stuff an extra gigabit NIC in the MythTV box
> and the storage box and connect them with a cross-over cable?  A quick
> peak suggests that gigabit NICs will cost from about $30...

Why bother?

> (Keep the regular network for everything else, of course).
>
> When/if you want to expand the gigabit network, just add a switch...
>
> Malc.

--
--John
Reply to jclarke at ae tee tee global dot net
(was jclarke at eye bee em dot net)

[ Post a follow-up to this message ]

Shailesh Humbad <noreply@nowhere.com> writes:
 
>
>I don't know much about MythTV, but I did some testing with Gigabit
>Ethernet.  The results are here:
>
>http://somacon.com/docs/gigabitnas.html
>

>
>With Gigabit Ethernet, maximum transfer speed is affected by the CPU

Not exactly.   Consider that maximum full-duplex throughput of a gigabit eth
ernet
card is 200 mega_bytes_ per second.   Put this on a standard 33Mhz PCI
bus (132 megabytes/second peak), and you can see that the speed of the
CPU doesn't matter from the standpoint of getting maximum throughput,
especially considering you may be using some of that available PCI
bandwidth for disk I/O.

FTP is hardly a useful benchmark for determining the peak ethernet
performance anyway.   Try with something lower overhead like rcp.

>speed.  The CPU burden is likely introduced by the TCP stack and
>network driver.  By comparison, current hardware will easily top out a
>100Mbit network, with a rate of 11 MB/sec.

[ Post a follow-up to this message ]