Hey 

I've been running Debian on my server for awhile now. I've had a
six-disk software RAID-5 array on it, with /boot on a non-RAID
partition.

Recently, one of my disks failed. I'm currently salvaging the data on
the array, rather than just pop in a new disk and have it rebuild the
array, I'm now preparing to move the server to my desktop which is a
lot more powerful, and I'm getting rid of the last traces of Windows as
well.

Okay, so having used software RAID-5 has been really cool, but I've
been thinking about ways to  make it (relatively) easy to expand a
RAID-5 array with extra disks while the system is still up and running.
What I'm thinking about doing is this:

Partition all my drives into 20 GB partitions. Let the largest drive be
the "base" drive, that is the one drive that will "hold" the parity
data for all other drives (I know it doesn't work like this and that
the parity data is spread across all disks, but it's a conceptually
nice way to think about it). My partition will then look like this:

120 GB hda P01 P02 P03 P04 P05 P06
120 GB hdb P01 P02 P03 P04 P05 P06
120 GB hdc P01 P02 P03 P04 P05 P06
120 GB hdd P01 P02 P03 P04 P05 P06
160 GB hde P01 P02 P03 P04 P05 P06 P07 P08
250 GB hdf P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12
320 GB hdg P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
P16
320 GB hdh P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
P16
320 GB hdi P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
P16
320 GB hdj P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
P16
320 GB hdk P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
P16
320 GB hdl P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
P16
320 GB hdm P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
P16
320 GB hdn P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
P16

I will then create RAID-5 mds "vertically", i.e. I create md0 from all
P01s on all drives that have a P01, giving me a 260 GB md with parity.
Once all mds (md0 - md15) have been created, I have RAID-5 mds ranging
from 260 GB to 140 GB, all with parity data, and all mds will survive
on disk failing. On top of this, I want to create a logical volume,
combining all mds into one 3225 GB large LV.

With this approach, if I want to add new disks, I can partition them
similarly, then first remove md0 from the LV, release the array here,
create a new array including the new disks, add it back to the LV,
remove md1 from the LV, do the same, etc., untill the new disks are
completely consumed by the LV. AFAICT, this would be possible while the
system is up and running, even if the LV held the root fs and the
system was located here.

So, (finally) my question is this... Would it be possible to create
this kind of setup, with md0-md15 making up a big LV, and install a
Debian-based distro (I'm thinking Kubuntu since this would now also be
my working computer, and I like Kubuntu a lot) onto this RAID-5 secured
LV? I read about the possibility of installing directly onto a RAID-5
md, but it was old litterature, and LVs weren't even mentioned.

Thanks in advance for any insight 

Daniel

[ Post a follow-up to this message ]

Sounds like a pretty good basic game plan.  I might make some
additional suggestions:
o Use more filesystems, rationale:
better security and/or performance, e.g. can separately set
ro,nosuid,nodev,noatime option(s) on specific filesystems as
appropriate, better ease of management (e.g. several filesystem
types can be grown while mounted, but none of the more common freely
available types can be shrunk while mounted, and of the non-free
type than can be shrunk while mounted, the shrinking isn't always
possible, even if the filesystem has the free space.)
o "protect" (RAID-1?) /boot - I haven't looked into the details of
this, but it's probably best, if feasible, to have redundancy (e.g.
RAID-1), for /boot, if that can be implemented, or some other means
to boot a separate copy of /boot if the primary /boot partition
fails.
o make sure you well have in place whatever's necessary to reliably
mount / (root filesystem), e.g. LVM and RAID stuff in kernel or
modules from initrd from /boot, etc.
o may want to have more than one partition size for most of the
storage.  Keep in mind that as drives are added/replaced, the
capacity of the newly added drives may be different (typically
larger).  There are also limits as to how many partitions can exist
on a drive ... partition schemes is however an area which isn't
entirely static, so watch for developments (and added kernel
support) in this area.
o keep in mind that changing partitioning on a disk that has any
partitions in use (open), can, at least from what I've seen thus
far, be a relatively hairy operation.

Daniel Buus wrote:
> Okay, so having used software RAID-5 has been really cool, but I've
> been thinking about ways to  make it (relatively) easy to expand a
> RAID-5 array with extra disks while the system is still up and running.
> What I'm thinking about doing is this:
>
> Partition all my drives into 20 GB partitions. Let the largest drive be
> the "base" drive, that is the one drive that will "hold" the parity
> data for all other drives (I know it doesn't work like this and that
> the parity data is spread across all disks, but it's a conceptually
> nice way to think about it). My partition will then look like this:
>
> 120 GB hda P01 P02 P03 P04 P05 P06
> 120 GB hdb P01 P02 P03 P04 P05 P06
> 120 GB hdc P01 P02 P03 P04 P05 P06
> 120 GB hdd P01 P02 P03 P04 P05 P06
> 160 GB hde P01 P02 P03 P04 P05 P06 P07 P08
> 250 GB hdf P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12
> 320 GB hdg P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
> P16
> 320 GB hdh P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
> P16
> 320 GB hdi P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
> P16
> 320 GB hdj P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
> P16
> 320 GB hdk P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
> P16
> 320 GB hdl P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
> P16
> 320 GB hdm P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
> P16
> 320 GB hdn P01 P02 P03 P04 P05 P06 P07 P08 P09 P10 P11 P12 P13 P14 P15
> P16
>
> I will then create RAID-5 mds "vertically", i.e. I create md0 from all
> P01s on all drives that have a P01, giving me a 260 GB md with parity.
> Once all mds (md0 - md15) have been created, I have RAID-5 mds ranging
> from 260 GB to 140 GB, all with parity data, and all mds will survive
> on disk failing. On top of this, I want to create a logical volume,
> combining all mds into one 3225 GB large LV.
>
> With this approach, if I want to add new disks, I can partition them
> similarly, then first remove md0 from the LV, release the array here,
> create a new array including the new disks, add it back to the LV,
> remove md1 from the LV, do the same, etc., untill the new disks are
> completely consumed by the LV. AFAICT, this would be possible while the
> system is up and running, even if the LV held the root fs and the
> system was located here.
>
> So, (finally) my question is this... Would it be possible to create
> this kind of setup, with md0-md15 making up a big LV, and install a
> Debian-based distro (I'm thinking Kubuntu since this would now also be
> my working computer, and I like Kubuntu a lot) onto this RAID-5 secured
> LV? I read about the possibility of installing directly onto a RAID-5
> md, but it was old litterature, and LVs weren't even mentioned.

[ Post a follow-up to this message ]

Hi Michael! Thanks for the tips 

Michael Paoli wrote:
> Sounds like a pretty good basic game plan.  I might make some
> additional suggestions:
> o Use more filesystems, rationale:
>   better security and/or performance, e.g. can separately set
>   ro,nosuid,nodev,noatime option(s) on specific filesystems as
>   appropriate, better ease of management (e.g. several filesystem
>   types can be grown while mounted, but none of the more common freely
>   available types can be shrunk while mounted, and of the non-free
>   type than can be shrunk while mounted, the shrinking isn't always
>   possible, even if the filesystem has the free space.)

Okay, just to be sure about this... I was thinking that it would be
possible for me to just have one large LV and format it with reiser as
/. I know this isn't the typical Linux way, but I like the simplicity
;) Wouldn't it be possible to shrink/expand this LV (i.e. "/" as
reiser) while mounted? I have a backup plan in case that's the case...
But would like it very much if it were not 

> o "protect" (RAID-1?) /boot - I haven't looked into the details of
>   this, but it's probably best, if feasible, to have redundancy (e.g.
>   RAID-1), for /boot, if that can be implemented, or some other means
>   to boot a separate copy of /boot if the primary /boot partition
>   fails.

Actually, my original plan was to have everything  on top of the LV,
which was why I came here with this long question, but testing setups
in VMware showed me that this was indeed not possible. My new layout
now includes a RAID-1 MD for boot - check it out at
http://www.rhesusb.dk/wp-content/uploads/raid-setup.gif (the dark
yellow 160 GB PATA drive is not in the array since it's a 5400 RPM
drive, and I'd like a spare anyway for my Firewire case.

> o make sure you well have in place whatever's necessary to reliably
>   mount / (root filesystem), e.g. LVM and RAID stuff in kernel or
>   modules from initrd from /boot, etc.

AFAICT from my testing with Kubuntu under VMware, I don't have to do
anything (?) manual for it to work - please correct me if I'm wrong, so
I don't get nasty surprises :D. I'm also thinking it's sufficient to dd
the MBR from hda (RAID-1 drive 0) to hdc (RAID-1 drive 1) in order to
be able to boot in case either one fails, is that correct?

> o may want to have more than one partition size for most of the
>   storage.  Keep in mind that as drives are added/replaced, the
>   capacity of the newly added drives may be different (typically
>   larger).  There are also limits as to how many partitions can exist
>   on a drive ... partition schemes is however an area which isn't
>   entirely static, so watch for developments (and added kernel
>   support) in this area.

I checked up on it, and it seems the magic max number of partitions
(now) is 64. The way I do it now, my large drives (320 GB) contain
logical partitions from hd*5 to hd*20. I figure the current
partitioning scheme should work for drives at least up to 1 TB, which
will probably be fine... for now ;)

> o keep in mind that changing partitioning on a disk that has any
>   partitions in use (open), can, at least from what I've seen thus
>   far, be a relatively hairy operation.
>

Yeah, I'm thinking that, too... What with the renumbering and all. I've
just finished moving data from my old server onto spare drives, and now
I'm ready to reboot into Knoppix to partition the remaining drives. I
have 6 320 GB underway by mail... This is gonna be so cool! :D

Cheers,
Daniel

[ Post a follow-up to this message ]