If you stick enough hard drives together, you can easily surpass 2
terabytes of combined space and if you try to connect them via SCSI,
you will apparently not see all your space.

That's because host bus adapters from Adaptec and LSI Logic can't see
anything larger than 2 terabytes.

I am using a Promise VTrak 15100 which uses SATA drives and a SCSI
controller. The Promise has no partitioning software and so my 3498 GB
RAID appears as 2048 GB to my Adaptec card.

What gives? Is everyone just using fibre channel?

[ Post a follow-up to this message ]

Maurice Volaski wrote:
> If you stick enough hard drives together, you can easily surpass 2
> terabytes of combined space and if you try to connect them via SCSI,
> you will apparently not see all your space.
>
> That's because host bus adapters from Adaptec and LSI Logic can't see
> anything larger than 2 terabytes.

No, it's because the SCSI spec defines a 32bit address for each block and
each block is 512 bytes, end result is a 2TB limit for a single disk, or
stripe handled by the controller.

>
> I am using a Promise VTrak 15100 which uses SATA drives and a SCSI
> controller. The Promise has no partitioning software and so my 3498 GB
> RAID appears as 2048 GB to my Adaptec card.
>
Break the stripe into a pair of 1.7TB RAIDs, assign each one to the host as
a seperate physical disk, then stripe them together at the host level to
create a 3.4TB logical volume.

> What gives? Is everyone just using fibre channel?

Fibre channel is the same, a 32bit address space with 512byte blocks.


--
Nik Simpson

[ Post a follow-up to this message ]

"Nik Simpson" <n_simpson@bellsouth.net> writes:
> Maurice Volaski wrote: 
>
> No, it's because the SCSI spec defines a 32bit address for each block and
> each block is 512 bytes, end result is a 2TB limit for a single disk, or
> stripe handled by the controller.

There's a newer SCSI command format which supports longer block addresses.

I don't know which cards (actually drivers) implement it so far.

-- Anton

[ Post a follow-up to this message ]

Anton Rang wrote:
> "Nik Simpson" <n_simpson@bellsouth.net> writes: 
>
> There's a newer SCSI command format which supports longer block
> addresses.
>
> I don't know which cards (actually drivers) implement it so far.
>
It's probably an OS level issue as well, not just the device driver on the
adapter, i.e. the OS has got to have some way to map a logical block address
on to something that requires a greater than 32bit physical address. Anyway,
regardless its a long way from universal support. The easiest way around the
problem is to use OS level volume managers to stripe across multiple 2TB
LUNs.


--
Nik Simpson

[ Post a follow-up to this message ]

In article <K0Qlc.70$UG2.5@bignews2.bellsouth.net>,
Nik Simpson <n_simpson@bellsouth.net> wrote:
>Maurice Volaski wrote: 
>
>No, it's because the SCSI spec defines a 32bit address for each block and
>each block is 512 bytes, end result is a 2TB limit for a single disk, or
>stripe handled by the controller.

That's false.  Both larger block addresses and larger block sizes are
supported -- both by the specification *and* by many devices.

You should contact your HBA and OS vendors for appropriate fixes; here,
now, in 2004, code that can't cope with disks that big is just plain
broken.

--
Thor Lancelot Simon	                                      tls@rek.tjls.com
But as he knew no bad language, he had called him all the names of common
objects that he could think of, and had screamed: "You lamp!  You towel!  Yo
u
plate!" and so on.              --Sigmund Freud

[ Post a follow-up to this message ]

Thor Lancelot Simon wrote:
> In article <K0Qlc.70$UG2.5@bignews2.bellsouth.net>,
> Nik Simpson <n_simpson@bellsouth.net> wrote: 
>
> That's false.  Both larger block addresses and larger block sizes are
> supported -- both by the specification *and* by many devices.

OK, which deviuces, and which OSes, shouldn't be that hard to come up with a
list if they are so "common"

>
> You should contact your HBA and OS vendors for appropriate fixes;
> here, now, in 2004, code that can't cope with disks that big is just
> plain broken.

It may not be a hard limit, buts it's a pretty common one, how many OS do
you know that don't have a 2TB limit for a single drive/LUN, heck until
relatively recently Solaris had a 1TB limit. While it maybe possible, to
exceed the 2TB limit its certainly not common practice and its not a big
deal for the OS or HBA vendors since customers wanting single disks >2TB are
a very small minority.


--
Nik Simpson

[ Post a follow-up to this message ]

In article <K0Qlc.70$UG2.5@bignews2.bellsouth.net>,
"Nik Simpson" <n_simpson@bellsouth.net> wrote:

> Maurice Volaski wrote: 
>
> No, it's because the SCSI spec defines a 32bit address for each block and
> each block is 512 bytes, end result is a 2TB limit for a single disk, or
> stripe handled by the controller.

I confirmed that this is untrue. The spec has been updated and now the
limit is 8 zettabytes!

I confirmed that Adaptec does not support this spec on any of its
hardware.

However, LSI Logic has contacted me and I got misinformation initially.
Their products do support the updated spec.
 
> Break the stripe into a pair of 1.7TB RAIDs, assign each one to the host a
s
> a seperate physical disk, then stripe them together at the host level to
> create a 3.4TB logical volume.

This is a moot point given that LSI is saying their stuff will work. The
VTrak is the RAID controller, and I have it implementing the RAID 5,
which is and should be its role in this. There is no way to create two
RAID 5 sets with the same amount of space as a single one and no way to
create logical volumes with their embedded software.

> It's probably an OS level issue as well, not just the device driver on the

Apparently not for Linux kernel 2.6.x.

[ Post a follow-up to this message ]

flux <support@fluxsoft.com> writes:
> I confirmed that Adaptec does not support this spec on any of its
> hardware.

Technically, the *hardware* supports it.  The Windows drivers from
Adaptec (and the BIOS) do not.  It would be a simple software update,
though I don't know if Microsoft supports large devices yet.

> However, LSI Logic has contacted me and I got misinformation initially.
> Their products do support the updated spec.

I think their RAID hardware does as well.

-- Anton

[ Post a follow-up to this message ]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Clinging to sanity, Nik Simpson mumbled in his beard:

[2TB block dev size limit]

> OK, which deviuces, and which OSes, shouldn't be that hard to come up with
a
> list if they are so "common"

I've seen that Linux 2.6 added a >2TB block device support kernel
configuration option. As I've not seen such a device up close, I can't say
how well this works, especially, if it works across all hardware drivers.

greets
- -- vbi

- --
You opted-in to receive these exciting offers by having an email
address.  Our spam cannot be considered spam because of this
disclaimer.  This is a one-time mailing.  To be removed from future
one-time mailings, don't receive email.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: get my key from http://fortytwo.ch/gpg/92082481

 iKcEARECAGcFAkCbNQRgGmh0dHA6Ly9mb3J0eXR3
by5jaC9sZWdhbC9ncGcvZW1h
aWwuMjAwMjA4MjI/ dmVyc2lvbj0xLjUmbWQ1c3VtPTVkZmY4NjhkMTE4
NDMyNzYw
NzFiMjVlYjcwMDZkYTNlAAoJEIukMYvlp/fW4iMAnRmtCsXvVqEi49/q7yI9Z4M7
pgvvAJwMaSx+ABH9ihNtslHaXJI+LCTv6w==
=Grhp
-----END PGP SIGNATURE-----

[ Post a follow-up to this message ]

> > list if they are so "common"
>
> I've seen that Linux 2.6 added a >2TB block device support kernel
> configuration option. As I've not seen such a device up close, I can't say
> how well this works, especially, if it works across all hardware drivers.

SGI already has backported those changes to their shipping 2.4 kernels
because SGI customers expect that large volumes (and have been for a bit
of time on IRIX), usually those are XVM logical volumes and not scsi
luns, though.

For a SCSI HBA driver to support >2TB luns t has to support 16byte scsi
commands which only newer HBAs do.  Except for that every driver should
be ready for that large volumes if the underlying hardware is ready for
it.  For block level drivers there have been a few bugs in handling
of large device found (e.g. in the linux 'md' softraid driver lately),
but for SCSI HBA drivers there's very little chance to get it wrong
because they're not exposed to volume at all (except a little geometry
hack for msdos partition tables - but those don't work TB sizes luns
anyway)

[ Post a follow-up to this message ]