Data Storage - CX600 / Navisphere to AIX / Naviagent connectivity problem?

Author

CX600 / Navisphere to AIX / Naviagent connectivity problem?

Sterve

2006-12-17, 1:17 am

I'm fairly new to storage and coming up to speed the hard way. I hope
I have a configuration
issue with the following:

CX600 (FLARE 2.16.600.5.012) and a new server P550 / AIX 5.3 ML 4 with
Naviagent 6.7.0.4

What I was hoping for was to see the HBA's from the P550 automatically
appear in the Navisphere "Connectivity Status" window after starting
the Naviagent on the AIX host. I'm in day 9, and it ain't happenin...
I *think* I've read everything on the PowerLink site and read every
posted thread that looks like my problem. So far it's been all sloppy,
and no joe.

Here's what I have on the AIX host:

# lslpp -L | grep -i NAV
NAVIAGENT 6.7.0.4 C F Navisphere Disk
Array
NAVICLI 6.7.0.4 C F Navisphere Disk
Array
#
# lslpp -L | grep -i EMC
EMC.CLARiiON.aix.rte 5.3.0.0 C F EMC CLARiiON AIX
Support
EMC.CLARiiON.fcp.rte 5.3.0.0 C F EMC CLARiiON FCP
Support
EMC.CLARiiON.iscsi.rte 5.3.0.0 C F EMC CLARiiON iSCSI
Support
EMCpower.base 5.0.0.0 C F PowerPath Base
Driver and
EMCpower.consistency_grp 5.0.0.0 C F PowerPath
Consistency Group
EMCpower.migration_enabler
EMCpower.mpx 5.0.0.0 C F PowerPath
Multi_Pathing
EMCpower.multi_path_clariion
devices.common.IBM.modemcfg.data
#
# emcpreg -list /* <------ License check

Key xxxx-xxxx-xxxx-xxxx-xxxx-xxxx
Product: PowerPath
Capabilities: All
#

# ps -ef | grep -i nav
/* Agent is running
root 344306 401622 0 15:31:27 pts/0 0:00 grep -i nav
root 356484 1 0 14:40:03 - 0:00 /usr/bin/naviagent -f
/etc/Navisphere/agent.config
#

# lspv /* Nothing but the boot disks so far (no love from
`powermt config`, cfgmgr)
hdisk0 000c44d06163706f rootvg
active
hdisk1 000c44d05e2eedcb rootvg
active
#

# pwd
/dev
# ls -al pow*
crw-rw---- 1 root system 48, 0 Dec 15 14:55 powerpath0
#

# ls -al hdiskpower*
ls: 0653-341 The file hdiskpower* does not exist.
#

# pwd
/etc/log
# tail agent.log

12/15/2006 14:40:02 @(#)Navisphere Agent Revision 6.7.0.4.7 for AIX on
05/27/2004 00:47:36
12/15/2006 14:40:02 Copyright (c) 2004 EMC Corporation. All Rights
Reserved
12/15/2006 14:40:02 Log File
12/15/2006 14:40:03 No entries in the device list. This is normal if
the host is attached
only to FC4700 arrays.
12/15/2006 14:40:03 Agent has started up.
#

One of the DS24-M2 switches: /* The other has the other HBA and SP
Port's

Show> zoning
Default Zone Enabled: False
Zone Set: NEW_ZONE_SET
........
........ /* Zones of Storage Groups that do work until the ones I added
below
........
Zone: hci_hba1_spa3
Zone Member: 10:00:00:00:C9:59:38:CB /* Host HBA
Zone Member: 50:06:01:63:10:60:0A:66 /*SP A Port 3
Zone: hci_hba1_spb2
Zone Member: 50:06:01:6A:10:60:0A:66 /*SP B Port 2
Zone Member: 10:00:00:00:C9:59:38:CB /* Host HBA

Show.Fabric> nodes
Domain ID Node WWN
--------- -----------------------
1 50:06:01:60:90:60:0A:66
1 50:06:01:60:90:60:0A:66
1 20:00:00:00:C9:33:1C:F6
1 20:00:00:00:C9:33:16:DF
1 20:00:00:00:C9:59:38:CB <-- This is my new host
1 20:00:00:00:C9:32:09:6C
1 20:00:00:00:C9:40:47:F8
1 20:00:00:00:C9:40:45:F8

/* I am on Port 8 on both switches. The incidents below are me
rebooting during the ML 3 to
ML 4 advised by EMC for 5.3 & CX600

Show> linkIncidentLog
Date / Time Port Link Incident Event
---------------- ----
------------------------------------------------------
12/15/06 12:02P 8 Loss-of-Signal or Loss-of-Synchronization. <-
Patching reboots today
12/15/06 12:00P 8 Loss-of-Signal or Loss-of-Synchronization.
12/15/06 10:00A 8 Loss-of-Signal or Loss-of-Synchronization.
12/15/06 09:58A 8 Loss-of-Signal or Loss-of-Synchronization.

Show.Port> status
Port State Type Attached WWN Beaconing
Reason
---- ------------------ ----- ----------------------- ---------
------
0 Online fPort 50:06:01:68:10:60:0A:66 Off
1 Online fPort 10:00:00:00:C9:40:45:F8 Off
2 No Light gxPort 00:00:00:00:00:00:00:00 Off
3 Online fPort 10:00:00:00:C9:33:16:DF Off
4 Online fPort 50:06:01:61:10:60:0A:66 Off
5 Online fPort 10:00:00:00:C9:40:47:F8 Off
6 No Light gxPort 00:00:00:00:00:00:00:00 Off
7 Online fPort 10:00:00:00:C9:33:1C:F6 Off
8 Online fPort 10:00:00:00:C9:59:38:CB Off <-- My
new server again
9 Online fPort 10:00:00:00:C9:32:09:6C Off
10 No Light gxPort 00:00:00:00:00:00:00:00 Off

CX600:
FLARE-Operating-Environment: 02.16.600.5.012

AIX 5300-04:
NAVIAGENT 6.7.0.4 C F Navisphere Disk
Array
NAVICLI 6.7.0.4 C F Navisphere Disk
Array

I've registered the HBA's manually with Navisphere to no avail.
Navisphere 6 -> Tools -> Connect Host / by IP advises that my host is
not managed or connected to a CLARiiON storage system.

I believe I've RTFM'd all the FM's I can find and hope someone will
point out the really obvious thing I'm overlooking.

Thanks much in advance all.

noname@gmail.com

2006-12-17, 7:13 am

I don't have navi in front of me, but there is an update option
somewhere.
Do you see other hosts on the CX? And how many hosts are there?
(not sure if you are running into it, but CXs have a limit on the
# of initiators they can support)

And finally do you see the host HBAs from the CLI (-port -list option)?

Sterve wrote:
> I'm fairly new to storage and coming up to speed the hard way. I hope
> I have a configuration
> issue with the following:
>
> CX600 (FLARE 2.16.600.5.012) and a new server P550 / AIX 5.3 ML 4 with
> Naviagent 6.7.0.4
>
> What I was hoping for was to see the HBA's from the P550 automatically
> appear in the Navisphere "Connectivity Status" window after starting
> the Naviagent on the AIX host. I'm in day 9, and it ain't happenin...
> I *think* I've read everything on the PowerLink site and read every
> posted thread that looks like my problem. So far it's been all sloppy,
> and no joe.
>
> Here's what I have on the AIX host:
>
> # lslpp -L | grep -i NAV
> NAVIAGENT 6.7.0.4 C F Navisphere Disk
> Array
> NAVICLI 6.7.0.4 C F Navisphere Disk
> Array
> #
> # lslpp -L | grep -i EMC
> EMC.CLARiiON.aix.rte 5.3.0.0 C F EMC CLARiiON AIX
> Support
> EMC.CLARiiON.fcp.rte 5.3.0.0 C F EMC CLARiiON FCP
> Support
> EMC.CLARiiON.iscsi.rte 5.3.0.0 C F EMC CLARiiON iSCSI
> Support
> EMCpower.base 5.0.0.0 C F PowerPath Base
> Driver and
> EMCpower.consistency_grp 5.0.0.0 C F PowerPath
> Consistency Group
> EMCpower.migration_enabler
> EMCpower.mpx 5.0.0.0 C F PowerPath
> Multi_Pathing
> EMCpower.multi_path_clariion
> devices.common.IBM.modemcfg.data
> #
> # emcpreg -list /* <------ License check
>
> Key xxxx-xxxx-xxxx-xxxx-xxxx-xxxx
> Product: PowerPath
> Capabilities: All
> #
>
> # ps -ef | grep -i nav
> /* Agent is running
> root 344306 401622 0 15:31:27 pts/0 0:00 grep -i nav
> root 356484 1 0 14:40:03 - 0:00 /usr/bin/naviagent -f
> /etc/Navisphere/agent.config
> #
>
> # lspv /* Nothing but the boot disks so far (no love from
> `powermt config`, cfgmgr)
> hdisk0 000c44d06163706f rootvg
> active
> hdisk1 000c44d05e2eedcb rootvg
> active
> #
>
> # pwd
> /dev
> # ls -al pow*
> crw-rw---- 1 root system 48, 0 Dec 15 14:55 powerpath0
> #
>
> # ls -al hdiskpower*
> ls: 0653-341 The file hdiskpower* does not exist.
> #
>
> # pwd
> /etc/log
> # tail agent.log
>
> 12/15/2006 14:40:02 @(#)Navisphere Agent Revision 6.7.0.4.7 for AIX on
> 05/27/2004 00:47:36
> 12/15/2006 14:40:02 Copyright (c) 2004 EMC Corporation. All Rights
> Reserved
> 12/15/2006 14:40:02 Log File
> 12/15/2006 14:40:03 No entries in the device list. This is normal if
> the host is attached
> only to FC4700 arrays.
> 12/15/2006 14:40:03 Agent has started up.
> #
>
>
> One of the DS24-M2 switches: /* The other has the other HBA and SP
> Port's
>
> Show> zoning
> Default Zone Enabled: False
> Zone Set: NEW_ZONE_SET
> .......
> ....... /* Zones of Storage Groups that do work until the ones I added
> below
> .......
> Zone: hci_hba1_spa3
> Zone Member: 10:00:00:00:C9:59:38:CB /* Host HBA
> Zone Member: 50:06:01:63:10:60:0A:66 /*SP A Port 3
> Zone: hci_hba1_spb2
> Zone Member: 50:06:01:6A:10:60:0A:66 /*SP B Port 2
> Zone Member: 10:00:00:00:C9:59:38:CB /* Host HBA
>
>
> Show.Fabric> nodes
> Domain ID Node WWN
> --------- -----------------------
> 1 50:06:01:60:90:60:0A:66
> 1 50:06:01:60:90:60:0A:66
> 1 20:00:00:00:C9:33:1C:F6
> 1 20:00:00:00:C9:33:16:DF
> 1 20:00:00:00:C9:59:38:CB <-- This is my new host
> 1 20:00:00:00:C9:32:09:6C
> 1 20:00:00:00:C9:40:47:F8
> 1 20:00:00:00:C9:40:45:F8
>
> /* I am on Port 8 on both switches. The incidents below are me
> rebooting during the ML 3 to
> ML 4 advised by EMC for 5.3 & CX600
>
> Show> linkIncidentLog
> Date / Time Port Link Incident Event
> ---------------- ----
> ------------------------------------------------------
> 12/15/06 12:02P 8 Loss-of-Signal or Loss-of-Synchronization. <-
> Patching reboots today
> 12/15/06 12:00P 8 Loss-of-Signal or Loss-of-Synchronization.
> 12/15/06 10:00A 8 Loss-of-Signal or Loss-of-Synchronization.
> 12/15/06 09:58A 8 Loss-of-Signal or Loss-of-Synchronization.
>
> Show.Port> status
> Port State Type Attached WWN Beaconing
> Reason
> ---- ------------------ ----- ----------------------- ---------
> ------
> 0 Online fPort 50:06:01:68:10:60:0A:66 Off
> 1 Online fPort 10:00:00:00:C9:40:45:F8 Off
> 2 No Light gxPort 00:00:00:00:00:00:00:00 Off
> 3 Online fPort 10:00:00:00:C9:33:16:DF Off
> 4 Online fPort 50:06:01:61:10:60:0A:66 Off
> 5 Online fPort 10:00:00:00:C9:40:47:F8 Off
> 6 No Light gxPort 00:00:00:00:00:00:00:00 Off
> 7 Online fPort 10:00:00:00:C9:33:1C:F6 Off
> 8 Online fPort 10:00:00:00:C9:59:38:CB Off <-- My
> new server again
> 9 Online fPort 10:00:00:00:C9:32:09:6C Off
> 10 No Light gxPort 00:00:00:00:00:00:00:00 Off
>
>
> CX600:
> FLARE-Operating-Environment: 02.16.600.5.012
>
> AIX 5300-04:
> NAVIAGENT 6.7.0.4 C F Navisphere Disk
> Array
> NAVICLI 6.7.0.4 C F Navisphere Disk
> Array
>
> I've registered the HBA's manually with Navisphere to no avail.
> Navisphere 6 -> Tools -> Connect Host / by IP advises that my host is
> not managed or connected to a CLARiiON storage system.
>
> I believe I've RTFM'd all the FM's I can find and hope someone will
> point out the really obvious thing I'm overlooking.
>
> Thanks much in advance all.

Sterve

2006-12-17, 1:12 pm

I used the "Update Now" several times at different stages, and even
gave it a few hours afterwards to be sure.
The navicli works, but doesn't see the new host:

# navicli -h 10.50.40.67 port -list -hba -all > /tmp/hba.out

# grep UID /tmp/hba.out
HBA UID:
20:00:00:00:C9:38:E5:C6:10:00:00:00:C9:3
8:E5:C6
HBA UID:
20:00:00:00:C9:33:16:DF:10:00:00:00:C9:3
3:16:DF
HBA UID:
20:00:00:00:C9:40:48:1D:10:00:00:00:C9:4
0:48:1D
HBA UID:
20:00:00:00:C9:40:46:47:10:00:00:00:C9:4
0:46:47
HBA UID:
20:00:00:00:C9:40:46:DE:10:00:00:00:C9:4
0:46:DE
HBA UID:
20:00:00:00:C9:40:45:F8:10:00:00:00:C9:4
0:45:F8
HBA UID:
20:00:00:00:C9:40:47:F8:10:00:00:00:C9:4
0:47:F8
HBA UID:
20:00:00:00:C9:40:46:87:10:00:00:00:C9:4
0:46:87
HBA UID:
20:00:00:00:C9:33:1C:F6:10:00:00:00:C9:3
3:1C:F6
HBA UID:
20:00:00:00:C9:33:1A:26:10:00:00:00:C9:3
3:1A:26
HBA UID:
20:00:00:00:C9:34:2D:99:10:00:00:00:C9:3
4:2D:99
HBA UID:
20:00:00:00:C9:34:2D:D4:10:00:00:00:C9:3
4:2D:D4
HBA UID:
20:00:00:00:C9:32:09:6C:10:00:00:00:C9:3
2:09:6C
HBA UID:
20:00:00:00:C9:32:03:E1:10:00:00:00:C9:3
2:03:E1
# grep UID /tmp/hba.out | wc -l
14

I had hoped to be #15 and #16. I don't think that the following is an
issue,
but I'll put it out there just in case.
The CX600 has 2 SP's - SP A & SP B, each with 4 ports. I noticed that
all of
the other initiators were assigned to ports 0 and 1 on each SP. I
hoped to introduce
balance, so I used ports 2 and 3 for the new HBA's. These ports are
present, error free,
and also available as pull-down options when I tried the manual
registration of the initiators.
I'll put one of the 2 HBA's on a 0/1 port just to see what (doesn't?)
happen.

noname@gmail.com wrote:[vbcol=seagreen]
> I don't have navi in front of me, but there is an update option
> somewhere.
> Do you see other hosts on the CX? And how many hosts are there?
> (not sure if you are running into it, but CXs have a limit on the
> # of initiators they can support)
>
> And finally do you see the host HBAs from the CLI (-port -list option)?
>
> Sterve wrote:

jcaparrotta@bltrading.com

2006-12-18, 1:13 pm

Steve you can try this out and see if it works

For Access Logix configurations, follow these steps (which assume that
the host is zoned to the array):

Install CLArrayS3 on the AIX host.

Install Navisphere host Agent.

Run cfgmgr to get sp device entries and LUNZ entries.

Start Navisphere Agent to push HBA and host information to the array.

Connect host to appropriate Storage Group containing LUNs.

Remove any LUNZ devices by doing the following:
a. Run lsdev -Cc disk - This will show all hdisks that belong to
the CLARiiON array in the description. You will see LUNZ and there will
be two for each array.
b. Remove all LUNZs using rmdev -dl hdisk# (where # is the
applicable device number for the LUNZ).

Set arraycommpath to 0 (disable) for all HBA paths to the array from
the AIX host and also
set failovermode to 1 (PowerPath) for all these paths.

Run cfgmgr to get all applicable hdisks.

Run emc_cfgmgr.sh if multiple adapters are connected to the same SP
ports.

Install EMC PowerPath.

Run powermt config to get hdiskpower devices.

Note: When adding a new array, you must run cfgmgr and NOT
emc_cfgmgr.sh to get the /dev/sp entries for the new devices.

CX600 - 128 initiators per SP and maximum of 256 initiators per CX600
available for server connections. Ports 0, 1, 2, and 3 on each SP in
any CX600 handle server connections. Port 3 on each SP in a CX600 with
MirrorView/A or MirrorView/S enabled also handles remote mirror
connections. Each path used in a MirrorView or SAN Copy relationship
between two storage system counts as an initiator for both storage
systems.

Joe Caparrotta
Jcaparrotta@bltrading.com

Sterve

2006-12-18, 7:15 pm

Thanks all,

I found it, and it was indeed a case of me traveling too fast on the
learning curve and hitting
an infrastructure guard-rail. When I attempted to "add balance" to the
port assignments, I had not physically followed the cabling all the
way from the switch rack to the SP's.
I mapped the HBA's to Ports 2 & 3.....and they have never been cabled
to anything.
Moved this server's HBA's to 0 & 1, and immediately observed the magic
I had hoped for on day on. Sorry to have wasted everyone's time on
something that dumb.

jcaparrotta@bltrading.com wrote:
> Steve you can try this out and see if it works
>
> For Access Logix configurations, follow these steps (which assume that
> the host is zoned to the array):
>
> Install CLArrayS3 on the AIX host.
>
> Install Navisphere host Agent.
>
> Run cfgmgr to get sp device entries and LUNZ entries.
>
> Start Navisphere Agent to push HBA and host information to the array.
>
> Connect host to appropriate Storage Group containing LUNs.
>
> Remove any LUNZ devices by doing the following:
> a. Run lsdev -Cc disk - This will show all hdisks that belong to
> the CLARiiON array in the description. You will see LUNZ and there will
> be two for each array.
> b. Remove all LUNZs using rmdev -dl hdisk# (where # is the
> applicable device number for the LUNZ).
>
> Set arraycommpath to 0 (disable) for all HBA paths to the array from
> the AIX host and also
> set failovermode to 1 (PowerPath) for all these paths.
>
> Run cfgmgr to get all applicable hdisks.
>
> Run emc_cfgmgr.sh if multiple adapters are connected to the same SP
> ports.
>
> Install EMC PowerPath.
>
> Run powermt config to get hdiskpower devices.
>
> Note: When adding a new array, you must run cfgmgr and NOT
> emc_cfgmgr.sh to get the /dev/sp entries for the new devices.
>
> CX600 - 128 initiators per SP and maximum of 256 initiators per CX600
> available for server connections. Ports 0, 1, 2, and 3 on each SP in
> any CX600 handle server connections. Port 3 on each SP in a CX600 with
> MirrorView/A or MirrorView/S enabled also handles remote mirror
> connections. Each path used in a MirrorView or SAN Copy relationship
> between two storage system counts as an initiator for both storage
> systems.
>
> Joe Caparrotta
> Jcaparrotta@bltrading.com