| Dan Kohn 2005-04-04, 5:47 pm |
| My wife's new Toshiba Tecra M3 was hobbled by two major cisco VPN
bugs. The first problem is caused by a bad interaction between the
Cisco VPN client and the Intel Wi-Fi driver, which fails to work when
resuming from a suspend. The second is that the newest cisco VPN
drivers cause random Matlab execution failures while the VPN is
connected. I fixed both bugs and thought others might find the
solutions useful. However, Windows Explorer still crashes
occassionally, so I'm very interested in a new cisco VPN Client
release that fixes both bugs without any side effects. Having to deal
with issues like this is a fairly damning indictment of the whole QA
process theoretically employed by the 3 biggest tech companies: Cisco,
Intel, and Microsoft. Each bug separately was a nightmare to diagnose
(who would think at first that the Matlab problems could be caused by
the VPN?).
SUSPEND/RESUME BUG
The suspend/resume error occurs between the last 5 versions (4.0.4
through 4.6.02.0011) of the cisco VPN client and the Intel(R)
PRO/Wireless 2915ABG built-in Wi-Fi running Network Connection Driver
9.0.1.9, dated 2004-10-29. This is the latest Wi-Fi driver for the
Toshiba Tecra M3 running Windows XP SP2. I can consistently connect to
the VPN with this software. But when I then suspend and resume the
laptop, the Wi-Fi no longer works, and therefore the VPN is unable to
reconnect. Specifically, the Wi-Fi can see and get on a local network,
and it can pass traffic within the subnet to itself, the router, and
another computer. But it is not able to communicate beyond the router,
including not being able to do any DNS lookups. Choosing Repair for
the connection, using ipconfig /release and /renew, and choosing a
different network all fail to work.
The problem is caused by corruption in the route tables, which store
the Default Gateway. When the VPN is active, this gateway is correctly
set to the VPN server, which is 199.9.0.41 in my case. The problem is
that when the laptop comes back from a suspend/resume, the VPN Client
loses the connection, but it does not restore the Default Gateway to
the IP address of my local router, which is 192.168.200.1. At that
point, all network connections besides pinging nodes on my local subnet
fail, since the computer can't talk through the router. VPN
reconnection also fails since the VPN server is no longer accessible
either.
The quick fix (which is also useful as a fallback), is to create a
batch file with two commands (or to just run them from a command
window). "route -f" deletes the (incorrect) Default Gateway from
the route table. "ipconfig /renew" reconfirms the IP address with
the Wi-Fi router, and also restores the correct gateway to the route
table. You can create a batch file by right-clicking on the Desktop,
selecting New Text File, naming it "networkfix.bat" and hitting
enter, right-clicking the file and selecting edit, enter the two
commands on two lines, and close and save. Run it by double-clicking.
The more permanent fix is to lower the interface metric for the Wi-Fi
so that its Default Gateway is ranked higher. However, this seems to
occasionally result in Windows Explorer crashing after a
suspend/resume. (If Explorer crashes, hit Ctrl-Alt-Del, choose Task
Manager, select File: New Task (Run...), type Explorer, and hit enter.)
Interface metric is a measure of cost, so lower is better. The
default is 30. When the cisco VPN is connected to the VPN server, it
sets the metric to 1, which is the highest priority. After
disconnection, when the cisco VPN incorrectly adds the VPN server as a
gateway, it sets the metric to 10. So, by setting the Wi-Fi router
metric to 5, we can have it be lower priority than the VPN gateway when
connected to the VPN but higher when we are not, which is the correct
behavior. To do this, right-click the Wi-Fi icon on the System tray,
choose Status, click Properties, select Internet Protocol (TCP/IP),
click Properties, click Advanced, unclick Automatic metric, enter 5,
and hit OK twice and Close once. More details, including the route
tables, are below.
The real fix is for cisco and Intel to get their drivers to work
together. And, of course, Microsoft should not be using a driver
architecture that occasionally causes Windows Explorer to crash.
MATLAB/CISCO VPN BUG
The Matlab problem occurs with the two newest cisco VPN Client
versions, 4.6.01.0019 and 4.6.02.0011, and Matlab 7.0.4.365 (R14SP2).
Matlab will fail to execute while they are connected, and even worse,
will throw a variety of different errors each time it is run. The
Matlab runs complete correctly when the cisco VPN Client is running but
disconnected. The problem was fixed by uninstalling the buggy versions
and then installing cisco VPN 4.0.5(B), and also does not occur with
versions 4.0.5 and 4.0.4. I've never seen a network driver bug cause
random errors in a regular program, particularly one that shouldn't
even be accessing the network.
Here are the Matlab errors from 3 sequential runs where cisco VPN
4.6.02.0011 was connected:
Run 1:
??? Subscript indices must either be real positive integers or
logicals.
Error in ==> sortrows>sort_back_to_front at 162
ndx = ndx(ind);
Error in ==> sortrows at 123
ndx = sort_back_to_front(x_sub);
Error in ==> griddata at 75
sxyz = sortrows([x y z],[2 1]);
Error in ==> put_ncom at 157
zic(:,:,k)=griddata(lon_roms,lat_roms,zi
2(:,:,k),lon6,lat6,'nearest');
Run 2:
??? NaN's cannot be converted to logicals.
Error in ==> interp1 at 142
if eqsp
Error in ==> put_ncom at 180
si_z(ii,jj, =interp1(z_in,s_in,z_out,'n
earest','extrap');
Run 3:
??? Attempted to access cmsk(-2.14748e+009,55); index must be a
positive integer or logical.
Error in ==> creep at 37
if (cmsk(i,j) < 0.5)
Error in ==> put_ncom at 164
& #91;sic2(:,:,k)]=creep(sic(:,:,k),n_lon,
n_lat,iter);
ROUTE TABLES FOR SUSPEND/RESUME BUG
Here are the route tables with the Wi-Fi interface metric set to
automatic.
Fresh from restart:
C:\Documents and Settings\pullen>route print
========================================
===================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ...00 0e 35 ea e3 8d ...... Intel(R) PRO/Wireless 2915ABG Network
Connection
- Packet Scheduler Miniport
0x3 ...00 0e 7b 9a 8d 23 ...... Marvell Yukon 88E8053 PCI-E Gigabit
Ethernet Con
troller - Packet Scheduler Miniport
========================================
===================================
========================================
===================================
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
30
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
30
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
30
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
30
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
30
255.255.255.255 255.255.255.255 192.168.200.105 3
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
Default Gateway: 192.168.200.1
========================================
===================================
Persistent Routes:
None
With VPN connected (note default gateway has correctly changed from
192.168.200.1 to 199.9.0.40):
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
31
0.0.0.0 0.0.0.0 199.9.0.40 199.9.0.40
1
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
138.18.187.2 255.255.255.255 192.168.200.1 192.168.200.105
1
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
30
192.168.200.0 255.255.255.0 199.9.0.40 199.9.0.40
1
192.168.200.1 255.255.255.255 192.168.200.105 192.168.200.105
1
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
30
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
30
199.9.0.0 255.255.255.0 199.9.0.40 199.9.0.40
10
199.9.0.40 255.255.255.255 127.0.0.1 127.0.0.1
10
199.9.0.255 255.255.255.255 199.9.0.40 199.9.0.40
10
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
30
224.0.0.0 240.0.0.0 199.9.0.40 199.9.0.40
10
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
255.255.255.255 255.255.255.255 199.9.0.40 3
1
255.255.255.255 255.255.255.255 199.9.0.40 199.9.0.40
1
Default Gateway: 199.9.0.40
After resume (ping slashdot.org works):
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
30
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
30
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
30
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
30
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
30
255.255.255.255 255.255.255.255 192.168.200.105 3
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
Default Gateway: 192.168.200.1
After selecting Connect in the VPN client (VPN fails to initialize and
ping slashdot.org now fails, probably because the VPN software is
changing the Default Gateway before the VPN connection is established):
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.160.158.190 192.160.158.26
10
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
30
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
192.160.158.0 255.255.255.0 192.160.158.26 192.160.158.26
10
192.160.158.26 255.255.255.255 127.0.0.1 127.0.0.1
10
192.160.158.255 255.255.255.255 192.160.158.26 192.160.158.26
10
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
30
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
30
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
30
224.0.0.0 240.0.0.0 192.160.158.26 192.160.158.26
10
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
30
255.255.255.255 255.255.255.255 192.160.158.26 192.160.158.26
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
Default Gateway: 192.160.158.190
After changing Interface metric to 5 to create a permanent fix (Note
that running networkfix.bat also restores connectivity here, but only
until the next suspend/resume):
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.160.158.190 192.160.158.26
10
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
5
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
192.160.158.0 255.255.255.0 192.160.158.26 192.160.158.26
10
192.160.158.26 255.255.255.255 127.0.0.1 127.0.0.1
10
192.160.158.255 255.255.255.255 192.160.158.26 192.160.158.26
10
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
5
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
5
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
5
224.0.0.0 240.0.0.0 192.160.158.26 192.160.158.26
10
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
5
255.255.255.255 255.255.255.255 192.160.158.26 192.160.158.26
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
Default Gateway: 192.168.200.1
After VPN successfully reconnects:
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.160.158.190 192.160.158.26
10
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
6
0.0.0.0 0.0.0.0 199.9.0.36 199.9.0.36
1
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
138.18.187.2 255.255.255.255 192.168.200.1 192.168.200.105
1
192.160.158.0 255.255.255.0 192.160.158.26 192.160.158.26
10
192.160.158.26 255.255.255.255 127.0.0.1 127.0.0.1
10
192.160.158.255 255.255.255.255 192.160.158.26 192.160.158.26
10
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
5
192.168.200.0 255.255.255.0 199.9.0.36 199.9.0.36
1
192.168.200.1 255.255.255.255 192.168.200.105 192.168.200.105
1
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
5
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
5
199.9.0.0 255.255.255.0 199.9.0.36 199.9.0.36
10
199.9.0.36 255.255.255.255 127.0.0.1 127.0.0.1
10
199.9.0.255 255.255.255.255 199.9.0.36 199.9.0.36
10
224.0.0.0 240.0.0.0 192.160.158.26 192.160.158.26
10
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
5
224.0.0.0 240.0.0.0 199.9.0.36 199.9.0.36
10
255.255.255.255 255.255.255.255 192.160.158.26 192.160.158.26
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
255.255.255.255 255.255.255.255 199.9.0.36 199.9.0.36
1
Default Gateway: 199.9.0.36
On suspend/resume Windows Explorer dies and then restarts again,
although network works fine:
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.160.158.190 192.160.158.26
10
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
5
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
192.160.158.0 255.255.255.0 192.160.158.26 192.160.158.26
10
192.160.158.26 255.255.255.255 127.0.0.1 127.0.0.1
10
192.160.158.255 255.255.255.255 192.160.158.26 192.160.158.26
10
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
5
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
5
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
5
224.0.0.0 240.0.0.0 192.160.158.26 192.160.158.26
10
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
5
255.255.255.255 255.255.255.255 192.160.158.26 192.160.158.26
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
Default Gateway: 192.168.200.1
After reconnection:
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.160.158.190 192.160.158.26
10
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
6
0.0.0.0 0.0.0.0 199.9.0.61 199.9.0.61
1
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
138.18.187.2 255.255.255.255 192.168.200.1 192.168.200.105
1
192.160.158.0 255.255.255.0 192.160.158.26 192.160.158.26
10
192.160.158.26 255.255.255.255 127.0.0.1 127.0.0.1
10
192.160.158.255 255.255.255.255 192.160.158.26 192.160.158.26
10
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
5
192.168.200.0 255.255.255.0 199.9.0.61 199.9.0.61
1
192.168.200.1 255.255.255.255 192.168.200.105 192.168.200.105
1
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
5
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
5
199.9.0.0 255.255.255.0 199.9.0.61 199.9.0.61
10
199.9.0.61 255.255.255.255 127.0.0.1 127.0.0.1
10
199.9.0.255 255.255.255.255 199.9.0.61 199.9.0.61
10
224.0.0.0 240.0.0.0 192.160.158.26 192.160.158.26
10
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
5
224.0.0.0 240.0.0.0 199.9.0.61 199.9.0.61
10
255.255.255.255 255.255.255.255 192.160.158.26 192.160.158.26
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
255.255.255.255 255.255.255.255 199.9.0.61 199.9.0.61
1
Default Gateway: 199.9.0.61
After suspend/resume, Windows Explorer quits again and has to be
manually restarted from Windows Task Manager. This is the first time
VPN has stayed active through suspend/resume. Ping slashdot.org fails,
because the Default Gateway is the local one instead of the VPN server:
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.160.158.190 192.160.158.26
10
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
4
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
192.160.158.0 255.255.255.0 192.160.158.26 192.160.158.26
10
192.160.158.26 255.255.255.255 127.0.0.1 127.0.0.1
10
192.160.158.255 255.255.255.255 192.160.158.26 192.160.158.26
10
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
5
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
5
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
5
224.0.0.0 240.0.0.0 192.160.158.26 192.160.158.26
10
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
5
255.255.255.255 255.255.255.255 192.160.158.26 192.160.158.26
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
Default Gateway: 192.168.200.1
On disconnecting VPN (network works again):
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.160.158.190 192.160.158.26
10
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
4
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
192.160.158.0 255.255.255.0 192.160.158.26 192.160.158.26
10
192.160.158.26 255.255.255.255 127.0.0.1 127.0.0.1
10
192.160.158.255 255.255.255.255 192.160.158.26 192.160.158.26
10
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
5
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
5
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
5
224.0.0.0 240.0.0.0 192.160.158.26 192.160.158.26
10
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
5
255.255.255.255 255.255.255.255 192.160.158.26 192.160.158.26
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
Default Gateway: 192.168.200.1
On reconnecting to VPN:
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.160.158.190 192.160.158.26
10
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
5
0.0.0.0 0.0.0.0 199.9.0.44 199.9.0.44
1
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
138.18.187.2 255.255.255.255 192.168.200.1 192.168.200.105
1
192.160.158.0 255.255.255.0 192.160.158.26 192.160.158.26
10
192.160.158.26 255.255.255.255 127.0.0.1 127.0.0.1
10
192.160.158.255 255.255.255.255 192.160.158.26 192.160.158.26
10
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
5
192.168.200.0 255.255.255.0 199.9.0.44 199.9.0.44
1
192.168.200.1 255.255.255.255 192.168.200.105 192.168.200.105
1
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
5
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
5
199.9.0.0 255.255.255.0 199.9.0.44 199.9.0.44
10
199.9.0.44 255.255.255.255 127.0.0.1 127.0.0.1
10
199.9.0.255 255.255.255.255 199.9.0.44 199.9.0.44
10
224.0.0.0 240.0.0.0 192.160.158.26 192.160.158.26
10
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
5
224.0.0.0 240.0.0.0 199.9.0.44 199.9.0.44
10
255.255.255.255 255.255.255.255 192.160.158.26 192.160.158.26
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
255.255.255.255 255.255.255.255 199.9.0.44 199.9.0.44
1
Default Gateway: 199.9.0.44
On resuming from suspend:
Active Routes:
Network Destination Netmask Gateway Interface
Metric
0.0.0.0 0.0.0.0 192.160.158.190 192.160.158.26
10
0.0.0.0 0.0.0.0 192.168.200.1 192.168.200.105
5
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1
1
192.160.158.0 255.255.255.0 192.160.158.26 192.160.158.26
10
192.160.158.26 255.255.255.255 127.0.0.1 127.0.0.1
10
192.160.158.255 255.255.255.255 192.160.158.26 192.160.158.26
10
192.168.200.0 255.255.255.0 192.168.200.105 192.168.200.105
5
192.168.200.105 255.255.255.255 127.0.0.1 127.0.0.1
5
192.168.200.255 255.255.255.255 192.168.200.105 192.168.200.105
5
224.0.0.0 240.0.0.0 192.160.158.26 192.160.158.26
10
224.0.0.0 240.0.0.0 192.168.200.105 192.168.200.105
5
255.255.255.255 255.255.255.255 192.160.158.26 192.160.158.26
1
255.255.255.255 255.255.255.255 192.168.200.105 192.168.200.105
1
Default Gateway: 192.168.200.1
- dan
--
Dan Kohn <mailto:dan@dankohn.com>
<http://www.dankohn.com/> <tel:+1-650-327-2600>
|