Voice over IP Cisco - Strict Priority Queuing over T1 for Voice

Author

Strict Priority Queuing over T1 for Voice

Keith Klevenski

2006-10-24, 7:11 pm

Hi all,

I've been battling a problem that seemed to have surfaced recently as we
have added more users to the office. We have an office of about 35
users now with a point to point T1 (HDLC) between this office and the
datacenter where the CallManager's (running 4.1.3) and PSTN gateway
(6608 blade) are. Basically this:

PSTN

|

6608/6509 (datacenter) (12.1(11b)E14 on msfc and 6.4(10) on sup)

| trunk

7200 (datacenter) 12.3(6a)

|

T1

|

2811 (remote office) 12.3(11)T7

| trunk

3750 (remote office)

|

Phones

We have strict priority queuing configured on the T1 interfaces (and the
FastEthernet interfaces although I don't think this is necessary) with
DSCP EF traffic in the high queue and signaling (AF31/21/22/23 and CS3)
the medium queue and other traffic in the normal and low queues.

The problem is when the T1 is saturated we start seeing high jitter (1-2
seconds) and rxlost packets on the phones and voice quality of course
suffers. The thing is I see NO dropped packets on any interface from
the PSTN to the switch port the phone is connected to other than the
call statistics on the phone. This confuses me greatly unless the phone
itself is dropping them... it would seem that if the phone determines
there are missing packets (rxlost) they would show up on some interface
somewhere but they do not. Sniffer traces at the phone show the missing
packets. Sniffer traces at the trunk at the remote office show missing
packets, but traces on the trunk in the datacenter (from the 7200 to
6509) show all packets accounted for. Seems like the T1 is the issue.
There are 24 line code violations on the T1 in the last 24 hours, but no
slips. The problem only seems to occur when there is high traffic on
the T1. We also noticed no other traffic is marked EF and the RTP
stream is EF at the datacenter and at the office so that isn't being
remarked or anything. But there are no input queue drops on the 2811
(which I wouldn't expect since the RTP stream should be CEF switched
anyway). If the input interface cannot handle the amount of traffic and
the traffic is all CEF switched where would you see input drops? There
are a handful of buffer failures, but nothing that would account for the
number of complaints I've been getting lately.

I've connected my phone directly to the 2811 and bypassed the 3750 with
the same results.

I also noticed that the rxsize fluctuates between 10ms and 20ms also.
Shouldn't that always be 20ms? I thought that was odd.

I guess my main question is when using strict priority queuing shouldn't
the voice packets ALWAYS get sent first no matter what?? I want to
believe they are since there are NO drops in the high priority queue
outbound from the datacenter. What is the best way to do QoS for voice
over a T1 when you only care about the voice? Seems like strict
priority queuing is the best way to assure voice traffic is sent first.
We don't really care about starving out any other traffic. If packets
are being dropped shouldn't I see them somewhere other than on the
phone?

Any input on this problem is welcome!

Keith

Keith Klevenski

2006-10-24, 7:11 pm

One thing to add...

I am testing now by saturating the link and making an internal call to
another IP phone at another remote site that of course goes over the
same T1 (plus another T1) and I do not see any jitter or rxlost
problems. I had noticed that I never saw any problem to our other
office so I wanted to test it now. Max jitter is only 29 and no rxlost
in 12 minutes so far with the T1 saturated plus the stream has to go
over another T1 as well.

Could this be an issue with the 6608? I remember coming across a bug
about the sample size changing from 10ms to 20ms using g729 on a 6608
resulting in poor quality. That bug fit perfectly, but it was for ccm
3.0.10 I think.

Anyway, this is getting stranger by the minute... ;)

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Keith Klevenski
Sent: Tuesday, October 24, 2006 4:43 PM
To: cisco-voip@puck.nether.net
Subject: [cisco-voip] Strict Priority Queuing over T1 for Voice

Hi all,

I've been battling a problem that seemed to have surfaced recently as we
have added more users to the office. We have an office of about 35
users now with a point to point T1 (HDLC) between this office and the
datacenter where the CallManager's (running 4.1.3) and PSTN gateway
(6608 blade) are. Basically this:

PSTN

|

6608/6509 (datacenter) (12.1(11b)E14 on msfc and 6.4(10) on sup)

| trunk

7200 (datacenter) 12.3(6a)

|

T1

|

2811 (remote office) 12.3(11)T7

| trunk

3750 (remote office)

|

Phones

We have strict priority queuing configured on the T1 interfaces (and the
FastEthernet interfaces although I don't think this is necessary) with
DSCP EF traffic in the high queue and signaling (AF31/21/22/23 and CS3)
the medium queue and other traffic in the normal and low queues.

The problem is when the T1 is saturated we start seeing high jitter (1-2
seconds) and rxlost packets on the phones and voice quality of course
suffers. The thing is I see NO dropped packets on any interface from
the PSTN to the switch port the phone is connected to other than the
call statistics on the phone. This confuses me greatly unless the phone
itself is dropping them... it would seem that if the phone determines
there are missing packets (rxlost) they would show up on some interface
somewhere but they do not. Sniffer traces at the phone show the missing
packets. Sniffer traces at the trunk at the remote office show missing
packets, but traces on the trunk in the datacenter (from the 7200 to
6509) show all packets accounted for. Seems like the T1 is the issue.
There are 24 line code violations on the T1 in the last 24 hours, but no
slips. The problem only seems to occur when there is high traffic on
the T1. We also noticed no other traffic is marked EF and the RTP
stream is EF at the datacenter and at the office so that isn't being
remarked or anything. But there are no input queue drops on the 2811
(which I wouldn't expect since the RTP stream should be CEF switched
anyway). If the input interface cannot handle the amount of traffic and
the traffic is all CEF switched where would you see input drops? There
are a handful of buffer failures, but nothing that would account for the
number of complaints I've been getting lately.

I've connected my phone directly to the 2811 and bypassed the 3750 with
the same results.

I also noticed that the rxsize fluctuates between 10ms and 20ms also.
Shouldn't that always be 20ms? I thought that was odd.

I guess my main question is when using strict priority queuing shouldn't
the voice packets ALWAYS get sent first no matter what?? I want to
believe they are since there are NO drops in the high priority queue
outbound from the datacenter. What is the best way to do QoS for voice
over a T1 when you only care about the voice? Seems like strict
priority queuing is the best way to assure voice traffic is sent first.
We don't really care about starving out any other traffic. If packets
are being dropped shouldn't I see them somewhere other than on the
phone?

Any input on this problem is welcome!

Keith

Andre Beck

2006-10-25, 7:11 am

Hi,

On Tue, Oct 24, 2006 at 04:43:15PM -0500, Keith Klevenski wrote:
>
> I've been battling a problem that seemed to have surfaced recently as we
> have added more users to the office. We have an office of about 35
> users now with a point to point T1 (HDLC) between this office and the

Running PPP instead of Cisco-HDLC would allow you to use MP and LFI with
the advantage of further reducing the serialization delay, but given that
it's a T1, that isn't really of the utmost priority - worst case jitter
introduced by SD on a T1 is approximately 8ms.

> We have strict priority queuing configured on the T1 interfaces (and the
> FastEthernet interfaces although I don't think this is necessary) with

Depends. FEs are usually FIFO as they are significantly faster than 2Mbps.
If you can generate traffic on the routers that could actually cause
congestion, you preferably QoS the router's FEs, too. A way to congest
them would e.g. be single-armed routing either on o stick (VLANs) or even
due to a default gateway setting that bounces off to another local router
in an environment where ICMP redirects don't work. If it's not an actual
DoS attack.

> DSCP EF traffic in the high queue and signaling (AF31/21/22/23 and CS3)
> the medium queue and other traffic in the normal and low queues.

Sounds like you're using classic/legacy PQ? Hopefully this isn't old
code that started to deteriorate. I'd prefer MQC in such cases, though
I've seen PQ work perfectly. You just might have to tune queue sizes.

> The problem is when the T1 is saturated we start seeing high jitter (1-2
> seconds) and rxlost packets on the phones and voice quality of course
> suffers. The thing is I see NO dropped packets on any interface from
> the PSTN to the switch port the phone is connected to other than the
> call statistics on the phone. This confuses me greatly unless the phone
> itself is dropping them...

Could mean they were defective, but that should have triggered one of
the L2 checksums that were in intermediate use (Ethernet, HDLC).

> it would seem that if the phone determines
> there are missing packets (rxlost) they would show up on some interface
> somewhere but they do not. Sniffer traces at the phone show the missing
> packets. Sniffer traces at the trunk at the remote office show missing
> packets, but traces on the trunk in the datacenter (from the 7200 to
> 6509) show all packets accounted for. Seems like the T1 is the issue.
> There are 24 line code violations on the T1 in the last 24 hours, but no
> slips. The problem only seems to occur when there is high traffic on
> the T1.

This is indeed rather strange.

> We also noticed no other traffic is marked EF and the RTP
> stream is EF at the datacenter and at the office so that isn't being
> remarked or anything. But there are no input queue drops on the 2811
> (which I wouldn't expect since the RTP stream should be CEF switched
> anyway). If the input interface cannot handle the amount of traffic and
> the traffic is all CEF switched where would you see input drops? There
> are a handful of buffer failures, but nothing that would account for the
> number of complaints I've been getting lately.

Is the router by chance experiencing high CPU load, especially interrupts?
Are you perfectly sure all switching is CEF? Try "show cef int brief".

> I've connected my phone directly to the 2811 and bypassed the 3750 with
> the same results.
>
> I also noticed that the rxsize fluctuates between 10ms and 20ms also.
> Shouldn't that always be 20ms? I thought that was odd.

Dont know about the 6608.

> I guess my main question is when using strict priority queuing shouldn't
> the voice packets ALWAYS get sent first no matter what?? I want to
> believe they are since there are NO drops in the high priority queue
> outbound from the datacenter.

That's the most interesting point here. If there are no drops here
(either in PQ interface stats or "sh policy-map interface" for MQC)
on the relevant queue/class, the packets should be there. Sometimes
there are odd issues, though - I have one where the remote offices are
connected through GRE tunnels established through an IPsec VPN, and I
have a cyclic phase of approx. 0.5s where I lose all packets which
repeats every approx. 253s. Not tracked to a rational cause, yet - and
no, it's not SA renegotiation, the loss is only *within* the GRE, not
in other flows through the VPN...

> What is the best way to do QoS for voice
> over a T1 when you only care about the voice? Seems like strict
> priority queuing is the best way to assure voice traffic is sent first.

It should work. I'd prefer MQC and having the priority class cupped at
the top at around 80% of the real line rate, but as long as starvation
is really no matter, PQ should do as well.

> We don't really care about starving out any other traffic.

I'd prefer at least the management traffic still works. But I assume
that 1) line keepalive is highest priority already and 2) if it would
be a problem from this area, you'd have seen it in the logs already.

> If packets are being dropped shouldn't I see them somewhere other
> than on the phone?

I'd think so. Either in a output queue or in an incoming error counter.
On any router or switch in the path. But it sounds like you already
investigated all of them. What's strange is the coupling of the problem
to high load on the T1. And the fact you don't just see RX losses, but
also jitter in the full seconds timerange. This must mean something is
queued and/or delayed somewhere. Any idea what's generating this load?

Andre.
--
The _S_anta _C_laus _O_peration
or "how to turn a complete illusion into a neverending money source"

-> Andre Beck +++ ABP-RIPE +++ IBH Prof. Dr. Horn GmbH, Dresden <-

Jason Aarons \(US\)

2006-10-25, 1:13 pm

Auto qos voip does a great job of generating the policy-maps. What does
your policy look like?.

What does your showpolicy-map interface x/y/z show when the problem
occurs? Are LLQ matched packets being discarded? Maybe the
service-policy is not protecting the voice like you think.

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Keith Klevenski
Sent: Tuesday, October 24, 2006 5:43 PM
To: cisco-voip@puck.nether.net
Subject: [cisco-voip] Strict Priority Queuing over T1 for Voice

Hi all,

I've been battling a problem that seemed to have surfaced recently as we
have added more users to the office. We have an office of about 35
users now with a point to point T1 (HDLC) between this office and the
datacenter where the CallManager's (running 4.1.3) and PSTN gateway
(6608 blade) are. Basically this:

PSTN

|

6608/6509 (datacenter) (12.1(11b)E14 on msfc and 6.4(10) on sup)

| trunk

7200 (datacenter) 12.3(6a)

|

T1

|

2811 (remote office) 12.3(11)T7

| trunk

3750 (remote office)

|

Phones

We have strict priority queuing configured on the T1 interfaces (and the
FastEthernet interfaces although I don't think this is necessary) with
DSCP EF traffic in the high queue and signaling (AF31/21/22/23 and CS3)
the medium queue and othertraffic in the normal and low queues.

The problem is when the T1 is saturated we start seeing high jitter (1-2
seconds) and rxlost packets on the phones and voice quality of course
suffers. The thing is I see NO dropped packets on any interface from
thePSTN to the switch port the phone is connected to other than the
callstatistics on the phone. This confuses me greatly unless the phone
itself is dropping them... it would seem that if the phone determines
there are missing packets (rxlost) they would show up on some interface
somewhere but they do not. Sniffer traces at the phone show the missing
packets. Sniffer traces at the trunk at the remote office show missing
packets, but traces on the trunk in the datacenter (from the 7200 to
6509) show all packets accounted for. Seems like the T1 is the issue.
There are 24 line code violations on the T1 in the last 24 hours, but no
slips. The problem only seems to occur when there is high traffic on
the T1. We also noticed no other traffic ismarked EF and the RTP
stream is EF at the datacenter and at the office so that isn't being
remarked or anything. But there are no input queue drops on the 2811
(which I wouldn't expect since the RTP stream should be CEF switched
anyway). If the input interface cannot handle the amount of traffic and
the traffic is all CEF switched where would you see input drops? There
are a handful of buffer failures, but nothing that would account for the
number of complaints I've been getting lately.

I've connected my phone directly to the2811 and bypassed the 3750 with
the same results.

I alsonoticed that the rxsize fluctuates between 10ms and 20ms also.
Shouldn't that always be 20ms? I thought that was odd.

I guess my main question is when using strict priority queuing shouldn't
the voice packets ALWAYS get sent first no matter what?? I want to
believe they are since there are NO drops in the high priority queue
outbound from the datacenter. What is the best way to do QoS for voice
over a T1 when you only care about the voice? Seems like strict
priority queuing is the best way to assure voice traffic is sent first.
We don't really care about starving out any other traffic. If packets
are being dropped shouldn't I see them somewhere other than onthe
phone?

Any input on this problem is welcome!

Keith

-----------------------------------------
Disclaimer:

This e-mail communication and any attachments may contain
confidential and privileged information and is for use by the
designated addressee(s) named above only. If you are not the
intended addressee, you are hereby notified that you have received
this communication in error and that any use or reproduction of
this email or its contents is strictly prohibited and may be
unlawful. If you have received this communication in error, please
notify us immediately by replying to this message and deleting it
from your computer. Thank you.

Keith Klevenski

2006-10-25, 1:13 pm

Just using plain old legacy priority queuing:

priority-list 1 protocol ip high list 110

priority-list 1 protocol ip medium list 111

priority-list 1 protocol ip normal list 112

priority-list 1 default low

access-list 110 permit ip any any dscp ef

access-list 111 permit ip any any dscp af31

access-list 111 permit ip any any dscp af21

access-list 111 permit ip any any dscp af22

access-list 111 permit ip any any dscp af23

access-list 111 permit ip any any dscp cs3

access-list 112 permit ip any any dscp af11

access-list 112 permit ip any any dscp af12

access-list 112 permit ip any any dscp af13

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Jason Aarons
(US)
Sent: Wednesday, October 25, 2006 8:08 AM
To: cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] Strict Priority Queuing over T1 for Voice

Auto qos voip does a great job of generating the policy-maps. What does
your policy look like?.

What does your show policy-map interface x/y/z show when the problem
occurs? Are LLQ matched packets being discarded? Maybe the
service-policy is not protecting the voice like you think.

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Keith Klevenski
Sent: Tuesday, October 24, 2006 5:43 PM
To: cisco-voip@puck.nether.net
Subject: [cisco-voip] Strict Priority Queuing over T1 for Voice

Hi all,

I've been battling a problem that seemed to have surfaced recently as we
have added more users to the office. We have an office of about 35
users now with a point to point T1 (HDLC) between this office and the
datacenter where the CallManager's (running 4.1.3) and PSTN gateway
(6608 blade) are. Basically this:

PSTN

|

6608/6509 (datacenter) (12.1(11b)E14 on msfc and 6.4(10) on sup)

| trunk

7200 (datacenter) 12.3(6a)

|

T1

|

2811 (remote office) 12.3(11)T7

| trunk

3750 (remote office)

|

Phones

We have strict priority queuing configured on the T1 interfaces (and the
FastEthernet interfaces although I don't think this is necessary) with
DSCP EF traffic in the high queue and signaling (AF31/21/22/23 and CS3)
the medium queue and other traffic in the normal and low queues.

The problem is when the T1 is saturated we start seeing high jitter (1-2
seconds) and rxlost packets on the phones and voice quality of course
suffers. The thing is I see NO dropped packets on any interface from
the PSTN to the switch port the phone is connected to other than the
call statistics on the phone. This confuses me greatly unless the phone
itself is dropping them... it would seem that if the phone determines
there are missing packets (rxlost) they would show up on some interface
somewhere but they do not. Sniffer traces at the phone show the missing
packets. Sniffer traces at the trunk at the remote office show missing
packets, but traces on the trunk in the datacenter (from the 7200 to
6509) show all packets accounted for. Seems like the T1 is the issue.
There are 24 line code violations on the T1 in the last 24 hours, but no
slips. The problem only seems to occur when there is high traffic on
the T1. We also noticed no other traffic is marked EF and the RTP
stream is EF at the datacenter and at the office so that isn't being
remarked or anything. But there are no input queue drops on the 2811
(which I wouldn't expect since the RTP stream should be CEF switched
anyway). If the input interface cannot handle the amount of traffic and
the traffic is all CEF switched where would you see input drops? There
are a handful of buffer failures, but nothing that would account for the
number of complaints I've been getting lately.

I've connected my phone directly to the 2811 and bypassed the 3750 with
the same results.

I also noticed that the rxsize fluctuates between 10ms and 20ms also.
Shouldn't that always be 20ms? I thought that was odd.

I guess my main question is when using strict priority queuing shouldn't
the voice packets ALWAYS get sent first no matter what?? I want to
believe they are since there are NO drops in the high priority queue
outbound from the datacenter. What is the best way to do QoS for voice
over a T1 when you only care about the voice? Seems like strict
priority queuing is the best way to assure voice traffic is sent first.
We don't really care about starving out any other traffic. If packets
are being dropped shouldn't I see them somewhere other than on the
phone?

Any input on this problem is welcome!

Keith

________________________________

Disclaimer: This e-mail communication and any attachments may contain
confidential and privileged information and is for use by the designated
addressee(s) named above only. If you are not the intended addressee,
you are hereby notified that you have received this communication in
error and that any use or reproduction of this email or its contents is
strictly prohibited and may be unlawful. If you have received this
communication in error, please notify us immediately by replying to this
message and deleting it from your computer. Thank you.

Keith Klevenski

2006-10-25, 1:13 pm

I appreciate all of the input Andre! See inline:

-----Original Message-----
From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Andre Beck
Sent: Wednesday, October 25, 2006 6:14 AM
To: cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] Strict Priority Queuing over T1 for Voice

Hi,

On Tue, Oct 24, 2006 at 04:43:15PM -0500, Keith Klevenski wrote:
>
> I've been battling a problem that seemed to have surfaced recently as
we
> have added more users to the office. We have an office of about 35
> users now with a point to point T1 (HDLC) between this office and the

Running PPP instead of Cisco-HDLC would allow you to use MP and LFI with
the advantage of further reducing the serialization delay, but given
that
it's a T1, that isn't really of the utmost priority - worst case jitter
introduced by SD on a T1 is approximately 8ms.

****************************************
****
I will certainly consider this
****************************************
****

> We have strict priority queuing configured on the T1 interfaces (and
the
> FastEthernet interfaces although I don't think this is necessary) with

Depends. FEs are usually FIFO as they are significantly faster than
2Mbps.
If you can generate traffic on the routers that could actually cause
congestion, you preferably QoS the router's FEs, too. A way to congest
them would e.g. be single-armed routing either on o stick (VLANs) or
even
due to a default gateway setting that bounces off to another local
router
in an environment where ICMP redirects don't work. If it's not an actual
DoS attack.

****************************************
****
It is a router on a stick, but there isn't that much traffic being
routed between the vlans
****************************************
****

> DSCP EF traffic in the high queue and signaling (AF31/21/22/23 and
CS3)
> the medium queue and other traffic in the normal and low queues.

Sounds like you're using classic/legacy PQ? Hopefully this isn't old
code that started to deteriorate. I'd prefer MQC in such cases, though
I've seen PQ work perfectly. You just might have to tune queue sizes.

****************************************
****
Yes, plain old PQ. priority-groups and priority-lists. Old school.
****************************************
****

> The problem is when the T1 is saturated we start seeing high jitter
(1-2
> seconds) and rxlost packets on the phones and voice quality of course
> suffers. The thing is I see NO dropped packets on any interface from
> the PSTN to the switch port the phone is connected to other than the
> call statistics on the phone. This confuses me greatly unless the
phone
> itself is dropping them...

Could mean they were defective, but that should have triggered one of
the L2 checksums that were in intermediate use (Ethernet, HDLC).

****************************************
****
I would think so too
****************************************
****

> it would seem that if the phone determines
> there are missing packets (rxlost) they would show up on some
interface
> somewhere but they do not. Sniffer traces at the phone show the
missing
> packets. Sniffer traces at the trunk at the remote office show
missing
> packets, but traces on the trunk in the datacenter (from the 7200 to
> 6509) show all packets accounted for. Seems like the T1 is the issue.
> There are 24 line code violations on the T1 in the last 24 hours, but
no
> slips. The problem only seems to occur when there is high traffic on
> the T1.

This is indeed rather strange.

> We also noticed no other traffic is marked EF and the RTP
> stream is EF at the datacenter and at the office so that isn't being
> remarked or anything. But there are no input queue drops on the 2811
> (which I wouldn't expect since the RTP stream should be CEF switched
> anyway). If the input interface cannot handle the amount of traffic
and
> the traffic is all CEF switched where would you see input drops?
There
> are a handful of buffer failures, but nothing that would account for
the
> number of complaints I've been getting lately.

Is the router by chance experiencing high CPU load, especially
interrupts?
Are you perfectly sure all switching is CEF? Try "show cef int brief".

****************************************
****
All interfaces are using CEF for sure
****************************************
****

> I've connected my phone directly to the 2811 and bypassed the 3750
with
> the same results.
>
> I also noticed that the rxsize fluctuates between 10ms and 20ms also.
> Shouldn't that always be 20ms? I thought that was odd.

Dont know about the 6608.

****************************************
****
I noticed later that it seemed that only traffic going through the 6608
was having the problem and I upgraded the loads to the latest last
night. I'm crossing my fingers that resolves the problem although I
doesn't make much sense. I've seen stranger..
****************************************
****

> I guess my main question is when using strict priority queuing
shouldn't
> the voice packets ALWAYS get sent first no matter what?? I want to
> believe they are since there are NO drops in the high priority queue
> outbound from the datacenter.

That's the most interesting point here. If there are no drops here
(either in PQ interface stats or "sh policy-map interface" for MQC)
on the relevant queue/class, the packets should be there. Sometimes
there are odd issues, though - I have one where the remote offices are
connected through GRE tunnels established through an IPsec VPN, and I
have a cyclic phase of approx. 0.5s where I lose all packets which
repeats every approx. 253s. Not tracked to a rational cause, yet - and
no, it's not SA renegotiation, the loss is only *within* the GRE, not
in other flows through the VPN...

****************************************
****
My biggest concern too. If the packets are being dropped then where is
the counter that shows me?
****************************************
****

> What is the best way to do QoS for voice
> over a T1 when you only care about the voice? Seems like strict
> priority queuing is the best way to assure voice traffic is sent
first.

It should work. I'd prefer MQC and having the priority class cupped at
the top at around 80% of the real line rate, but as long as starvation
is really no matter, PQ should do as well.

> We don't really care about starving out any other traffic.

I'd prefer at least the management traffic still works. But I assume
that 1) line keepalive is highest priority already and 2) if it would
be a problem from this area, you'd have seen it in the logs already.

> If packets are being dropped shouldn't I see them somewhere other
> than on the phone?

I'd think so. Either in a output queue or in an incoming error counter.
On any router or switch in the path. But it sounds like you already
investigated all of them. What's strange is the coupling of the problem
to high load on the T1. And the fact you don't just see RX losses, but
also jitter in the full seconds timerange. This must mean something is
queued and/or delayed somewhere. Any idea what's generating this load?

****************************************
****
I can simply download a file and the T1 is saturated. I've been
monitoring it and the test that I did was to get a couple of big file
downloads going and then I could pretty much reproduce the problem at
will. So the traffic that is saturating the T1 is just normal traffic
when people are downloading.
****************************************
****

Again I appreciate all of your input Andre!

Andre.
--
The _S_anta _C_laus _O_peration
or "how to turn a complete illusion into a neverending money source"

-> Andre Beck +++ ABP-RIPE +++ IBH Prof. Dr. Horn GmbH, Dresden <-
________________________________________
_______
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

Jason Aarons \(US\)

2006-10-25, 1:15 pm

I would change the QoS match the cisco SRND recommendations. Plus Cisco
can't say your are not using their best practices.

It seems to me you get more deatail in show policy-map interface versus
show queueing interface.

Aside from that look for show queueing interface, show interface and
drops. You indicate none are found, maybe verify that with
Ethereal/WireShark. The Smart Decode in Ethereal is pretty good.

________________________________

From: Keith Klevenski [mailto:keith.klevenski@rig.net]
Sent: Wednesday, October 25, 2006 10:36 AM
To: Jason Aarons (US); cisco-voip@puck.nether.net
Subject: RE: [cisco-voip] Strict Priority Queuing over T1 for Voice

Just using plain old legacy priority queuing:

priority-list 1 protocol ip high list 110

priority-list 1 protocol ip medium list 111

priority-list 1 protocol ip normal list112

priority-list 1 default low

access-list 110 permit ip any any dscp ef

access-list 111 permit ip anyany dscp af31

access-list 111 permit ip any any dscp af21

access-list 111 permit ip any any dscp af22

access-list 111 permit ip any any dscp af23

access-list 111 permit ip any any dscp cs3

access-list 112 permit ip any any dscp af11

access-list 112 permit ip any any dscp af12

access-list 112 permit ip any any dscp af13

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On BehalfOf Jason Aarons
(US)
Sent: Wednesday, October 25, 2006 8:08 AM
To: cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] Strict Priority Queuing over T1 for Voice

Auto qos voip does a great job of generating the policy-maps. What does
your policy look like?.

What does your show policy-map interface x/y/z show when the problem
occurs? Are LLQ matched packets being discarded? Maybe the
service-policy is not protecting the voice like you think.

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Keith Klevenski
Sent: Tuesday, October 24, 2006 5:43 PM
To: cisco-voip@puck.nether.net
Subject: [cisco-voip] Strict Priority Queuing over T1 for Voice

Hi all,

I've been battling a problem that seemed to have surfaced recently as we
have added more usersto the office. We have an office of about 35
users now with a point to point T1 (HDLC) between this office and the
datacenter where the CallManager's (running 4.1.3) and PSTN gateway
(6608 blade) are.Basically this:

PSTN

|

6608/6509 (datacenter) (12.1(11b)E14 on msfc and 6.4(10) on sup)

| trunk

7200 (datacenter) 12.3(6a)

|

T1

|

2811 (remote office) 12.3(11)T7

| trunk

3750 (remote office)

|

Phones

We have strict priority queuing configured onthe T1 interfaces (and the
FastEthernet interfaces although I don't think this is necessary) with
DSCP EF traffic in the high queue and signaling (AF31/21/22/23 and CS3)
the medium queue and other traffic in the normal and low queues.

The problem is when the T1 is saturated we start seeing high jitter (1-2
seconds) and rxlost packets on the phones and voice quality of course
suffers. The thing is I see NO dropped packets on any interface from
the PSTN to the switch port the phone is connected to other than the
call statistics on the phone. This confuses me greatly unless the phone
itself is dropping them... it would seem that if the phone determines
there are missing packets (rxlost) they would show up on some interface
somewhere but they do not. Sniffer traces at the phone show the missing
packets. Sniffer traces at the trunk at the remote office show missing
packets, but traces on the trunk in the datacenter (from the 7200 to
6509) show all packets accounted for. Seems like the T1 is the issue.
There are 24 line code violations on the T1 in the last 24 hours, but no
slips. The problem only seems to occur when there is high traffic on
the T1. We also noticed no other traffic is marked EF and the RTP
stream is EF at the datacenter and at the office so that isn't being
remarked or anything. But there are no input queue drops on the 2811
(which I wouldn't expect since the RTP stream should be CEFswitched
anyway). If the input interface cannot handle the amount of traffic and
the traffic is all CEF switched where would you see input drops? There
are a handful of buffer failures, but nothing that would account for the
number of complaints I've been getting lately.

I've connected my phone directly to the 2811 and bypassed the 3750 with
the same results.

I also noticed that the rxsize fluctuates between 10ms and 20ms also.
Shouldn't that always be 20ms? I thought that was odd.

I guess my main question is when using strict priority queuing shouldn't
the voice packets ALWAYS get sent first no matter what?? I want to
believethey are since there are NO drops in the high priority queue
outboundfrom the datacenter. What is the best way to do QoS for voice
overa T1 when you only care about the voice? Seems like strict
priority queuing is the best way to assure voice traffic is sent first.
We don't really care about starving out any other traffic. If packets
are being dropped shouldn't I see them somewhere other than on the
phone?

Any input on this problem is welcome!

Keith

________________________________

Disclaimer: This e-mail communication and any attachments may contain
confidential and privileged information and is for use by the designated
addressee(s) named above only. If you are not the intendedaddressee,
you are hereby notified that you have received this communication in
error and that any use or reproduction of this email or itscontents is
strictly prohibited and may be unlawful. If you have received this
communication in error, please notify us immediately by replying to this
message and deleting it from your computer. Thank you.

-----------------------------------------
Disclaimer:

This e-mail communication and any attachments may contain
confidential and privileged information and is for use by the
designated addressee(s) named above only. If you are not the
intended addressee, you are hereby notified that you have received
this communication in error and that any use or reproduction of
this email or its contents is strictly prohibited and may be
unlawful. If you have receivedthis communication in error, please
notify us immediately by replying tothis message and deleting it
from your computer. Thank you.

Keith Klevenski

2006-10-25, 1:16 pm

Thanks Jason.

After doing some more research I think I may be hitting CSCsb49202:

6608 : Incorrect marking DSCP value to 0

It is first fixed in version 4.1(3)ES53 which is newer than SR3c which
is what we're running and is the latest SR.

If this is the problem this actually makes sense. I'm not seeing any
drops in the HIGH priority queue, but plenty in the low queue when there
is congestion on the T1. All of the RTP packets I saw were marked EF,
but I didn't see if every single packet in sequence was marked with EF.
If some are being marked with DSCP 0 then they would be dropped in the
low priority queue which I see plenty of drops there when there is
congestion on the T1. And the fact that if I am on an internal call
(over the T1) or out a different gateway I do not see any problems.

PQ has been working fine for us and works fine with links over satellite
for oil rigs. We only care about voice so that's why we use legacy PQ.
Easy and straightforward.

Instead of upgrading to the latest ES we're going to mark all RTP
packets with DSCP EF manually to make sure they are marked properly. I
will update the group with the results.

Thanks for all of the input!

Keith

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Jason Aarons
(US)
Sent: Wednesday, October 25, 2006 11:54 AM
To: cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] Strict Priority Queuing over T1 for Voice

I would change the QoS match the cisco SRND recommendations. Plus Cisco
can't say your are not using their best practices.

It seems to me you get more deatail in show policy-map interface versus
show queueing interface.

Aside from that look for show queueing interface, show interface and
drops. You indicate none are found, maybe verify that with
Ethereal/WireShark. The Smart Decode in Ethereal is pretty good.

________________________________

From: Keith Klevenski [mailto:keith.klevenski@rig.net]
Sent: Wednesday, October 25, 2006 10:36 AM
To: Jason Aarons (US); cisco-voip@puck.nether.net
Subject: RE: [cisco-voip] Strict Priority Queuing over T1 for Voice

Just using plain old legacy priority queuing:

priority-list 1 protocol ip high list 110

priority-list 1 protocol ip medium list 111

priority-list 1 protocol ip normal list 112

priority-list 1 default low

access-list 110 permit ip any any dscp ef

access-list 111 permit ip any any dscp af31

access-list 111 permit ip any any dscp af21

access-list 111 permit ip any any dscp af22

access-list 111 permit ip any any dscp af23

access-list 111 permit ip any any dscp cs3

access-list 112 permit ip any any dscp af11

access-list 112 permit ip any any dscp af12

access-list 112 permit ip any any dscp af13

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Jason Aarons
(US)
Sent: Wednesday, October 25, 2006 8:08 AM
To: cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] Strict Priority Queuing over T1 for Voice

Auto qos voip does a great job of generating the policy-maps. What does
your policy look like?.

What does your show policy-map interface x/y/z show when the problem
occurs? Are LLQ matched packets being discarded? Maybe the
service-policy is not protecting the voice like you think.

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Keith Klevenski
Sent: Tuesday, October 24, 2006 5:43 PM
To: cisco-voip@puck.nether.net
Subject: [cisco-voip] Strict Priority Queuing over T1 for Voice

Hi all,

I've been battling a problem that seemed to have surfaced recently as we
have added more users to the office. We have an office of about 35
users now with a point to point T1 (HDLC) between this office and the
datacenter where the CallManager's (running 4.1.3) and PSTN gateway
(6608 blade) are. Basically this:

PSTN

|

6608/6509 (datacenter) (12.1(11b)E14 on msfc and 6.4(10) on sup)

| trunk

7200 (datacenter) 12.3(6a)

|

T1

|

2811 (remote office) 12.3(11)T7

| trunk

3750 (remote office)

|

Phones

We have strict priority queuing configured on the T1 interfaces (and the
FastEthernet interfaces although I don't think this is necessary) with
DSCP EF traffic in the high queue and signaling (AF31/21/22/23 and CS3)
the medium queue and other traffic in the normal and low queues.

The problem is when the T1 is saturated we start seeing high jitter (1-2
seconds) and rxlost packets on the phones and voice quality of course
suffers. The thing is I see NO dropped packets on any interface from
the PSTN to the switch port the phone is connected to other than the
call statistics on the phone. This confuses me greatly unless the phone
itself is dropping them... it would seem that if the phone determines
there are missing packets (rxlost) they would show up on some interface
somewhere but they do not. Sniffer traces at the phone show the missing
packets. Sniffer traces at the trunk at the remote office show missing
packets, but traces on the trunk in the datacenter (from the 7200 to
6509) show all packets accounted for. Seems like the T1 is the issue.
There are 24 line code violations on the T1 in the last 24 hours, but no
slips. The problem only seems to occur when there is high traffic on
the T1. We also noticed no other traffic is marked EF and the RTP
stream is EF at the datacenter and at the office so that isn't being
remarked or anything. But there are no input queue drops on the 2811
(which I wouldn't expect since the RTP stream should be CEF switched
anyway). If the input interface cannot handle the amount of traffic and
the traffic is all CEF switched where would you see input drops? There
are a handful of buffer failures, but nothing that would account for the
number of complaints I've been getting lately.

I've connected my phone directly to the 2811 and bypassed the 3750 with
the same results.

I also noticed that the rxsize fluctuates between 10ms and 20ms also.
Shouldn't that always be 20ms? I thought that was odd.

I guess my main question is when using strict priority queuing shouldn't
the voice packets ALWAYS get sent first no matter what?? I want to
believe they are since there are NO drops in the high priority queue
outbound from the datacenter. What is the best way to do QoS for voice
over a T1 when you only care about the voice? Seems like strict
priority queuing is the best way to assure voice traffic is sent first.
We don't really care about starving out any other traffic. If packets
are being dropped shouldn't I see them somewhere other than on the
phone?

Any input on this problem is welcome!

Keith

________________________________

Disclaimer: This e-mail communication and any attachments may contain
confidential and privileged information and is for use by the designated
addressee(s) named above only. If you are not the intended addressee,
you are hereby notified that you have received this communication in
error and that any use or reproduction of this email or its contents is
strictly prohibited and may be unlawful. If you have received this
communication in error, please notify us immediately by replying to this
message and deleting it from your computer. Thank you.

________________________________

Disclaimer: This e-mail communication and any attachments may contain
confidential and privileged information and is for use by the designated
addressee(s) named above only. If you are not the intended addressee,
you are hereby notified that you have received this communication in
error and that any use or reproduction of this email or its contents is
strictly prohibited and may be unlawful. If you have received this
communication in error, please notify us immediately by replying to this
message and deleting it from your computer. Thank you.

Andre Beck

2006-10-26, 7:11 am

Re Keith,

On Wed, Oct 25, 2006 at 12:06:21PM -0500, Keith Klevenski wrote:
>
> After doing some more research I think I may be hitting CSCsb49202:
>
> 6608 : Incorrect marking DSCP value to 0

Wuargh.

> If this is the problem this actually makes sense.

Perfect sense.

> I'm not seeing any
> drops in the HIGH priority queue, but plenty in the low queue when there
> is congestion on the T1. All of the RTP packets I saw were marked EF,
> but I didn't see if every single packet in sequence was marked with EF.
> If some are being marked with DSCP 0 then they would be dropped in the
> low priority queue which I see plenty of drops there when there is
> congestion on the T1. And the fact that if I am on an internal call
> (over the T1) or out a different gateway I do not see any problems.

You could probably verify that it is this problem using interface
precedence accounting, but it might get a little hard to differentiate
the packets hitting prec 0 intentionally from those hitting it
unintentionally. But at least it allows you to get a summary on this
without actually beeing on location with a sniffer. Or use an ACL that
doesn't actually drops anything, but allows you to count RTP packets
with a non-EF DSCP.

> PQ has been working fine for us and works fine with links over satellite
> for oil rigs. We only care about voice so that's why we use legacy PQ.
> Easy and straightforward.

You can even cup it at the top using rate-limit (CAR). Already done this
to prevent starvation. If present, I'd also reserve the high queue for
network management traffic (IGP), but cup it at 5% or so of the line rate.

> Instead of upgrading to the latest ES we're going to mark all RTP
> packets with DSCP EF manually to make sure they are marked properly. I
> will update the group with the results.

You might also just add an RTP match on the ACL that classifies your
high queue traffic (the one currently only matching DSCP EF), but
remarking once is probably easier than touching a lot of routers, if
there are many. If you happen to have a 3550/60/3750 switch somewhere
in the path, maybe you can convince it to do this for you at practically
no cost (100/1000Base wirespeed).

If you'd be on ATM PVCs I'd had shouted "Fix your tx-ring!" (some ATM
PAs essentially have a hidden hardware queue that can zero out all
positive effects of a previous software CBWFQ/LLQ by beeing too large
and involving serialization delays way beyond 200ms), but given *that*
bug, you very likely found your issue.

--
The _S_anta _C_laus _O_peration
or "how to turn a complete illusion into a neverending money source"

-> Andre Beck +++ ABP-RIPE +++ IBH Prof. Dr. Horn GmbH, Dresden <-

Keith Klevenski

2006-10-26, 1:11 pm

It turns out the problem is indeed the 6608. It intermittently marks
RTP packets with DSCP 0. This explains why I didn't see any drops on
any interfaces because I falsely assumed the packets were being marked
with EF so if they were dropped I should see them dropped in the high
priority queue. There were plenty of low priority queue drops, but I
never though the RTP stream would be part of those drops! We did look
at sniffer traces before, but didn't look at every single packet to see
how it was marked. That is where we didn't absolutely confirm that ALL
packets were being marked properly.

Oddly manually marking the packets on the vlan interface where all the
6608 ports are doesn't work either. You can see the matches in the show
policy-map int command and it looks like they are getting marked with
DSCP 46, but upon looking at the sniffer traces after the problem
continued the DSCP was still marked 0 for some packets just as it was
before.

Going to open a TAC case to see what the deal is with this bug. It is
first found in CatOS, but fixed in a CallManager release. That seems
pretty strange to me and even stranger why a service policy on the MSFC
didn't have any effect. I'm certain it was configured properly. It is
also odd that cisco says there is no known workaround for the problem.
Why can't you just manually mark the packets? Well, apparently you
can't at least from the MSFC so I guess it's all related somehow.

Anyway, just an FYI for you all. We ended bandaiding the problem by
putting all UDP traffic sourced from the 6608 ports in the high priority
queue on the outbound interface of the T1 (the 7200) at the datacenter
to the office until we get it resolved. I will keep you posted. ;)

Keith

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Keith Klevenski
Sent: Wednesday, October 25, 2006 12:06 PM
To: Jason Aarons (US); cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] Strict Priority Queuing over T1 for Voice

Thanks Jason.

After doing some more research I think I may be hitting CSCsb49202:

6608 : Incorrect marking DSCP value to 0

It is first fixed in version 4.1(3)ES53 which is newer than SR3c which
is what we're running and is the latest SR.

If this is the problem this actually makes sense. I'm not seeing any
drops in the HIGH priority queue, but plenty in the low queue when there
is congestion on the T1. All of the RTP packets I saw were marked EF,
but I didn't see if every single packet in sequence was marked with EF.
If some are being marked with DSCP 0 then they would be dropped in the
low priority queue which I see plenty of drops there when there is
congestion on the T1. And the fact that if I am on an internal call
(over the T1) or out a different gateway I do not see any problems.

PQ has been working fine for us and works fine with links over satellite
for oil rigs. We only care about voice so that's why we use legacy PQ.
Easy and straightforward.

Instead of upgrading to the latest ES we're going to mark all RTP
packets with DSCP EF manually to make sure they are marked properly. I
will update the group with the results.

Thanks for all of the input!

Keith

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Jason Aarons
(US)
Sent: Wednesday, October 25, 2006 11:54 AM
To: cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] Strict Priority Queuing over T1 for Voice

I would change the QoS match the cisco SRND recommendations. Plus Cisco
can't say your are not using their best practices.

It seems to me you get more deatail in show policy-map interface versus
show queueing interface.

Aside from that look for show queueing interface, show interface and
drops. You indicate none are found, maybe verify that with
Ethereal/WireShark. The Smart Decode in Ethereal is pretty good.

________________________________

From: Keith Klevenski [mailto:keith.klevenski@rig.net]
Sent: Wednesday, October 25, 2006 10:36 AM
To: Jason Aarons (US); cisco-voip@puck.nether.net
Subject: RE: [cisco-voip] Strict Priority Queuing over T1 for Voice

Just using plain old legacy priority queuing:

priority-list 1 protocol ip high list 110

priority-list 1 protocol ip medium list 111

priority-list 1 protocol ip normal list 112

priority-list 1 default low

access-list 110 permit ip any any dscp ef

access-list 111 permit ip any any dscp af31

access-list 111 permit ip any any dscp af21

access-list 111 permit ip any any dscp af22

access-list 111 permit ip any any dscp af23

access-list 111 permit ip any any dscp cs3

access-list 112 permit ip any any dscp af11

access-list 112 permit ip any any dscp af12

access-list 112 permit ip any any dscp af13

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Jason Aarons
(US)
Sent: Wednesday, October 25, 2006 8:08 AM
To: cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] Strict Priority Queuing over T1 for Voice

Auto qos voip does a great job of generating the policy-maps. What does
your policy look like?.

What does your show policy-map interface x/y/z show when the problem
occurs? Are LLQ matched packets being discarded? Maybe the
service-policy is not protecting the voice like you think.

________________________________

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Keith Klevenski
Sent: Tuesday, October 24, 2006 5:43 PM
To: cisco-voip@puck.nether.net
Subject: [cisco-voip] Strict Priority Queuing over T1 for Voice

Hi all,

I've been battling a problem that seemed to have surfaced recently as we
have added more users to the office. We have an office of about 35
users now with a point to point T1 (HDLC) between this office and the
datacenter where the CallManager's (running 4.1.3) and PSTN gateway
(6608 blade) are. Basically this:

PSTN

|

6608/6509 (datacenter) (12.1(11b)E14 on msfc and 6.4(10) on sup)

| trunk

7200 (datacenter) 12.3(6a)

|

T1

|

2811 (remote office) 12.3(11)T7

| trunk

3750 (remote office)

|

Phones

We have strict priority queuing configured on the T1 interfaces (and the
FastEthernet interfaces although I don't think this is necessary) with
DSCP EF traffic in the high queue and signaling (AF31/21/22/23 and CS3)
the medium queue and other traffic in the normal and low queues.

The problem is when the T1 is saturated we start seeing high jitter (1-2
seconds) and rxlost packets on the phones and voice quality of course
suffers. The thing is I see NO dropped packets on any interface from
the PSTN to the switch port the phone is connected to other than the
call statistics on the phone. This confuses me greatly unless the phone
itself is dropping them... it would seem that if the phone determines
there are missing packets (rxlost) they would show up on some interface
somewhere but they do not. Sniffer traces at the phone show the missing
packets. Sniffer traces at the trunk at the remote office show missing
packets, but traces on the trunk in the datacenter (from the 7200 to
6509) show all packets accounted for. Seems like the T1 is the issue.
There are 24 line code violations on the T1 in the last 24 hours, but no
slips. The problem only seems to occur when there is high traffic on
the T1. We also noticed no other traffic is marked EF and the RTP
stream is EF at the datacenter and at the office so that isn't being
remarked or anything. But there are no input queue drops on the 2811
(which I wouldn't expect since the RTP stream should be CEF switched
anyway). If the input interface cannot handle the amount of traffic and
the traffic is all CEF switched where would you see input drops? There
are a handful of buffer failures, but nothing that would account for the
number of complaints I've been getting lately.

I've connected my phone directly to the 2811 and bypassed the 3750 with
the same results.

I also noticed that the rxsize fluctuates between 10ms and 20ms also.
Shouldn't that always be 20ms? I thought that was odd.

I guess my main question is when using strict priority queuing shouldn't
the voice packets ALWAYS get sent first no matter what?? I want to
believe they are since there are NO drops in the high priority queue
outbound from the datacenter. What is the best way to do QoS for voice
over a T1 when you only care about the voice? Seems like strict
priority queuing is the best way to assure voice traffic is sent first.
We don't really care about starving out any other traffic. If packets
are being dropped shouldn't I see them somewhere other than on the
phone?

Any input on this problem is welcome!

Keith

________________________________

Disclaimer: This e-mail communication and any attachments may contain
confidential and privileged information and is for use by the designated
addressee(s) named above only. If you are not the intended addressee,
you are hereby notified that you have received this communication in
error and that any use or reproduction of this email or its contents is
strictly prohibited and may be unlawful. If you have received this
communication in error, please notify us immediately by replying to this
message and deleting it from your computer. Thank you.

________________________________

Disclaimer: This e-mail communication and any attachments may contain
confidential and privileged information and is for use by the designated
addressee(s) named above only. If you are not the intended addressee,
you are hereby notified that you have received this communication in
error and that any use or reproduction of this email or its contents is
strictly prohibited and may be unlawful. If you have received this
communication in error, please notify us immediately by replying to this
message and deleting it from your computer. Thank you.

Erick Bergquist

2006-10-28, 1:11 am

Another thing you could do as a workaround is to use 'match protocol rtp audio' on your class-map (match any) with dscp ef or some other combination you like. I've had to use this in places to fix voice quality issues where QoS wasn't being set correctly throughout the network and no one had accessto rest of network, etc or knew what was being done where.

----- Original Message ----
From: Keith Klevenski <keith.klevenski@rig.net>
To:Jason Aarons (US) <jason.aarons@us.didata.com>; cisco-voip@puck.nether.net
Sent: Thursday, October 26, 2006 11:32:22 AM
Subject: Re: [cisco-voip]Strict Priority Queuing over T1 for Voice



It turns outthe problem is indeed the
6608. It intermittently marks RTP packets with DSCP 0. This explains why I didn’t
see any drops on any interfaces because I falsely assumed the packets were
being marked with EF so if theywere dropped I should see them dropped in the high
priority queue. There were plenty of low priority queue drops, but I never
though the RTP stream would be part of those drops! We did look at sniffer
traces before, but didn’t look at every single packet to see how it was
marked. That is where we didn’t absolutely confirm that ALL packets were
being marked properly.

Oddly manually marking the packets on the
vlan interface where all the 6608 ports are doesn’t work either. You can
see the matches in the show policy-map int command and it looks like they are
getting marked with DSCP 46, but upon looking at the sniffer traces after the problem
continued the DSCP was still marked 0 for some packets just as it was before.

Going to open a TAC case to see what the
deal is with this bug. It is first found in CatOS, but fixed in a CallManager
release. That seems pretty strange to me and even stranger why a service
policy on the MSFC didn’t have any effect. I’m certain it was
configured properly. It is also odd that cisco says there is no known
workaround for the problem. Why can’t you just manuallymark the
packets? Well, apparently you can’t at least from the MSFC so I guess it’s
all related somehow.

Anyway, just an FYI for you all. We ended
bandaiding the problem by putting all UDP traffic sourced from the 6608 ports
in the high priority queue on the outbound interface of the T1 (the 7200) at
the datacenter to the office until we get it resolved. I will keep you
posted. ;)

Keith

From:
cisco-voip-bounces@puck.nether.net [mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Keith Klevenski

Sent: Wednesday, October 25, 2006
12:06 PM

To: Jason Aarons (US);
cisco-voip@puck.nether.net

Subject: Re: [cisco-voip] Strict
Priority Queuing over T1 for Voice

Thanks Jason.

After doing some more research I think I
may be hitting CSCsb49202:

6608 : Incorrect marking DSCPvalue to 0

It is first fixed in version 4.1(3)ES53
which is newer than SR3c which is what we’re running and is the latest
SR.

If this is the problem this actually makes
sense.. I’m not seeing any drops in the HIGH priority queue, but
plenty in the low queue when there is congestion on the T1. All of the
RTP packets I saw were marked EF, but I didn’t see if every single packet
in sequence was marked with EF. If some are being marked with DSCP 0 then
they would be dropped in the low priority queue which I see plenty of drops
there when there is congestion on the T1. And the fact that if I am on an
internal call (over the T1) or out a different gateway I do not see any
problems.

PQ has been working fine for us and works
fine with links over satellite for oil rigs. We only care about voice so
that’s why we use legacy PQ. Easy and straightforward.

Instead of upgrading to the latest ES
we’re going to mark all RTP packets with DSCP EF manually to make sure
they are marked properly. I will update the group with the results.

Thanks for all of the input!

Keith

From:
cisco-voip-bounces@puck.nether.net [mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of Jason Aarons (US)

Sent: Wednesday, October 25, 2006
11:54 AM

To: cisco-voip@puck.nether.net

Subject: Re: [cisco-voip] Strict
Priority Queuing over T1 for Voice

I would change the QoS match the cisco SRND
recommendations. Plus cisco can’t say your are not using their best
practices.

It seems to me you get more deatail in
show policy-map interface versus show queueing interface.

Aside from that look forshow queueing
interface, show interface and drops. You indicate none arefound, maybe verify
that with Ethereal/WireShark. The Smart Decode in Ethereal is pretty good.

From: Keith Klevenski
[mailto:keith.klevenski@rig.net]

Sent: Wednesday,October 25, 2006
10:36 AM

To: Jason Aarons (US);
cisco-voip@puck.nether.net

Subject: RE: [cisco-voip] Strict
Priority Queuing over T1for Voice

Just using plain old legacy priority
queuing:

priority-list 1 protocol ip high list 110

priority-list 1 protocol ip medium list
111

priority-list 1 protocol ip normal list
112

priority-list 1 default low

access-list 110 permit ip any any dscp ef

access-list 111 permit ip any any dscp
af31

access-list 111 permitip any any dscp
af21

access-list 111 permit ip any any dscp
af22

access-list 111 permit ip any any dscp
af23

access-list 111 permit ip any any dscp cs3

access-list 112 permit ip any any dscp
af11

access-list 112 permit ip any any dscp
af12

access-list 112 permit ip any any dscp
af13

From: cisco-voip-bounces@puck.nether.net
[mailto:cisco-voip-bounces@puck.nether.net] On
Behalf OfJason Aarons (US)

Sent: Wednesday, October 25, 2006
8:08 AM

To: cisco-voip@puck.nether.net

Subject: Re: [cisco-voip] Strict
Priority Queuing over T1 for Voice

Auto qos voip does a great job of
generating the policy-maps. What does your policy look like?.

What does your show policy-map interface
x/y/z show when the problem occurs? Are LLQ matched packets being
discarded?Maybe the service-policy is not protecting the voice like you think.

From:
cisco-voip-bounces@puck.nether.net [mailto:cisco-voip-bounces@puck.nether.net] On Behalf Of KeithKlevenski

Sent: Tuesday, October 24, 2006
5:43 PM

To: cisco-voip@puck.nether.net

Subject: [cisco-voip] Strict
Priority Queuing over T1 for Voice

Hi all,

I’ve been battling a problem that seemed to have
surfaced recently as wehave added more users to the office. We have an
office of about 35 users now with a point to point T1 (HDLC) between this
office and the datacenter where the CallManager’s (running 4.1.3) and
PSTN gateway (6608 blade) are. Basically this:

PSTN

|

6608/6509 (datacenter) (12.1(11b)E14 on msfc and 6.4(10) on
sup)

| trunk

7200 (datacenter) 12.3(6a)

|

T1

|

2811 (remote office) 12.3(11)T7

| trunk

3750 (remote office)

|

Phones

We have strict priority queuing configured on the T1
interfaces (and the FastEthernet interfaces although I don’t think this
is necessary) with DSCP EF traffic in the high queue and signaling (AF31/21/22/23
and CS3) the medium queue and other traffic in the normal and low queues.

The problem is when the T1 is saturated we start seeing high
jitter (1-2 seconds) and rxlost packets on the phones and voice quality of
course suffers. The thing is I see NO dropped packets on any interface
from the PSTN to the switch port the phone is connected to other than the call
statistics on the phone. This confuses me greatly unless the phone itself
is dropping them… it would seem that if the phone determines there are
missing packets (rxlost) they would show up on some interface somewhere but
they do not. Sniffer traces at the phone show the missing packets.
Sniffer traces at the trunk at the remote office show missingpackets, but
traces on the trunk in the datacenter (from the 7200 to 6509) show all packets
accounted for. Seems like the T1 is the issue. There are 24 line
code violations on the T1 in the last 24 hours, but no slips. The problem
only seems to occur when there is high traffic on the T1.We also noticed
no other traffic is marked EF and the RTP stream is EF at the datacenter and at
the office so that isn’t being remarked or anything. But there are
no input queue drops on the 2811 (which I wouldn’t expect since the RTP stream
should be CEF switched anyway). If the input interface cannot handle the
amount of traffic and the traffic is all CEF switched where would you see input
drops? There are a handful of buffer failures, but nothing that would
account for the number of complaints I’ve been getting lately.

I’ve connected my phone directly to the 2811 and
bypassed the 3750 with the same results.

I also noticed that the rxsize fluctuates between 10ms and 20ms also.
Shouldn’t that always be 20ms? I thought that was odd.

I guess my main question is when using strict priority
queuing shouldn’t the voice packets ALWAYS get sent first no matter
what?? I want to believe they are since there are NO drops in the high
priority queue outbound from the datacenter. What is the best way to do
QoS for voice over a T1 when you only care about the voice? Seems like
strict priority queuing is the best way to assure voice traffic is sent
first. We don’t really care about starving out any other
traffic. If packets are being dropped shouldn’t I see them
somewhere other than on the phone?

Any input on this problem is welcome!

Keith

Disclaimer:

This e-mail communication and any attachments may contain confidential and privileged information and is for use by the designated addressee(s) named above only. If you are not the intended addressee, you are hereby notified that you have received this communication in error and that any use or reproduction of this email or its contents is strictly prohibited and may be unlawful.. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you.

Disclaimer:

This e-mail communication and any attachments may contain confidential and privileged information and is for use by the designated addressee(s) named above only. If you are not the intended addressee, you are hereby notified that you have received this communication in error and that any use or reproduction of this email or its contents is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by replying to this message and deleting it from your computer. Thank you.

________________________________________
_______
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip