I am hoping someone can shed some light on an interesting problem we are
having -
When we set up a customer for MLPPP, things tend to go well for a period
of time. Then - all of a sudden - we will begin to have problems with
our multilink bundles (generally only one at a time) and the only fix is
to reload our 7513. This problem happens on both of our 7513 routers
from time-to-time. Once we reload - the problem will stay gone for as
long as several months, or in the last case only about 12 hours.
Once we see the problem, it is apparent only in one direction. For
example the customer can push the full capacity of their circuits to us,
but they cannot pull anything above about 300k back to them on a two-T1
bundle. This is the same every single time we have the problem.
We have changed multilink bundles, tried different types of switching
and route caching, turning on and off fragmentation - the only thing
that solves the problem is reloading the entire router. We can pull the
T1s from the multilink bundle and each individual T1 works great. No
line errors, no crc errors - nothing. No errors are apparent while in
MLPPP mode either. No throttles or anything similar.
We have had this problem in the past and it was recommended that we
upgrade the code on our 7513s. We are currently running version
12.2(13)T5 on both our 7513 as well as the customers router. Upgrading
the code did not solve the problem. I have been unable to locate a Cisco
bug defining this type of problem for any version of their code.
This particular customer's T1s are both terminated on the same VIP (we
are running DMLP) but are terminated on separate PAs and hence separate
CT3s. We have noticed the problem even with T1 bundles on the exact same
PA and CT3. We are not doing multi-chassis DMLP. dCEF is enable on both
routers, however the problem remains the same even after disable dCEF.
Here are configs and router info:
interface Multilink6
description Eastgate Mall (s2/1/0/8:0 and s2/0/0/12:0) [20291]
ip address 207.158.1.133 255.255.255.252
no cdp enable
ppp multilink
ppp multilink interleave
multilink-group 6
interface Serial2/0/0/12:0
description Eastgate Mall #2
no ip address
encapsulation ppp
no fair-queue
ppp multilink
multilink-group 6
interface Serial2/1/0/8:0
description Eastgate Mall #1
no ip address
encapsulation ppp
no fair-queue
ppp multilink
multilink-group 6
AR04#sh diag 2
Slot 2:
Physical slot 2, ~physical slot 0xD, logical slot 2, CBus 0
Microcode Status 0x4
Master Enable, LED, WCS Loaded
Board is analyzed
Pending I/O Status: None
EEPROM format version 1
VIP2 R5K controller, HW rev 2.02, board revision D0
Serial number: 17953368 Part number: 73-2167-05
Test history: 0x00 RMA number: 00-00-00
Flags: cisco 7000 board; 7500 compatible
EEPROM contents (hex):
0x20: 01 1E 02 02 01 11 F2 58 49 08 77 05 00 00 00 00
0x30: 68 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
Slot database information:
Flags: 0x4 Insertion time: 0x41C8 (1d08h ago)
Controller Memory Size: 32 MBytes DRAM, 4096 KBytes SRAM
PA Bay 0 Information:
CT3 single wide PA, 1 port
EEPROM format version 1
HW rev 1.00, Board revision A0
Serial number: 17814822 Part number: 73-3037-01
PA Bay 1 Information:
CT3 single wide PA, 1 port
EEPROM format version 1
HW rev 1.00, Board revision A0
Serial number: 09725065 Part number: 73-3037-01
--Boot log begin--
Cisco Internetwork Operating System Software
IOS (tm) VIP Software (SVIP-DW-M), Version 12.2(13)T5, RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2003 by cisco Systems, Inc.
Compiled Wed 28-May-03 21:57 by nmasa
Image text-base: 0x60010930, data-base: 0x604C0000
AR04#sh ver
Cisco Internetwork Operating System Software
IOS (tm) RSP Software (RSP-JSV-M), Version 12.2(13)T5, RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2003 by cisco Systems, Inc.
Compiled Wed 28-May-03 22:00 by nmasa
Image text-base: 0x60010948, data-base: 0x61F0A000
ROM: System Bootstrap, Version 11.1(8)CA1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
AR04 uptime is 1 day, 8 hours, 24 minutes
System returned to ROM by reload at 09:45:48 UTC Tue Dec 23 2003
System image file is "slot0:rsp-jsv-mz.122-13.T5.bin"
cisco RSP4 (R5000) processor with 262144K/2072K bytes of memory.
R5000 CPU at 200Mhz, Implementation 35, Rev 2.1, 512KB L2 Cache
Last reset from power-on
G.703/E1 software, Version 1.0.
G.703/JT2 software, Version 1.0.
X.25 software, Version 3.0.0.
SuperLAT software (copyright 1990 by Meridian Technology Corp).
Bridging software.
TN3270 Emulation software.
Primary Rate ISDN software, Version 1.1.
Chassis Interface.
3 VIP2 R5K controllers (2 FastEthernet)(6 Channelized T3).
2 FastEthernet/IEEE 802.3 interface(s)
168 Serial network interface(s)
6 Channelized T3 port(s)
123K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
8192K bytes of Flash internal SIMM (Sector size 256K).
Slave in slot 7 is running Cisco Internetwork Operating System Software
IOS (tm) RSP Software (RSP-DW-M), Version 12.2(13)T5, RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2003 by cisco Systems, Inc.
Compiled Wed 28-May-03 22:33 by nmasa
Slave: Loaded from system
Slave: cisco RSP4 (R5000) processor with 262144K bytes of memory.
Configuration register is 0x2102
Any help would be greatly appreciated.