Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing
table is going nuts with Bell advertising a lot of routes they shouldn't be
Bell leaked a full table. To add to the fun, it seems that TATA took
the full table and releaked it.
Outages mailing list is reporting that Tata is having problems in Montreal
affecting 'many routers'.......maybe this is related?
Chris
That's what it looked like. We are connected (@ McGill University) to RISQ
and a lot of our routes started to go via RISQ->Bell instead of
RISQ->CANET/Canarie
I am a transit customer of both TATA and Bell Canada. We saw route
churn and heavy packet loss via both Bell and TATA beginning at
approximately 13:25 ET. It took us some time to assess the situation
and deactivate both our TATA and Bell BGP sessions.
It also took over 10 minutes for my BGP withdraws to propagate from
Bell to their neighbors, including Level3. I would guess Bell
Canada's routers all have very busy CPU.
No information from either NOC yet.
I saw the same sort of behaviour from TATA. I shut my peer with them,
yet TATA was still advertising to Level3 a good 15min after !??!
route-views> traceroute 199.212.134.1
Type escape sequence to abort.
Tracing the route to 199.212.134.1
1 vl-51.uonet1-gw.uoregon.edu (128.223.51.2) [AS 3582] 280 msec 364
msec 280 msec
2 3.xe-1-3-0.uonet10-gw.uoregon.edu (128.223.3.10) [AS 3582] 252 msec
272 msec 272 msec
3 eugn-car1-gw.nero.net (207.98.68.177) [AS 3701] 272 msec 268 msec
244 msec
4 eugn-core1-gw.nero.net (207.98.64.161) [AS 3701] 260 msec 272 msec
280 msec
5 ge-6-21.car1.Sacramento1.Level3.net (4.53.200.1) [AS 3356] 296 msec
260 msec 312 msec
6 ae-11-11.car2.Sacramento1.Level3.net (4.69.132.150) [AS 3356] [MPLS:
Label 83 Exp 0] 304 msec 328 msec 376 msec
7 ae-4-4.ebr2.SanJose1.Level3.net (4.69.132.158) [AS 3356] [MPLS:
Label 1123 Exp 0] 328 msec 252 msec 308 msec
8 ae-3-3.ebr1.Denver1.Level3.net (4.69.132.58) [AS 3356] [MPLS: Label
1191 Exp 0] 332 msec 240 msec 232 msec
9 ae-1-100.ebr2.Denver1.Level3.net (4.69.151.182) [AS 3356] [MPLS:
Label 1189 Exp 0] 252 msec 56 msec 40 msec
10 ae-3-3.ebr1.Chicago2.Level3.net (4.69.132.62) [AS 3356] [MPLS: Label
1186 Exp 0] 72 msec 72 msec 248 msec
11 ae-1-51.edge3.Chicago3.Level3.net (4.69.138.136) [AS 3356] 200 msec
344 msec 200 msec
12 tata-level3.chicago3.level3.net (4.68.110.194) [AS 3356] 204 msec
200 msec 216 msec
13 * * *
14 if-4-2.tcore1.MTT-Montreal.as6453.net (64.86.31.17) [AS 6453] 80 msec
if-13-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.32.2) [AS 6453]
[MPLS: Label 3208 Exp 0] 84 msec 228 msec
15 * * *
16 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453]
[MPLS: Label 1678 Exp 0] 260 msec 244 msec 236 msec
17 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 272 msec
if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453]
[MPLS: Label 3208 Exp 0] 272 msec
if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 260 msec
18 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453]
[MPLS: Label 3208 Exp 0] 224 msec 208 msec 204 msec
19 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453]
[MPLS: Label 1678 Exp 0] 260 msec 344 msec 300 msec
20 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 256 msec
if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453]
[MPLS: Label 1678 Exp 0] 240 msec
if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 268 msec
21 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453]
[MPLS: Label 3208 Exp 0] 232 msec
if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 248 msec
if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453]
[MPLS: Label 3208 Exp 0] 256 msec
22 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453]
[MPLS: Label 3208 Exp 0] 232 msec 240 msec 240 msec
23 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453]
[MPLS: Label 1678 Exp 0] 244 msec * *
24 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 284
msec 208 msec 220 msec
25 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 204
msec 256 msec
if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453]
[MPLS: Label 3208 Exp 0] 204 msec
26 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453]
[MPLS: Label 3208 Exp 0] 224 msec 424 msec 208 msec
27 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453]
[MPLS: Label 1678 Exp 0] 220 msec * *
28 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453]
[MPLS: Label 1678 Exp 0] 136 msec 128 msec
if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 116 msec
29 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453]
[MPLS: Label 3208 Exp 0] 128 msec 124 msec 116 msec
30 * 152 msec 120 msec
route-views>
Hi,
.-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius
Jahandarie wrote:
Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing
table is going nuts with Bell advertising a lot of routes they shouldn't beBell leaked a full table. To add to the fun, it seems that TATA took
the full table and releaked it.
A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the
cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems
to have happened is that they leaked routes learned from VIDEOTRON to Bell.
Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc)
started to leak a 'full table', or at least a significant chunk of it to
its provider Bell AS577.
Bell propagated that to it's peers. Tata was one of the ones that
accepted all of that.
I can see that Bell propagated at least 74,109 prefixes learned from
AS46618 to Tata. Tata selected 70,160 of those routes.
Cheers,
Andree
Taking a cursory look at the data processed by my now (very old) leak
auditing tool, this should give you an idea of the size of the impact...
bgp=# select count(*) from leakinfo where aprox_time::date='2012-08-08';
count
Could this be causing issue with XO in east coast ?
Thanks for the info, looks like Bell needs to put some filtering on their
customer links!
Things seem to have calmed down now but I'm still watching my prefixes
received from my upstream provider:
while true; do snmpget -v2c -c <community>
<router> 1.3.6.1.4.1.9.9.187.1.2.4.1.1.<ip of bgp peer>.1.128; sleep 1; done
I also monitor all my prefixes and bgp messages in Cacti
I remember when AS577 had those... 
CPU's were pegged for a customer of mine in California. tracked it
down to 2 events that went down at that time with a large message
volume.
1) Peering between GLBX and Level3 dopped somewhere, causing many
prefixes to shift away from L3 paths.
2) Some IPv6 prefixes were aggressively bouncing from HE starting at
11:25. Can't trace it any further upstream for HE unfortunately.
This cause lots of CPU churn on one of my customers networks
May or may not be related.
Further analysis shows that there were actually 107,409 prefixes
affected of 14,391 unique origin ASn's.
Interested if your prefixes was affected?
I've uploaded a list of prefixes and ASn's that were leaked here:
http://www.bgpmon.net/bell-leak.txt
Cheers,
Andree
.-- My secret spy satellite informs me that at 12-08-08 12:50 PM Andree
Toonk wrote:
Ouch!! That's a lot! What do you think the outcome of this will be? What
do you think that Bell did/will do (hopefully) to fix this so it doesn't
happen again?
CPU's were pegged for a customer of mine in California. tracked it
down to 2 events that went down at that time with a large message
volume.1) Peering between GLBX and Level3 dopped somewhere, causing many
prefixes to shift away from L3 paths.
FYI: We saw significant route churn on GBLX (AS3549) as well as on Tata.
We have been advised that TATA/6453 is back to normal, and
re-activated our BGP to them. Everything seems okay on this front.
No update from Bell Canada yet.
On 8 August 2012 16:10, Zachary McGibbon
Thanks for the info, looks like Bell needs to put some filtering on their
customer links!I remember when AS577 had those...
We actually have asked Bell Canada not to filter routes from us, and
use prefix-limit only, because they were not able to build a
prefix-list for us if we have any downstream customer ASNs, which we
do. :-/
If someone at Bell Canada is reading and cared to contact me off-list,
I would sure love to get my own route filtering fixed. I have had
little success through the normal channels.