It appears there have been a large number of routing leaks from 6453 today based on my detection scripts that have been running.
(shameless plug for http://puck.nether.net/bgp/leakinfo.cgi)
A quick report of the data show (for today so far) a few thousand of leaks more than is normal for a day like today. I included a snapshot of yesterday below as well.
I've included a more detailed report of the prefixes observed involved here:
http://puck.nether.net/~jared/tata-leak-20110224.txt
This seems to be a somewhat common event for 6453, loking through the history of data available, another event happened on 2011-01-28 as well.
I'm interested in what best operational practices people have employed to help avoid the leaks seen here so I can document them for others to learn to prevent this from happening again.
- Jared
bgp=# select count(blame_asn),blame_asn,asn_responsible from leakinfo where aprox_time::date = '2011-02-24' group by blame_asn,asn_responsible order by 1 desc;
count | blame_asn | asn_responsible
Update:
I have had a source ask me to post the following:
-- snip --
The problem with route leaking was caused by specific routing platform resulting in some peer routes not being properly tagged.
We are deploying additional measures to prevent this from happening in the future
-- snip --
- Jared
Hopefully someone learned a lesson about BGP community design, and how
it should fail safe by NOT leaking if you accidentally fail to tag a
route. Always require a positive match on a route to advertise to peers,
not the absence of a negative match.
Yes, very scary actually....
Human error is unavoidable - it's going to happen at times - BUT....
In our communities design, there has been times where we have missed a tag
on an inbound customer for example. It scares the crap out of me to think
that something like that simple mistake could cause route leakage.
Thankfully, anytime it has happened it would caught pretty quickly and fixed
- in the meantime the routes simply didn't leave our network (the way it
should be).
Obviously the scales are different between someone like ourselves and that
of TATA - but the principles and common sense remain.
Paul
Would love a pm on the platform in question