Anyone from Verizon want to comment on a possible route leak on the 25th 10:30 PM EDT? Saw route table jump up to 737629 routes last night from 701.
Marcus Josephson
mjosephson@inap.com<mailto:mjosephson@inap.com> * www.inap.com<http://www.inap.com>
INAP (r)
connectivity | colocation | managed hosting | cloud
One Ravinia Drive * Suite 1300 * Atlanta * GA * 30346
Hello,
Do you mean this one ?
https://dyn.com/blog/large-bgp-leak-by-google-disrupts-internet-in-japan/
https://bgpmon.net/bgp-leak-causing-internet-outages-in-japan-and-beyond/
https://mobile.twitter.com/bgpmon/status/901565301048803329
Best regards.
Damn you Google.. yup. Thanks for links.
-Marcus
I am not sure it is fair to say "damn you Google", because accidents
happen (be it through human error or software defects). All of us have
entered commands at some point and subsequently
https://media.giphy.com/media/bI5BEfwbdVPcA/giphy.gif
Software defects are a real risk as well: Major BGP implementations have
had bugs which caused NO_EXPORT functionality to malfunction, or bugs
where random pieces of your configuration are ignored (for instance the
part of the configuration which contains a prefix-filter). Allegedly the
famous AS 7007 incident was also caused by a software defect.
The key concept here is that, since we know accidents an happen, we
ought to look out for each other and impose multiple layers of
protection for our mutual benefit.
Mechanisms that come to mind are maximum prefix limits and coarse
as-path filters. At NANOG67 I described a few of these tricks
https://www.youtube.com/watch?v=CSLpWBrHy10 /
https://www.nanog.org/sites/default/files/Snijders_Everyday_Practical_Bgp.pdf
To further reduce the risk surface, it may be good to ask your BGP
vendor for specific behaviour applicable to EBGP sessions: ask for RFC
8212 support. RFC 8212 defines that when you configure an EBGP session
without any routing policy associated with that neighbor, no routes will
be exchanged until it is explicitly instructed to do so.
Finally, it may be worthwhile exploring if we can standardize and
promote maximum prefix limits applied on the the _sending_ side. This
way you protect your neighbor (and the Internet at large) by
self-destructing when you inadvertently announce more than what you'd
expect to announce. BIRD has this functionality
http://bird.network.cz/?get_doc&f=bird-3.html#proto-export-limit
however I am not aware of other implementations. Feedback welcome!
Kind regards,
Job
Having just dug up the reference for some strange reason...
Back at NANOG38 (2006) Tom Scholl mentioned in a talk on max prefix:
"Perhaps maximum-prefix outbound?
(Suggested by Eric Bell years ago)"
https://www.nanog.org/meetings/nanog38/presentations/scholl-maxpfx.pdf
Notably Juniper does now have prefix-export-limit, but only for
readvertisement into IS-IS or OSPF:
https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/prefix-export-limit-edit-protocols-isis.html
A public post-mortem would be highly appreciated (from all parties).
Damn you Google.. yup. Thanks for links.
A public post-mortem would be highly appreciated (from all parties).
there has been more press hysteria on this than actual packet droppage.
goog fat fingered or otherwise misannounced a numer of large consumer
isp's prefixes. the leak was for aybe 20 minutes.
almost no one over here noticed.
but the press, isoc, ... said "japan knocked off the internet." take
that as a calibration of the press, isoc, ...
randy
Good use-case for https://datatracker.ietf.org/doc/draft-ietf-grow-bmp-adj-rib-out and snapshot auditing before and after changes. Leak didn't last long but it could have been caught within milliseconds verses minutes via oh sh** alarms.
--Tim
>> Damn you Google.. yup. Thanks for links.
> A public post-mortem would be highly appreciated (from all parties).
there has been more press hysteria on this than actual packet droppage.
goog fat fingered or otherwise misannounced a numer of large consumer
isp's prefixes. the leak was for aybe 20 minutes.
almost no one over here noticed.
but the press, isoc, ... said "japan knocked off the internet." take
that as a calibration of the press, isoc, ...
randy
Good use-case for
RFC 8671 - Support for Adj-RIB-Out in the BGP Monitoring Protocol (BMP) and
snapshot auditing before and after changes. Leak didn't last long but
it could have been caught within milliseconds verses minutes via oh
sh** alarms.
[ i happen to like bmp, but ... ]
if the sender did not have the automation or the mops to not leak in the
first place, how well will they apply post hoc detection and repair?
if the receiver did not filter, and an tier-1 as-path filter would have
sufficed in this case, how well do you think they will be at applying
post hoc detection and repair?
this was an easily preventable ops failure. but what we will do is go
to idr and grow and invent 42 more hacks, kinda like ipv6 transition
mechanisms. </snark>
randy