On Thursday March 26th 2015 at 12:18 UTC (and on-going) we are seeing more specifics on one of our prefixes. Anyone else seeing similar or is it just us?
198.98.180.0/23 4795 4795 4761 9304 40633 18978 4436 29889
198.98.182.0/23 4795 4795 4761 9304 40633 18978 4436 29889
On Thursday March 26th 2015 at 12:18 UTC (and on-going) we are seeing more
specifics on one of our prefixes. Anyone else seeing similar or is it just
us?
is your AS in the path below? (what is your AS so folk can check for
your prefixes/customer-prefixes and attempt to help?)
Sorry, we're 29889.
We just received a similar alert from bgpmon - part of 108.168.0.0/17 is being advertised as /20's - although we're still listed as the origin. We are 40788.
108.168.64.0/20 4795 4795 4761 9304 40633 18978 6939 40788
108.168.80.0/20 4795 4795 4761 9304 40633 18978 6939 40788
108.168.96.0/20 4795 4795 4761 9304 40633 18978 6939 40788
108.168.112.0/20 4795 4795 4761 9304 40633 18978 6939 40788
ok, and it looks like the path you clipped is:
198.98.180.0/23 4795 4795 4761 9304 40633 18978 4436 29889
possibly LAIX is passing along your /24 you didn't mean them to pass on?
We just received a similar alert from bgpmon - part of 108.168.0.0/17 is being advertised as /20's - although we're still listed as the origin. We are 40788.
108.168.64.0/20 4795 4795 4761 9304 40633 18978 6939 40788
108.168.80.0/20 4795 4795 4761 9304 40633 18978 6939 40788
108.168.96.0/20 4795 4795 4761 9304 40633 18978 6939 40788
108.168.112.0/20 4795 4795 4761 9304 40633 18978 6939 40788
common point looks like LAIX ? their routeserver go crazy perhaps? or
did they change in/out prefix management information?
All,
Info gathered off-list indicates this may be a couple of issues in our case - possible routing leak by 18978 (check your tables!) and more specifics on our prefixes from 4795 that we couldn't see before the leak hence the apparent hijack.
Hi,
On Thursday March 26th 2015 at 12:18 UTC (and on-going) we are seeing more
specifics on one of our prefixes. Anyone else seeing similar or is it just
us?198.98.180.0/23 4795 4795 4761 9304 40633 18978 4436 29889
198.98.182.0/23 4795 4795 4761 9304 40633 18978 4436 29889
We (as3215) are seeing almost the same path with 40633 18978 3257
3215, for some quite a lot of prefixes.
Some alerts from bgpmon:
193.251.32.0/20 271 6939 40633 18978 3257 3215
193.251.32.0/20 271 6939 40633 18978 3257 3215
We are not directly connected to 3257. Looks like 18978 deaggregated
to /20 and reannounced to 40633 (LAIX).
Rgds,
pierre
Same here. These Indosat guys can't seem to catch a break =/
We are AS 10326 130.215.0.0/16 and I just received a BGPmon alert as
well:
130.215.160.0/20 4795 4795 4761 9304 40633 18978 4436 10326
130.215.176.0/20 4795 4795 4761 9304 40633 18978 4436 10326
Hi Randy,
Assuming that your prefix is 198.98.180.0/22 (AS29889 - FSNET-1 - Fast
Serv Networks, LLC) none of the mentioned more specifics are currently
seen from the RIPE NCC's RIS network, see the Looking Glass widget:
https://stat.ripe.net/198.98.180.0/23#tabId=routing
https://stat.ripe.net/198.98.182.0/23#tabId=at-a-glance
though there has been some BGP activity going on since 11:49:42, see the
BGPlay and BGP Update Activity widget. In both cases the originating ASN
was AS29889.
Cheers,
Christian
Hi List,
this morning our BGPmon system picked up many new more specific
announcements by a variety of Origin ASns, the interesting part is that
the majority of them were classified as BGP Man In The middle attacks
(MITM).
A typical alert would look like:
+1
The summary below aligns with our analysis as well.
We've reached out to AS18978 to determine the status of the leak but at this time we're not seeing any operational impact.
+2, after the morning coffee sunk in and helpful off list replies I can finally see it's probably not INDOSAT involved at all.
FYI, the more specifics are still active:
2015-03-26 13:56:11 Update AS4795 ID 198.98.180.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active
2015-03-26 13:56:11 Update AS4795 ID 198.98.182.0/23 4795 4795 4761 9304 40633 18978 6939 29889 Active
This should be resolved from AS18978. If you experience anything else please let me know and I will get it addressed immediately.
Regards,
Nick Rose
CTO @ Enzu Inc.
Several people asked me off list for more details, here is what I have regarding it.
This morning a tier2 isp that connects to our network made an error in their router configuration causing the route leakage. The issue has been addressed and we will be performing a full post mortem to ensure this does not happen again.
While investigating the issue we did find that the noction appliance stopped advertising the no export community string with its advertisements which is why certain prefixes were also seen.
Regards,
Nick Rose
CTO @ Enzu Inc.
Wouldn't it be a BCP to set no-export from the Noction device too?
Sure, but even that might not always prevent the fake paths from leaking
to your eBGP neighbors. For instance, not too long ago there was this
bug:
"Routes learned with the no-export community from an iBGP neighbor
are being advertised to eBGP neighbors. This may occur on Cisco ASR
9000 Series Aggregation Services Routers." (don't remember BugID)
In other words: it can happen to the best of us.
You should not lie to yourself by inserting fake more-specific paths
into routing tables. The moment your lies somehow manage to escape into
the default-free-zone you are taking other businesses down. Whether the
leak is caused by a bug in the router's software or human error,
destroying other people's online presence is far beyond acceptable.
If the same leak would've happened /without/ the fake more-specifics,
it'd still be an issue, but the collateral damage would have been
dampened. The leaked paths would have to compete with the normal paths
and best-path selectors like as-path length apply.
Using software to insert fake more-specific paths into your routing
domain should be discouraged and frowned upon.
Kind regards,
Job
Your upstream could also re-write any BGP communities you attach to your BGP updates; so unless co-ordinated, there is no real guarantee a NO_EXPORT community will be maintained/honoured within your upstream's network.
Mark.
I guess AS18978 didn't learn from their mistake. Got a slew of identical bgpmon alerts for withdrawals and more specifics within the last 30 minutes. Worse than last time. Some still active, like:
update time (UTC) Update Type Probe ASn Probe Location Prefix AS path Cleared Duration
2015-03-26 12:18:41 Update AS4795 ID 198.98.180.0/23 4795 4795 4761 9304 40633 18978 4436 29889 Active