AS4788 Telecom Malaysia major route leak?

I see tons of bogus routes show up with AS4788 in the path, and at
least AS3549 is acceping them.

E.g. for the RIPE NCC (193.0.0.0/21):

                    [BGP/170] 00:20:29, MED 1000, localpref 150
                      AS path: 3549 4788 12859 3333 I, validation-state: valid
                    > to 64.210.69.85 via xe-1/1/0.0

Tore

It appears that AS3549 propagated the (almost?) full routing table leak
to its peers, where in lots of instances max prefix kicked in.

This has global impact, lots of alerts on the SQA collector page http://sqa.ring.nlnog.net/

Kind regards,

Job

* Tore Anderson <tore@fud.no> [2015-06-12 11:12]:

I see tons of bogus routes show up with AS4788 in the path, and at
least AS3549 is acceping them.

E.g. for the RIPE NCC (193.0.0.0/21):

                    [BGP/170] 00:20:29, MED 1000, localpref 150
                      AS path: 3549 4788 12859 3333 I, validation-state: valid
                    > to 64.210.69.85 via xe-1/1/0.0

I confirm, something is going on:


Regards

Sebastian

I'm reaching out to them now.

a message of 10 lines which said:

I see tons of bogus routes show up with AS4788 in the path, and at
least AS3549 is acceping them.

E.g. for the RIPE NCC (193.0.0.0/21):

                    [BGP/170] 00:20:29, MED 1000, localpref 150
                      AS path: 3549 4788 12859 3333 I, validation-state: valid

Unlike most BGP leaks, they kept the proper origin, so validation by ROA was
useless :frowning:

It *looks* like GBLX stopped accepting the leak.

Regards,
Marty Strong

These aren't just leaks - they're more specifics of what's normally advertised, but keeping the proper origin. Hard to see how that could be accidental...

Chris

* Marty Strong via NANOG <nanog@nanog.org>

It *looks* like GBLX stopped accepting the leak.

If so, it's a partial fix at best, I still see plenty of leaked routes,
both via 3356 and 3549, e.g.:

tore@cr1-osl3> show route 195.24.168.98 all
Jun 12 12:03:54 +0200

inet.0: 544405 destinations, 1591203 routes (543086 active, 3 holddown, 526626 hidden)
+ = Active Route, - = Last Active, * = Both

195.24.160.0/19 *[BGP/170] 00:03:59, MED 2000, localpref 50, from 87.238.63.5
                      AS path: 3356 3549 4788 6939 39648 I, validation-state: unverified
                    > to 87.238.63.56 via ae0.0
                    [BGP/170] 00:05:24, MED 0, localpref 50, from 87.238.63.2
                      AS path: 3356 3549 4788 6939 39648 I, validation-state: unverified
                    > to 87.238.63.56 via ae0.0
                    [BGP ] 01:16:00, MED 25245, localpref 100
                      AS path: 3549 4788 6939 39648 I, validation-state: unverified
                    > to 64.210.69.85 via xe-1/1/0.0

It seems to have started around 08:47 UTC, that's when I got my first
alarm from ring-sqa at least.

Tore

Hi Marty,

Noted. We are still checking this issue.

Regards,

LEE BON SHENG | leebonsheng@tm.com.my | ipmc_ipcore@tm.com.my
NOC2 IPCORE, ISP Network Management, Telekom Malaysia, AS4788
TOLLFREE: 1-800-88-2646 (Opt 4) / International: +603-22466646 (Opt 4)

We're committed to perform.
We strive to excel.
We deliver THE BEST!

Regards,
Marty Strong

I think you just saw it flapping. :slight_smile:
That's what I've been seeing since ~ 0845 UTC :frowning:

I disagree. Since 08:44 UTC up until now (10:15) the DFZ has been a
radio-active wasteland with hordes of unwelcome announcements.

Kind regards,

Job

Yes, you’re right, I was too trigger happy :frowning:

Regards,
Marty Strong

Nope. Churn is ongoing, nothing has been fixed.
Global outage began 08:44 UTC and is still ongoing.

It's been so long people have now had time to come up with things like
"33.333%".

Also, possible explanation for why nobody's fixing it:
https://twitter.com/TMCorp/status/609167065300271104 :slight_smile:

/M

OK, as of now (~ 10:40) UTC things look normalised.

Kind regards,

Job

Just got off the phone, I think things may be in hand, now.

Still on hold with Level3, but some of my sites are clearing up.

Chris

* Roland Dobbins <rdobbins@arbor.net> [2015-06-12 12:57]:

> OK, as of now (~ 10:40) UTC things look normalised.

Just got off the phone, I think things may be in hand, now.

Still seeing a lot more updates than usual:


Is this just folks turning up their sessions again? Looks a bit
much...

Regards

Sebastian

These aren't just leaks - they're more specifics of what's normally advertised, but keeping the proper origin. Hard to see how that could be accidental...

Having looked further - the examples of these I was looking at
(advertisements from AS34556 & AS17709) were being advertised before the
leak, but only with limited visibility. The leak caused them to be
(intermittently) globally visible.

Tin foil hat off - can all just be accidental.

Chris

Yes, I suspect tons of 3356 / 3549 customers shut down their BGP
sessions waiting for the storm to blow over. I expect more churn then
usual the next 6 ~ 12 hours, due to customers slowly turning session
back on.

Kind regards,

Job

First news:

http://www.xgn.nl/nieuws/69593/grote-internetstoring-in-europa-problemen-door-route-leak