NTT communications horrible routing, unresponsive NOC

Hi all,

I've been trying to get this issue resolved for the entire day now, but NTT
has been pretty unreceptive here.

We're announcing a large prefix for a client across our network, and we
discovered some insanely high latency.

After tracking down the issue, we determined it to be something wrong with
NTT at their Seattle location. We anycast this prefix, but no matter where
in the world traffic is originating from, it's going to Seattle and then to
Atlanta. Example: Rotterdam in the Netherlands routes from Europe -> east
coast -> west coast Seattle -> los angeles -> atlanta. The gist of it is
that something is seriously messed up at NTT in Seattle.

We contacted our transit provider to try and carry the issue upstream, and
what they told us was

Sorry for delay, I've asked NTT to clear the more specific for this one as
The problem seems to be a bug on the NTT side which keeps stale routes in
the routing table for more specifics ( at random ).
If you have more routes affected please notify us of the routes and I will
ask them to clear the routing table for these routes.
NTT is working on this with their vendor to get this resolved as soon as

I had spoken to a sales rep for NTT a few weeks prior, and they assured me
that the NOC was top notch, and that all routes were redundant, and they
guaranteed less than 50ms in the US, and all kinds of marketing. However,
it looks like it's all marketing - for this entire day this router has been
causing tons of issues for our clients.

No-exporting it to NTT does not even solve the problem, as NTT's router in
Seattle apparently just decides to keep random small prefixes in it,
causing traffic to go there.

At a loss as to what to do now, since their NOC isn't receptive. Anyone
have someone I can contact off-list to get this issue resolved? It's
especially frustrating because the problem absolutely cannot be resolved on
our end, even with a no-export since NTT is keeping the routes in their

which route and did you look at their lookingglass during the period of
time when things were/are bad to see what the LG saw?

shouting into the wind doesn't really help, information though could be

It would be helpful to show how you came to these conclusions about
prefix caching and routing issues etc (share the data you gathered).

If you are an NTT customer (it's sounds a bit like you're a customer
of an NTT customer, it's not clear) then this should be account
management 101, you should have an escalations process/matrix, kick
off that process now.

There are NTT dudes on this list so probably they will reach out to
you off list, if not post back requesting an NTT contact.



Could you send me some more details in private about who you spoke with including email addresses and ticket numbers with our NOC if any?

I'd like to understand what happened here as this is an uncharacteristic outcome. I know there was planned work in Seattle last night for a software upgrade combined with hardware work, but an operational issue like this should have been resolved.

I'd like to understand the timeline and what broke down if anything.


Jared Mauch