I've got a problem where AS20115 continues to announce prefixes after BGP neighbors were shutdown. They claim it's a wedged BGP process but aren't in any hurry to fix it outside of a maintenance window.
I'm at a loss of what else I can do. They admit the problem but won't take action saying it needs to wait for a maintenance window. Am I out of line insisting that's an unacceptable response to a problem that results in prefix/traffic hijacking?
~Seth
I've got a problem where AS20115 continues to announce prefixes after BGP
neighbors were shutdown. They claim it's a wedged BGP process but aren't in
any hurry to fix it outside of a maintenance window.
If they weren't lying to you, they'd fix it now. That's not the kind
of problem that waits.
Thing is: they lied to you. Long ago they "helpfully" programmed their
router to announce your route regardless of whether you sent a route
to them. They want to wait for a maintenance window to remove that
configuration.
I'm at a loss of what else I can do. They admit the problem but won't take
action saying it needs to wait for a maintenance window. Am I out of line
insisting that's an unacceptable response to a problem that results in
prefix/traffic hijacking?
Try dropping the link entirely. If they still announce your addresses,
bring it back up but report it as emergency down, escalate, and call
back every 10 minutes until the junior tech understands that it's time
to call and wake up the guy who makes the decision to fix it now.
Regards,
Bill Herrin
I'm at the tail end here almost 8 hours later since the hijacking started. Their NOC is just blowing me off now and they're happy to continue the hijacking until it's convenient for them to have a maintenance window. And that's apparently the final decision.
~Seth
Start announcing their prefixes?
Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373
Is this related to 104.73.161.0/24? That's ours. 
We'll take a look and get back to you. Thanks for caring!
Best,
Marty
Yep, that's one of the affected prefixes.
~Seth
That's something I would do. Announce announce and keep adding ports until
I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in a
blackhole route for the prefixes. Try to pick blocks that are as
geographically located to your peering routers as possible ...IE in Reno
pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
..... when that batch of customers makes their phones ring all night
someone will listen.
Would be nice if our membership organization ARIN ( that we all pay to
keep us somewhat organized) had an ability to do something for you.... I
never looked into it...i don't know....maybe it does ?
But, in the mean time I am pretty sure you can document this well and
prove your announcements of theirs was due to the fact you couldn't get
proper technical attention and needed to desperately before your customers
cancel after 8 hours of this. Tomorrow call your lawyers and begin to sue
that cable company (did I recognize that ASN as cable TV ? ) for damages
this must be causing you in ill-will amongst your customer base.
I wonder just how you prove the damage...some equation based on customer
calls and complaints together with how many years you have been in
business as well as the number of contracts that are coming up for
renewal. etc etc. Now that would be interesting to see a formula for that
if anyone has been through it.
Thank You
Bob Evans
CTO
Start announcing their prefixes?
Contact the upstreams of AS20115 - Cogent, Level3, HE and XO.
-Hank
That's something I would do. Announce announce and keep adding ports until
I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in a
blackhole route for the prefixes. Try to pick blocks that are as
geographically located to your peering routers as possible ...IE in Reno
pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
..... when that batch of customers makes their phones ring all night
someone will listen.
that seems like a pretty poor strategy... guaranteed to get you into
some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't
the same thing as the customer-service-center. There's likely little
to link the 2 things together there 
Would be nice if our membership organization ARIN ( that we all pay to
keep us somewhat organized) had an ability to do something for you.... I
never looked into it...i don't know....maybe it does ?
arin does not guarantee 'routability' of netblocks assigned to your org.
But, in the mean time I am pretty sure you can document this well and
prove your announcements of theirs was due to the fact you couldn't get
proper technical attention and needed to desperately before your customers
cancel after 8 hours of this. Tomorrow call your lawyers and begin to sue
that cable company (did I recognize that ASN as cable TV ? ) for damages
this must be causing you in ill-will amongst your customer base.I wonder just how you prove the damage...some equation based on customer
calls and complaints together with how many years you have been in
business as well as the number of contracts that are coming up for
renewal. etc etc. Now that would be interesting to see a formula for that
if anyone has been through it.
you COULD find a charter person on-list...there are nine names on the
attendees list for the upcoming meeting... I imagine peeringdb likely
has folk listed... gosh it sure does:
<https://www.peeringdb.com/private/participant_view.php?id=2144>
what with their emails and everything.
+1, this is the only sensible advice here.
NSPs actually do seem to care about not letting things like these happen.
Cogent and Level3 will tell you that you are not their customer ...HE and XO will react.
Jürgen Jaritsch
Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300
Telefax: +43-5-0556-500
E-Mail: jj@anexia.at
Web: http://www.anexia.at
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Willful negligence. Will only be in your favor when it comes to collect damages.
-Dan
That's something I would do. Announce announce and keep adding ports
until
I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in
a
blackhole route for the prefixes. Try to pick blocks that are as
geographically located to your peering routers as possible ...IE in Reno
pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
..... when that batch of customers makes their phones ring all night
someone will listen.that seems like a pretty poor strategy... guaranteed to get you into
some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't
the same thing as the customer-service-center. There's likely little
to link the 2 things together there
You are right - probably creates more problems than good.
Would be nice if our membership organization ARIN ( that we all pay to
keep us somewhat organized) had an ability to do something for you.... I
never looked into it...i don't know....maybe it does ?arin does not guarantee 'routability' of netblocks assigned to your org.
Yep, I was pretty sure of that - but wouldn't it be nice if arin could
have some communication line or at least try. Yes, never any guarantees
really.
bob
That's something I would do. Announce announce and keep adding ports
until
I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in
a
blackhole route for the prefixes. Try to pick blocks that are as
geographically located to your peering routers as possible ...IE in Reno
pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
..... when that batch of customers makes their phones ring all night
someone will listen.that seems like a pretty poor strategy... guaranteed to get you into
some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't
the same thing as the customer-service-center. There's likely little
to link the 2 things together thereYou are right - probably creates more problems than good.
Would be nice if our membership organization ARIN ( that we all pay to
keep us somewhat organized) had an ability to do something for you.... I
never looked into it...i don't know....maybe it does ?arin does not guarantee 'routability' of netblocks assigned to your org.
Yep, I was pretty sure of that - but wouldn't it be nice if arin could
have some communication line or at least try. Yes, never any guarantees
really.
I'm fairly sure that the arin (or ripe or apnic or...) answer to your
question is: "read the contact info in whois... call the stated
numbers."
pretty sure that's also not going to be super helpful, email the poc's
in the peering-db.
On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen"
You'd be surprised how often this happens, especially on the back of a
conference rocking into a city/country and the local provider having
minimal BGP experience. Once the conference is done, folk leave, and the
provider forgets about things - which is not a problem since the
conference would have come with its own IP address space.
The issue goes unnoticed for 12x months when the conference is trying to
route their usual block in some other city/country, and things just seem
"strange". Someone remembers the previous year's event, calls up the
previous provider, and finds out that the tech. who worked the
activation has since left.
It's not easy...
Many other situations closer to home (i.e., paying customers) where
things like this happen, especially if the customer has IP address space
but does not do BGP (until they want to or leave to the competition).
Blackholing operations that go wrong that folk forget about as well, not
to mention other networks that cut themselves off by using public IP
address space for their enterprise network.
It's not easy at all...
Mark.
Reno, NV. I do believe they've finally withdrawn this morning (I just woke up, it was a long night).
~Seth
Nice of you to check Jim. This brings up the old idea - A long time ago I
had an INOC phone by PCH.NET - It never rang, as we filter our outbound
with detail everywhere we announce. ISPs need to provide us their address
list.
And the few times I needed to use it , no one ever answered. ( It was a
decade ago before NANOG membership.) So after a while I too ignored it.
Maybe this was an idea ahead of it's time ? From this painful mishap, it
could have been a great solution for NOC Engineers to help each. I find
peeringdb often outdated as companies change around and sluggish return
call if at all. Most are like a sales line number post.
I see now a long list of registered networks in the PCH directory. Are
networks actually paying attention and using it. Is it time to take
another look ? At midnight in your organization could you get a NOC
person with " proper BGP skills and access " to answer and care about a
bad announcement ?
https://inoc-dba-web.pch.net/inoc-dba/console.cgi?op=show_pubdir&list=org
Link above shows lots more networks listed on the
INOC-DBA Public Directory: Organizations
But have you used it? Did it work for you when you needed it ?
Any further comments are appreciated.
This seems like a very good proper civil approach - maybe this or
something like it ARIN might help promote and endorse as a benefit to the
community ? Be nice if with the cash they did something simple like this
and got all of us to use it? Special line forwarding ? A Emergency Only
NOC App for our phones for just this kind of situation - one that
registers a specific ASN and pin code we set on the registration page ?
Thank You
Bob Evans
CTO
Hi Bob,
A friend is not someone that allows their company to hijack your prefixes.
A friend is one that can get it to stop. Dude - wake up and drink some
coffee.
Thank You
Bob Evans
CTO