Thanks & Let's Prevent this in the Future.

Mailman List Mirror (Read Only)
1

First off, I'd like to thank everyone on this list who have reached out
today and offered us help with our hijacked network space. It's so
refreshing to see that there are still so many who refuse to leave a
man/woman down.

I'm not going to place any blame, its useless. There were lies, there were
incompetencies, and there was negligence but that is now water under the
bridge.

However, I think that we as network operators have a duty to each other to
make sure we don't allow a downstream customer wreck the operations of
another entity who has been rightfully allocated resources.

A few months ago, when establishing a new peering relationship I was
encouraged (actually required) to utilize one of the IRRs. I took the time
to register all of my routes, ASNs, etc. However, as I learned today, this
was probably done in vain. Too many people won't spend the extra
30-seconds to verify the information listed there or in ARINs WHOIS.

I don't care what a customer tells me, too many times I've found they
aren't 100% honest either for malicious/fraudulent reasons or they are
unknowing. So, for our networks or the networks we manage, we want to
verify what a customer is saying to prevent what happened to us today.

I'd like to get a conversation going and possibly some support of an
initiative to spend that extra 30-seconds to verify ownership and
authorization of network space to be advertised. Additionally, if someone
rings your NOC's line an industry-standard process of verifying "ownership"
and immediately responding by filtering out announcements. There's no sense
in allowing a service provider to be impaired because a spammer doesn't
want to give up clean IP space. Do you protect a bad customer or the
Internet as a whole? I pick the Internet as a whole.

How can we prevent anyone else from ever enduring this again? While we may
never stop it from ever happening, spammers (that's what we got hit by
today) are a dime a dozen and will do everything possible to hit an Inbox,
so how can we establish a protocol to immediate mitigate the effects of an
traffic-stopping advertisement?

I thought registering with IRRs and up-to-date information in ARINs WHOIS
was sufficient, apparently I was wrong. Not everyone respects them, but
then again, they aren't very well managed (I've got several networks with
antiquated information I've been unable to remove, it doesn't impair us
normally, but its still there).

What can we do? Better yet, how do we as a whole respond when we encounter
upstream providers who refuse to look at the facts and allow another to
stay down?

kw

2

It's amazing isn't it. It isn't only fraud and maliciousness that this prevents.

A number of times I have been asked to advertise space or open filters for space based on typos and people not understanding address notation and CIDR. So it's with doing if only to prevent this.

3

I'd like to get a conversation going and possibly some support of an
initiative to spend that extra 30-seconds to verify ownership and
authorization of network space to be advertised. Additionally, if
someone rings your NOC's line an industry-standard process of verifying
"ownership"
and immediately responding by filtering out announcements. There's no
sense in allowing a service provider to be impaired because a spammer
doesn't want to give up clean IP space. Do you protect a bad customer
or the Internet as a whole? I pick the Internet as a whole.

How can we prevent anyone else from ever enduring this again? While we
may never stop it from ever happening, spammers (that's what we got hit
by
today) are a dime a dozen and will do everything possible to hit an
Inbox, so how can we establish a protocol to immediate mitigate the
effects of an traffic-stopping advertisement?

One problem is the number of routing registries and the requirements differ for them. The nefarious operator can enter routes in an IRR just as easily as a legitimate operator. There was a time when some significant networks used the IRRs for their filtration policy. I'm not sure how many still do.

But generally speaking, if someone calls me and I can verify that they really are a POC for the entity that is assigned an address allocation (generally some verification method beyond email if the subnet their MX record points to is part of the hijacking!) then I am going to do whatever I can to help them out provided what they are asking for is reasonable and within my capabilities. It shouldn't be too hard to verify. If someone claims to be with a commercial entity of Foo.COM then the entity is likely listed in the phone book and a phone call can take care of my personal verification requirement.

Back in the days of Cyberpromo and Sanford Wallace, what I did was used TCP wrappers on my mail server so that when I received a connection from a Cyberpromo net block, I hairpinned the connection back to his MX server using netcat so when he connected to me, the HELO he received was from his own mail server, which gladly accepted mail from Cyberpromo. He could pump mail to me all day long if he wanted to, but his mailq wasn't going to get any smaller.

The problem of email spam is an interesting one that has been battled for a very long time and is probably better discussed on a list dedicated to that topic.

4

One problem is the number of routing registries and the requirements differ for them. The nefarious operator can enter routes in an IRR just as easily as a legitimate operator. There was a time when some significant networks used the IRRs for their filtration policy. I'm not sure how many still do.

A few do, but IRR data really can't be trusted as a means of authenticating authority to advertise IP space. It's a nice system for automating route filter updates (between the customer and provider...i.e. I add data to an IRR, and Level3 auto-updates my prefix filter), but when anyone can put anything into the IRR dbs, obviously everyone using the IRR dbs isn't going to stop someone from hijacking someone else's space.

As a regional ISP / hosting provider, my policy for accepting BGP routes from customers has been to check RIR whois data, make sure the space appears to be owned by the customer, and if there's any question, ask for clarification and/or an LOA from the customer showing that they are authorized to use the space. Every customer gets a prefix filter that allows them to announce only what we've verified they're authorized to announce.

Level3, I suppose, just trusts customers won't announce space they shouldn't. The alternative, which I've dealt with with some other "Tier 1's" is that they require customers to provide an LOA every time they want to add a CIDR to their prefix filter. That's more paper work for me and for them, and tends to be slower, as someone has to manually approve (maybe manually apply) the change request.

Given what we have available today, there's security or convenience...pick one.

It seems like the IRR thing could reasonably easily be solved if the RIRs got more deeply involved.

Suppose, as an ARIN member, I used ARIN's IRR. I'd start out by registering my maintainer object, my ASN, my routes, and an as-set. Then, when I wanted to add customer ASNs to my as-set, the system wouldn't allow me to do so unless the customer had already registered their ASN with my ASN as an approved provider/path. The customer, if they had no ASN, would be responsible for updating their route (PI or PA) in the IRR db, stating my ASN is a valid origin for their route. This would solve the problem of larger providers like Level3 blindly accepting my as-set because all the authentication/authorization would already have been done before the data went into the IRR db. Things get a little more complicated when I pick up a customer who has "out of region" objects, i.e. RIPE IPs/ASN, but if all the RIRs worked together and standardized how the information is maintained, it could work. i.e. When I try to add the RIPE ASN to my as-set, ARIN's system would check RIPE's system to see if that ASN's owner had set my ASN as a valid provider/path for their ASN.

This seems so simple, either it should have been implemented a decade or more ago, or perhaps I'm overlooking something. It wouldn't stop route advertisements from providers who don't care / don't filter, but it would make systems like Level3's more secure...and if all the "Tier 1's" adopted similar automated filter generators based on the RIR IRR dbs, it would stop unauthorized announcements from getting far.

The problem of email spam is an interesting one that has been battled for a very long time and is probably better discussed on a list dedicated to that topic.

Definitely...but the issue here is ownership/authorization to use IP space, and how stop unauthorized BGP announcements when providers don't or won't filter their BGP customers.

5

One option is to use RPKI and origin validation. But it won't help much unless prefix holders create their certificates and ROAs and networks operators use those to validate origins. It won't solve all the issues but at least some fat fingers/un-expierience errors.

  We are running an experiment to detect route-hijacks/missconf using RPKI. So far, not many routes are "signed" but at least we can periodically check our own prefix (or any other with ROAs) to detect some inconsistencies:

http://www.labs.lacnic.net/rpkitools/looking_glass/rest/all/pfx/200.7.84.0/

http://www.labs.lacnic.net/rpkitools/looking_glass/

Regards,
-as

6

In related news, the IETF working group that is writing standards for
the RPKI is having an interim meeting in San Diego just after NANOG.
They deliberately chose that place/time to make it easy for NANOG
attendees to contribute, so comments from this community are
definitely welcome.
<http://www.ietf.org/mail-archive/web/sidr/current/msg03923.html>
<http://trac.tools.ietf.org/wg/sidr/trac/wiki/InterimMeeting20120209>

7

Thanks for the reminder, Richard.

Yes, as I announced earlier (see
http://mailman.nanog.org/pipermail/nanog/2012-January/044493.html
- the message with the corrected date), there is an interim sidr meeting on
Thu *9* Feb in San Diego.

Registration is free.
Registration is easy (email).
Registration is open to all.
Registration is open ended.
Registration is encouraged (so we know how many to expect).

As the message says, see the sidr wiki http://trac.tools.ietf.org/wg/sidr/trac/wiki/InterimMeeting20120209 for agenda and a list of attendees.

--Sandy Murphy

8

I've dealt with AfriNIC and APNIC WHOIS databases, and they
normally control the 'inetnum' and inet6num' entries that go
into the WHOIS databases. So there is some degree of
certainty that what is in there is generally true.

You're right, anyone can create an IRR record, and it's
quite terrible how easy it is to create false information
that could break another person's network. This is why we
don't generally trust IRR or PeeringDB data when verifying
downstream prefixes which we should permit through our
filters. We rely on the RIR 'inetnum' and 'inet6num' records
for that.

My memory fails me on what ARIN do, but before AfriNIC was
established and the majority of Africa's prefixes were
allocated by RIPE and ARIN, I recall the ARIN policy (SWIP
templates, et al) being a hassle-rich experience that
anything else is long forgotten :-).

Mark.