Unwanted Traffic Removal Service (UTRS)

Mailman List Mirror (Read Only)
1

Friends and colleagues,

Yesterday I briefly discussed a new project we've recently launched and
for which invited participation from the NANOG 62 attendees. This is a
not so subtle wider request for consideration. UTRS is essentially a
community RTBH that people have suggested to us would be a good service
to provide, so we're giving it a go. Yesterday's lightning talk page
has the slide deck I presented:

  <https://www.nanog.org/meetings/abstract?id=2436>

I've also put some additional technical detail here:

  <http://www.cymru.com/jtk/misc/utrs.html>

If you have an assigned ASN and this sounds like something you or your
network would like to take advantage of, drop me a line, I'd sure like
to hear about it.

If you think this is a terrible idea and want to express all that is
wrong with it, tell me that too, I can take it.

Even if you're unsure if you would ultimately deploy it in production,
but are interested in testing it, do contact me. We won't expend energy
on this project and will drop it if we there isn't sufficient interest
from the community, but enough trial users would be encouraging.

Kindly,

John

2

Dear John,

UTRS is essentially a community RTBH that people have suggested to us
would be a good service to provide, so we're giving it a go.

FYI, there are various projects which are similar to this concept:

    http://www.de-cix.net/products-services/de-cix-frankfurt/blackholing/
    https://ripe68.ripe.net/presentations/369-bgp_bh_ripe.pdf
    https://wiki.rtbh.me/

If you think this is a terrible idea and want to express all that is
wrong with it, tell me that too, I can take it.

Just like chicory, personally I don't like it. Yes, Cymru has build a
reputation as clearing house for redistribution of security related
information. But... (aside from any local safety net filter), it's quite
a leap to allow a single entity to inject blackholes for any prefix.

There are various flavors at the moment in terms of validation (please
correct me if I am wrong): The Polish blackholing project only allows
blackholes which fall within the set of prefixes which an ASN
originates, the DE-CIX BS service accepts anything that is a subset of
your AS-SET.

Both approaches have their downsides: you can make any AS or AS-SET a
member of your AS-SET and thereby gain a degree of control on the RTBH
server, and for $500/year you can register any route-object you want in
RADB.

RIPE is the only RIR which has the IRR service as a truely integral part
of the database, allowing advanced automatable authentication schemes
for purposes such as these. However, they only administrate for a subset
of the Internet, making this direction inpractical for a universal
solution.

Might I suggest an alternative approach, without central validation or
need for a clearing house:

IXPs could offer BGP or API triggered ACLs which are inserted into the
peering fabric and only affect the participant's peering port(s). This
way, any blackholing (either correctly applied or malicious) only
affects the initator of that blackhole and nobody else. Advantages are
that aclserver does not require peers to cooperate with each other and
no validation is required.

Kind regards,

Job

3

Just like chicory, personally I don't like it. Yes, Cymru has build a
reputation as clearing house for redistribution of security related
information. But... (aside from any local safety net filter), it's
quite a leap to allow a single entity to inject blackholes for any
prefix.

Hi Job,

Thanks for your comments. I'm aware of some other projects, including
another one, much more elaborate, talked about in another session at
NANOG this week. Do note, UTRS does not allow a single entity to inject
black holes for any prefix, only a limited number of /32's for their own
prefixes. The presentation and the information page I linked to have
some additional details.

IXPs could offer BGP or API triggered ACLs which are inserted into the
peering fabric and only affect the participant's peering port(s). This
way, any blackholing (either correctly applied or malicious) only
affects the initator of that blackhole and nobody else. Advantages are
that aclserver does not require peers to cooperate with each other and
no validation is required.

I've heard of some IXPs recently offering this service, sounds great.
It has also been suggested we might talk to ISPs how to RTBH to their
customers and see if there was a way for those routes to be passed
further along, perhaps to something like UTRS for further
dissemination. I'm not sure that would work, but it was an interesting
idea too.

Thanks for your comments,

John

4

Hi Job,

We already do that on the /24 level... it's called "BGP" and we all
have fun when a service provider in one of the 'Stans injects a route
overriding Youtube.

How could John's system do worse?

Regards,
Bill Herrin

5

information. But... (aside from any local safety net filter), it's quite
a leap to allow a single entity to inject blackholes for any prefix.

Spamhaus has been distributing their DROP list by BGP for years. The
world hasn't ended, and I can't recall the last time it had a
plausible false positive.

http://www.spamhaus.org/drop/

R's,
John

6

The Spamhaus context is different: I know people take that DROP BGP feed
and converting it into ACLs applicable locally on their mailservers, I
don't know of people injecting these prefixes straight into the core.

Kind regards,

Job

7

There is also "dynamic validation" approach: blackhole route is considered
valid for injection if and only if there is a covering less-specific route
with the best-path pointing to the same exit point as blackhole route.
(definition of "exit point" can vary from "next ASn is the same
we received blackhole from" to "both as-path and next-hops must be the
same and aggregate route must be marked as customer's one").

This approach has its downside too: it requires you to run task-specific
bgp speaker. Worse yet, usually you have to write that speaker :slight_smile:

8

Hi John,

It's a good idea, I think, but it has a problem: it's an escalation in
an arms race that doesn't end well for the blue team. If we ever get
good at keeping traffic to a single IP far enough away to not cripple
us, the attacker need only spray the /24. Or spray our entire address
space, easily identifiable from our BGP announcement. All this effort
on our end and it took the attacker 15 minutes to modify his code.

Two general types of DDOS traffic: botnets and forged source addresses.

For the botnets, lots of real machines, each with a legitimate source
IP address, we need to get to a router interface as close to each
source address as we can get. Then temporarily shut down traffic from
that source address crossing that link until the data flow suggests
the problem traffic has ceased. Even if we have to pay the ISP who
owns that link to do it for us.

Quickly find it with automation. Quickly authenticate the attack flow.
Quickly pay for remediation.

For the address forgers, we need some kind of public detection system
where ISPs who care provide the trace tools that let us figure out
where the rogue attacking our network is _actually_ coming from. After
which we can pay the ISP to interdict any traffic destined for
anywhere in our network which enters from that link. Quickly with
automation

We can't win the arms race based on the destination; we'll only win it
if we find a way to zero in on and interdict the source.

Regards,
Bill Herrin

P.S. Also worth noting that paying a DDOS mitigation service can
already accomplish the best-case result from something like UTRS. The
mitigator announces the affected /24, sinks the attacked IP address
and tunnels the rest of the packets back to us. Expensive but easy
peasy.

9

Hello,

If you think this is a terrible idea and want to express all that is
wrong with it, tell me that too, I can take it.

I support the RTBH idea. I also set up https://wiki.rtbh.me/ for this some
time ago and there is also a discuss mailing list where already 65 people are
subscribed. Currently there is not much discussion on the list, but you all
can change this :wink: Also there is no other content besides the mailing list in
the wiki right now. I have removed it because somebody thought that it may be
confusing for some people on the list as he wanted to use it for a general
RTBH discussion instead of my project. I will try to restore it in the next
days...

Just like chicory, personally I don't like it. Yes, Cymru has build a
reputation as clearing house for redistribution of security related
information. But... (aside from any local safety net filter), it's quite
a leap to allow a single entity to inject blackholes for any prefix.

What I do not like at this UTRS idea is that I cannot announce a prefix via
BGP. Somebody has to inject it for me. I would like to announce it in real
time and not with delay because of manual approval.

One problem that I also see here is that this single entity could be forced by
someone (eg. government) to blackhole some prefix. If this ever happens such a
project will have to be terminated.

There are various flavors at the moment in terms of validation (please
correct me if I am wrong): The Polish blackholing project only allows
blackholes which fall within the set of prefixes which an ASN
originates, the DE-CIX BS service accepts anything that is a subset of
your AS-SET.

Both approaches have their downsides: you can make any AS or AS-SET a
member of your AS-SET and thereby gain a degree of control on the RTBH
server, and for $500/year you can register any route-object you want in
RADB.

Allowing ASN to blackhole a prefix based on AS sets is dangerous from my point
of view. In the RIPE database you can add any AS to your AS set without
verification. Ok, it doesn't make much difference because most IP transit
providers also filter on the AS set, but a worldwide announced /24 prefix is
much more visible than a /32 blackhole route that is only announced to the
participants.

My idea was that an ASN can only announce more specifics prefixes (not just
/32) from officially registered routes. Downstream ASN should also be able to
define their upstream ASN who may blackhole their prefixes if they do not set
up their own session to the RTBH route servers. Most IP transit providers have
an RTBH service for their customers that these downstream ASN could use.
Usually they do not offer such a service for peers. This is the useful part here.

We also had some DDoS attacks via IPv6. I think it's important to also have
such a service for IPv6. Starting with IPv4 is ok and better than nothing, but
IPv6 should not be on the roadmap for 2018 :wink:

Might I suggest an alternative approach, without central validation or
need for a clearing house:

IXPs could offer BGP or API triggered ACLs which are inserted into the
peering fabric and only affect the participant's peering port(s). This
way, any blackholing (either correctly applied or malicious) only
affects the initator of that blackhole and nobody else. Advantages are
that aclserver does not require peers to cooperate with each other and
no validation is required.

Why is there no validation required when this is done by an IXP? "All peers
are my customers and I do trust them"? What about private peerings via PNIs?

Regards,

Chris

10

Hi Christian,

<snip>

Why is there no validation required when this is done by an IXP? "All
peers are my customers and I do trust them"? What about private
peerings via PNIs?

Validation is not required because the requester can only affect his or
her own peering ports. Conceptually the IXP participant sets an ACL,
just not on their own ingress routerport but on the egress port just
across the crossconnect, this prevents congestion of said crossconnect.

Modern switching/routing equipment used in IXPs can do far more then
mere L2 switching. Lots of chipsets these days allow you to apply
combined layer-3 + layer-2 ACLs, an example would be "Discard traffic if
(destination IP is A && destination MAC is B)".

The blackhole route is announced to a special network component which I
dub the "ACL Server". The ACL Server is operated by the IXP (exabgp +
customizations?). The ACL Server takes the prefix and associated origin
(origin is a static lookup table based on source IP or ASN, IXPs know
who their members are) and then inserts a layer3+layer2 ACL into their
switching fabric.

If the IXP, on every ingress port, have a ACL that says "drop traffic to
this IP if the dest MAC address corresponds with the originator of the
ACL request", the ddos traffic doesn't even hit the IXPs backbone, and
only affects the peering participant who requested the ACL to be
inserted.

So the blackhole route only lives inside the IXP's switching fabric so
to speak, as an ACL only applicable to the participant itself.

Regarding implementation: some IXPs might have to screenscrape/expect to
apply ACLs on their switches with clogin, others will just convert the
announcement and insert flowspec or sdn rules in the fabric. It is the
IXP's job to sanitize the ACL trigger in such a way that it only applies
to the peering ports of the participant requesting it.

I hope this clarifies a bit.

Kind regards,

Job

11

What I do not like at this UTRS idea is that I cannot announce a
prefix via BGP. Somebody has to inject it for me. I would like to
announce it in real time and not with delay because of manual
approval.

While true today, it might not be true for long. It requires code to
be written in order to perform the desired verification we want before
blindly passing along an announcement. Code we're not motivated to write
if there is insufficient interest in UTRS. Interest is looking good, so
the code may soon follow. In other words, this a valid complaint, but it
may have a limited life span.

One problem that I also see here is that this single entity could be
forced by someone (eg. government) to blackhole some prefix. If this
ever happens such a project will have to be terminated.

I've heard this once before too. I admit we probably can't provide
a satisfactory answer to some who will be so distrustful of government
or influence peddling to win them over, but I'll try to offer a
response that I hope is fairly reasonable and satisfies the majority,
and presumably any of the actual participants.

There are legal questions, maneuvers and responses that might be
interesting to speculate on, but I'll say simply this. Team Cymru,
while established and operated within the U.S., is a global
organization with team members outside of the U.S. and we rely heavily
on the cooperation of global partners to do what we do. If we could
be compelled to announce a black hole by someone, government or
otherwise, the cooperation and inherent trust we might have with the
Internet community is probably gone and we are likely finished as an
organization. It would be counter to our very existence and so on that
basis I hope most would agree is extremely unlikely to occur. Now if
someone came up to me with a gun to my head and said type the
equivalent of "ip route foonet mask 192.0.2.1" or die, I might just
type it out of self preservation.

We also had some DDoS attacks via IPv6. I think it's important to
also have such a service for IPv6. Starting with IPv4 is ok and
better than nothing, but IPv6 should not be on the roadmap for
2018 :wink:

You are only the second person I've heard from to explicitly state as
such. This is actually not terribly hard to do and I'm pretty certain
could be done way before 2018. Simple to start, careful and necessary
improvements as we go.

Thanks for your comments Chris,

John

12

I understand the concerns but it seems to me that there are already plenty of ways for any large government to black hole whatever they want and they do not need UTRS to do so. The only thing stopping (most) governments from doing this regularly are fears of turning the Internet into another arms race. It's a stigma thing like the different between launching the first nuke vs. being the responder. We all know they do a lot of cyber stuff out there but it is mostly behind a veil of deniability.

First, if they have access to a tier 1 carrier (or at least enough carriers to make an impact) in their jurisdiction they could just order that carrier to do it with whatever court system (or not) is required. Most large governments also have enough connectivity to bury a route by brute force. The only thing stopping (most) governments from doing this regularly are fears of turning the Internet into another arms race and possibly losing easy access to that resource. We all know they do a lot of cyber crime stuff out there but it is mostly behind a veil of deniability.

There has actually been more black hole events that occur by accident or as part of denial of service attacks than government launched. The global routing structure of the Internet has always been highly cooperative and vulnerable to a bad actor at a lot of points. My only real concern with UTRS is designing a system that cannot be gamed or exploited to turn it into a very effective DoS weapon system. I admit that I don't know enough about how it works to make that decision yet.

Steven Naslund
Chicago IL

13

See:
http://www.ripe.net/ripe/mail/archives/routing-wg/2014-June/002696.html

-Hank