Blackholes and IXs and Completing the Attack.

Mailman List Mirror (Read Only)
1

Hi,

I have been working away on remote trigger blackholing and community
based client initiated blackholing into transit ASes. It got me
thinking that while this works great with a handful of upstream transit
peers it does not really scale very well at an Internet Exchange with a
high overhead configuring things for many peers. Plus if your IX
connection is saturated that means legitimate traffic must be getting
degraded - even if your router is coping and blackholing the
interconnect is still flat lined.

The only ways into an AS are via transit, public IX or private
interconnects. If we want to extend the blackholing to secure IXs peers
as well as into transits.

So my idea....

Is to have an IX route reflector configured with ACLs locking it down to
exclusively BGP with the IX peer IP of the member. The IX route
reflector would be configured to have per peer prefix filters per peer
auto generated from registered AS macro for each peer from the
RIPE,ARIN,APNIC etc databases. This should mean the router will not
accept announcements for any /32 that is not part of the routes
announced by that AS (it would be even better to tie it down to a match
on origin AS as well). Plus the router will only talk to IX peers - no
global transit.

This hopefully will ensure a relatively protected router that is only
accessible from the edge routers we want and also secured to only accept
filtered announcements for black holing and in consequence enable the
system to be trusted similar to Team Cymaru.

Then all a member AS of the exchange does is announce any /32 from their
IP block that they would like other members to Null route in their AS to
this reflector.

There are people way smarter than me on this list and the above is not
implemented at any of the IXs I am connected to, so why is the above a
dumb idea / what have I missed that makes the above unworkable because
it does seem kind of obvious now I have done some work with this.

Kind Regards

Ben Butler

2

ben.butler@c2internet.net ("Ben Butler") writes:

...
This hopefully will ensure a relatively protected router that is only
accessible from the edge routers we want and also secured to only accept
filtered announcements for black holing and in consequence enable the
system to be trusted similar to Team Cymaru.
...

This sounds like another attempt to separate the Internet's control plane
from its data plane, and most such attempts do succeed and are helpful
(like NSP OOB, or like enterprise-level anycast of DNS). However, I'm not
sure that remote triggered blackholes are a good direction, worthy of the
protection you're proposing, for three reasons.

First, because large NSP's simply cannot afford the risk associated with
letting a third party, automatically and without controls or audits, decide
in real time what sources or destinations shall become unreachable. With
all respect (which is a lot) for spamhaus and cymru and even MAPS (which I
had a hand in, back in the day), feeding BGP null-routes to a multinational
backbone is a privilege that ISO9000 and SarBox and liability insurance
providers don't usually want to extend.

Second, because many backbone routers in use today can't do policy routing
routing (which is in this case dropping packets because their source address,
not their destination address, has a particular community associated with it)
at line speed. Note, this is many-not-all -- I'm perfectly aware that lots
of backbone routers can do this but not everybody has them or can afford them
and those who have them tend to be the multinational NSPs discussed earlier.
To prevent our DDoS protection reflexes from lowering an attacker's cost (by
automatically blackholing victims to protect the nonvictims), we have to be
able to blackhole the abusive traffic by source, not by destination.

Third, because many OPNs (other people's networks) still don't filter on
source address on their customer-facing edge, and thus allow spoofed-source
traffic to exit toward "the core" or toward a victim's NSP who cannot filter
by source due to path ambiguities inherent in "the core", any wide scale
implementation of this, even if we could get trusted automation of it at
scale and even if everybody had policy-routing-at-like-speed, would just push
the attackers toward spoofed-source. That means a huge amount of work and
money for the world, without changing the endgame for attackers and victims
at all. (See BCP38 and SAC004 for prior rants on this controversial topic.)

3

Hi,

I was not proposing he Null routing of the attack source in the other
ISPs network but the destination in my network being Null routed as a
destination from your network out.

This has no danger to the other network as it is my network that is
going to be my IP space that is blackholed in your network, and the
space blackholed is going to be an address that is being knocked of the
air anyway under DoS and we are trying to minimise collateral damage. I
cant see where the risk to the large NSP is - given that the route
reflector will only reflect /32s that legitimately originate (as a
destination not a source) in the AS announcing them as please blackhole.

For complete clarity: AS13005 announces 213.170.128.0/19 and has its
route object in RIPE as being announced by AS13005. My router at IX -
BENIX say - announces 213.170.128.1/32 to the router reflector their,
the announcement from my IX assigned address 212.121.34.30 is known to
be my router on the exchange, and I am announcing a /32 from my AS for a
route object registered as being announced by my AS - so the reflector
accepts my announcement and reflects it to any other members that choose
to peer with this reflector - effectively this is a please blackhole
this destination in AS13005 - the other members that receive this
announcement can then deal with it in anyway they see fit from ignoring
it to setting next-hop 192.0.2.1 -> Null0.

The effect of this would be that any BotNet controlled hosts in the
other member network would now be able to drop any attack traffic in
their network on destination at their customer aggregation routers.

I think you might have thought I was suggesting we blackhole sources in
other peoples networks - this is definatly not what I was saying.

So, given we all now understand each other - why is no one doing the
above?

At the end of the day if an IX member doesn't want the announcements
don't peer with the blackhole reflector, simple, and it will get Null
routed as soon as it hits my edge router at the IX - it would just seem
more sensible to enable people to block the traffic before it traverses
the IX and further back in their own networks.

So???

Ben

4

You could achieve the exact same result simply by not advertising the
network to your peers, or by advertising a bogus route (prefixing a
known bogon AS for the addresses you want null-routed). I realize you
would have to subnet/deaggregate your netblocks, and therefore could
wind up with a prefix-length issue for peers who won't accept routes
longer than a certain mask, in some cases, but if you are already being
DDOSed, this should represent SOME improvement.

If you're trying to do it on a /32 basis, I doubt you'd find too many
border router operators interested in accepting a route that small, but
I may be wrong.

The bigger issue with all these approaches is that they run afoul of a
patent applied for by AT&T:

http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u
=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=200
60031575&OS=20060031575&RS=20060031575

USPTO App Number 20060031575

5

So, given we all now understand each other - why is no one doing the
above?

Some folks are doing this, just not via some third-party
route servers. For example, either via customer peering
sessions, or other BGP interconnections between peers.
E.g.:

http://www.nanog.org/mtg-0402/morrow.html

I'm not sure it's ideal to employ third-party route servers
for this purpose, as it only increases the attacks/error
surface. I suppose if folks rely on it for native peering
then it might be reasonable.

At the end of the day if an IX member doesn't want the announcements
don't peer with the blackhole reflector, simple, and it will get Null
routed as soon as it hits my edge router at the IX - it would just seem
more sensible to enable people to block the traffic before it traverses
the IX and further back in their own networks.

Yes, helping to ease effects of collateral damage from
large-scale attacks by conveying drop policies to upstream
ASes for prefixes which you originate. And perhaps as
significant, being able to allow the target AS to remove
those policies dynamically, rather than having to contact
each upstream AS that appears to have imposed them
manually out-of-band.

I think Paul's comments were more regarding the fact
that destination-based blackhole routing for mitigation
*effectively completes the attack*, which is often times
undesirable. Inter-domain source-based blackhole
routing is pretty much a non-option.

One other offshoot is that advertised more-specifics
are going to further contribute to routing AND forwarding
table bloat, and a single new prefixes might result in
10+ new paths in the iBGP RIBs.

If you do implement something like this you may wish to
scope advertisement only to adjacent ASes via
NO_EXPORT or the like, to scope both more-specific
propagation, and to ensure that some lack of consistent
drop community semantic interpretation doesn't hose
something.

Also, if you impose this as a standard attack response
mechanism recall that you lose visibility of attack scales,
and knowing just when to resume normal forwarding
policies is a bit more complex. As such, your policy sets
may want to provide hooks that enable selective prefix
advertise/withdraw drop policies so that it can be applied
or removed incrementally.

-danny

6

Somene from ATT may want to consider pulling this patent application
since it seems to fail on prior art...

http://www.nanog.org/mtg-0410/soricelli.html

presented by a juniper employee (Joe Soricelli ) and Wayne Gustavus
from Verizon. IANAL though...

7

I was not proposing he Null routing of the attack source in the other
ISPs network but the destination in my network being Null routed as a
destination from your network out.

i explained why this is bad -- it lowers the attacker's costs in what
amounts to an economics war. they can get a web site taken down by its
own provider just by attacking it. they need fewer resources for their
attack once they know the provider's going to blackhole the victim.

This has no danger to the other network as it is my network that is
going to be my IP space that is blackholed in your network, and the
space blackholed is going to be an address that is being knocked of the
air anyway under DoS and we are trying to minimise collateral damage.

your collateral damage is of precious little interest to someone else's
backbone staff, unless they can route-filter the potential announcements
so that you are unable to also remotely blackhole addresses you don't
advertise. i explained this as an insurance/ISO9000 problem.

I think you might have thought I was suggesting we blackhole sources in
other peoples networks - this is definatly not what I was saying.

i explained why this would be a more sensible approach, but STILL unworkable.

So, given we all now understand each other - why is no one doing the above?

now that we've rehashed what we both said, i think we're done here.

8

"destination-based blackhole routing for mitigation *effectively
completes the attack*, which is often times undesirable. Inter-domain
source-based blackhole routing is pretty much a non-option."

That is why I put Completing the Attack in my subject line - and didnt
attempt to sujest this as an approach for source based filtering.

"If you do implement something like this you may wish to scope
advertisement only to adjacent ASes via NO_EXPORT or the like, to scope
both more-specific propagation, and to ensure that some lack of
consistent drop community semantic interpretation doesn't hose
something."

It is upto the receiving AS what they do with the announcement and
whether they advertise it to their BGP customers or not - they shouldnt
be announcing to anyone else anyway. I would sugest they dont, but I
wouldnt presume. Advertisments to transits do not get propigated
outside their AS.

"I suppose if folks rely on it for native peering then it might be
reasonable."

This is envisage to be used between members of the same Internet
Exchange only. Where there are so many peers the overhead of trying to
do this on a peer peer basis / agreeing comunities is going to be a pain
in the arse. And that this gives an easier way to get to the goal - we
do after all trust the IX operator to run the critical bit of
infrastrucutre which is the exchange.

Kind Regards

Ben

9

Hi,

"i explained why this is bad -- it lowers the attacker's costs in what
amounts to an economics war. they can get a web site taken down by its
own provider just by attacking it. they need fewer resources for their
attack once they know the provider's going to blackhole the victim."

I thought the cold war nuclear arms race had shown up to be truly MAD.
Who is paying for this ever escalating capacity of infrastructure as a
way to survive large DoS attacks.

Smaller attacks can be absorbed, but I really cant see a strategy of
endlessly upgrading network router and WAN infrastructure to ensure
enough head room ideal capacity is a particularly economically sensible
approach to the problem.

Ben

10

"If you're trying to do it on a /32 basis, I doubt you'd find too many
border router operators interested in accepting a route that small, but
I may be wrong."

Well then they wouldn't be peering with this route reflector in the
first place.

11

While I am not sure I fully understand your suggestion, I don’t think it would be that hard to set up manually.

Sure it would require asking the individual peers for their black hole communities, but of they don’t have one they are unlikely to honor the infrastructure you describe anyway.

Assume your network is set up to discard packets marked with community 13005:666

Get a list of your peers blackhole communities, when you announce the route from a location on your network, tag it with community 13005:666 but also 1111:777, 2222:888 etc. for the individual peers from the source. This prevents you from having to update multiple policies in multiple locations for each attack.

As long as they accept the /32 announced to them with their black hole community, they should discard the traffic without sending it to you.

Not all peers will have a blackhole community, but you need some way to know when the attack is over to know when to withdraw the route, and they are useful for this.

If you are real lazy, on the router you announce the black hole from, add an export policy that says from community 13005:666, then community add 1111:777, 2222:888 etc.

This way you only need to:

  1. Update one policy in one place when peers change
  2. Announce the route from one location adding one community to it.
12

ATT has no reason to pull their application, what needs to happen is
that the publisher of the prior art contact the USPTO.

If ATT willingly failed to note the prior art in their app, that may be
a problem, but it isn't their duty to report ALL prior art, just the
stuff they know about.

IANAL, but I have filed some patents, and reviewed a bunch more.

13

ATT has no reason to pull their application, what needs to happen is
that the publisher of the prior art contact the USPTO.

If ATT willingly failed to note the prior art in their app, that may be
a problem, but it isn't their duty to report ALL prior art, just the
stuff they know about.

sweetness, hopefully Wayne or Verizon (they have lots of lawyers) or
Juniper will ping USPTO... or not, I suppose I don't care directly
anymore :slight_smile:

14

I see your point, but I think maintaining the box for the control session would also require a decent amount of work.
Presumably, since you must all adhere to some quasi-standard to communicate with the control peer, you could probably also agree on creating a standard BGP community (ie. 64666:666 & no-export) to use and just skip the middle man.

Granted, I am kind of new as well, and I assume if the solution were that simple more people would be using it.

15

"Well then they wouldn't be peering with this route reflector "

Well then, the utility is probably close to 0, isn't it?

I doubt most of the sources of DDOS traffic, especially those without
ingress source filtering, are going to peer with your route reflector.
What's their economic incentive to expend the engineering time to do so?

I sincerely doubt that any backbone provider will filter at a /32. That
means they have to check EVERY PACKET AT FULL IP DEST against your AS
advertised routes. Since most backbone routers build circuits at the /18
and above mask on MPLS, just to keep up with traffic, I sincerely doubt
they are going to expend the CPU, and potentially RAM, never mind prefix
table entries (you know, those things we're running out of) to have a
full table of every host that every hoster says is being DDOSed. In this
case, there's a clear economic cost, for no economic benefit (they do
actually make money delivering that DDOS traffic).

A better approach would be to move your DDOS target and all the rest of
its co-subnet hosts into a different /24, update the DNS RRs, and cease
advertising that /24.

If you really want to be nice, they don't need to renumber, you just
need to stop advertising the target subnet, change the DNS RR's and NAT
at your borders, if you control DNS and IP. The added benefit of this is
that you can swap them back when the DDOs is over, and they get to stay
up while it's happening. All you need to do this is some spare, never to
be allocated, IP space.

Works with the existing infrastructure, doesn't require an "Internet
God", and in general, is likely to be more effective in the real world.

That whole "rough consensus and running code" ethos of the IETF thing,
as opposed to the "Cathedral" mentality of the ITU (and ICANNt), which
your approach would imply.

You control which routes you advertise, after all.

Plus, AT&T's legal beagles don't get to send you a dunning notice when
their patent gets granted.

From: Ben Butler [mailto:ben.butler@c2internet.net]
Sent: Saturday, February 02, 2008 2:42 PM
To: Tomas L. Byrnes; nanog@merit.edu
Subject: RE: Blackholes and IXs and Completing the Attack.

"If you're trying to do it on a /32 basis, I doubt you'd
find too many border router operators interested in accepting
a route that small, but I may be wrong."

Well then they wouldn't be peering with this route reflector
in the first place.

From: Tomas L. Byrnes [mailto:tomb@byrneit.net]
Sent: 02 February 2008 20:39
To: Ben Butler; Paul Vixie; nanog@merit.edu
Subject: RE: Blackholes and IXs and Completing the Attack.

You could achieve the exact same result simply by not
advertising the network to your peers, or by advertising a
bogus route (prefixing a known bogon AS for the addresses you
want null-routed). I realize you would have to
subnet/deaggregate your netblocks, and therefore could wind
up with a prefix-length issue for peers who won't accept
routes longer than a certain mask, in some cases, but if you
are already being DDOSed, this should represent SOME improvement.

If you're trying to do it on a /32 basis, I doubt you'd find
too many border router operators interested in accepting a
route that small, but I may be wrong.

The bigger issue with all these approaches is that they run
afoul of a patent applied for by AT&T:

http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HI

TOFF&p=1&u

=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=P

G01&s1=200

16

I sincerely doubt that any backbone provider will filter at a /32. That
means they have to check EVERY PACKET AT FULL IP DEST against your AS
advertised routes. Since most backbone routers build circuits at the /18
and above mask on MPLS, just to keep up with traffic, I sincerely doubt
they are going to expend the CPU, and potentially RAM, never mind prefix
table entries (you know, those things we're running out of) to have a
full table of every host that every hoster says is being DDOSed. In this
case, there's a clear economic cost, for no economic benefit (they do
actually make money delivering that DDOS traffic).

"most backbone routers build circuits at the /18 and above mask on MPLS" -
that part is seriously funny.

However:
a) Yes, if such proposal was to be widely accepted, it would generate more
entries in RIB/FIB.

b) However, if this service was actually operated by IX's, the limits to
prevent "too much" growth could be applied centrally (max-prefixes per
ASN, automatic removal of those routes after X days, unless manually
requested by host, etc).

c) Since only your peers will have those :666 entries, it is less "route
growth" than than the alternative of announcing the affected block as /24
(which you seem to suggest).

A better approach would be to move your DDOS target and all the rest of
its co-subnet hosts into a different /24, update the DNS RRs, and cease
advertising that /24.

That...is...perverted. Not to mention, you can't "cease advertising /24".
what you would need to do is to deaggregate your (say) /20 into /21, /22,
/23 and /24. That's 3 extra entries in FIB for everyone in the world to
carry.

If you really want to be nice, they don't need to renumber, you just
need to stop advertising the target subnet, change the DNS RR's and NAT
at your borders, if you control DNS and IP. The added benefit of this is
that you can swap them back when the DDOs is over, and they get to stay
up while it's happening. All you need to do this is some spare, never to
be allocated, IP space.

That...is...perverted.

-alex [not speaking as mlc anything]

17

yes absolutely, if an agreement could be reached - then that is a neater solution, but I wonder if an agreement could ever be reached in a timescale that doesn’t make deployment of the alternative more attractive as it doesn’t require everyone to agree.

18

Hi,

I am no lawyer, but I question whether ATT can patent anything that uses
the existing technology and commands implemented in BGP. The idea that
you can patent a persons intent in advertising a prefix in BGP seems
somewhat bizarre. Surely a patent relates to the use of a new bit of
technology that they have developed.

Btw, I think I might be a good idea to do all sort of things, that
doesn't mean I can file legally enforceable patents in case someone in
the future shares my point of view.

With regards to:

"A better approach would be to move your DDOS target and all the rest of
its co-subnet hosts into a different /24, update the DNS RRs, and cease
advertising that /24."

I just think you don't get it. Apart from the impracticality of
renumbering all the other non-effected hosts in the same /24 and all the
associated DNS records and the amount of time that is going to take.
Plus then announce another /24 for the renumbered hosts. You are are
saying that I should de-aggregate the /19 announcement into components
so that I can stop advertising the original /24 - because your worried
about route table prefix pollution!!!!

If I blackhole a /32 to my transits, it does not appear in your routing
table. If I blackhole a /32 to one of my peers and you are not even at
the same IX, then there is no change of it appearing in your route table
either. Conversely you seem to be suggesting I announce a bunch of
prefixes to everyone?

Kind Regards

Ben

19

IANAL, but my understanding of patents is that anyone can patent any
innovative combination of existing things, if it produces a new effect
or is used in a new way. According to the patent lawyers I've worked
with, that is, in fact, the most common type of patent. As an example,
Watt's steam engine combined a piston, a balance beam, a wheel, and a
crank, all of which existed before, but his invention was patented.

The ATT patent is on black-holing using BGP null routing to mitigate
DDOS.

As far as your proposal goes, perhaps I had misunderstood. My
understanding was that you wanted everyone in the Internet to accept
routes from an AS that was intended to be black-holed. Those routes
would be hosts (/32) that were, in fact, placed in the route server by
some trusted community (presumably hosters). The benefit of this would
be to eliminate the collateral damage to other customers of the hosters
caused by traffic targeting the DDOS target also denying service to
them.

If I have misunderstood, then I'm sorry, but my responses were based on
the following assumptions:

1: The routes to be black-holed would all be /32s

2: The routes to be black-holed would have to be, in order to be
effective, accepted by routers that carried the vast majority of the
Internet's traffic, since you want to drop DDOS traffic as close to its
source as possible. In my mind that would, at the very least include all
the routers at major aggregation and peering points.

3: Backbone routers can't reasonably filter on a bunch of /32s and also
forward traffic at wire speed.

4: It would be much harder to get all the ingress networks, which
include all sorts of small local and regional ISPs, to join such a
scheme than it would be to get larger ISPS to do so, assuming item 3
above is not true.

5: When one /32 is under DDOS, the rest of the hosts served by the same
links are also effectively DOSed, ergo renumbering them out of the DOSed
space, while painful, might be less painful than continuing to deal with
the DOS.

6: You control what routes you advertise, and therefore can,
effectively, make any prefix as short as or shorter than the shortest
prefix accepted by your peers go dark, simply by stopping advertising
it.

7: The increase in route prefixes caused by disaggregating would,
assuming that there were multiple DDOS targets at any one time, not be
materially larger than that caused by accepting a bunch of /32s, and may
well be less.

8: Disaggregation can be done now, with the tools currently available,
and requires no additional hardware, software, or legal agreements.

It seems that you are saying that you're just asking your immediate
peers to accept this private AS from you, to black-hole traffic to that
AS locally, and not propagate the routes to their peers. That's
completely different than what I thought you were talking about, which
was some sort of Internet-wide black-hole AS that everyone was supposed
to peer with. What you and your immediate peers do is between you and
them, and I could see this as working quite well on that basis. All it's
doing, however, is absorbing the DDOS one step upstream, which is
probably a place that can do so more easily, and clearly is doing so any
time the DDOS is getting through.

In any event, as I pointed out, null routing via BGP to defend against
DDOS is Patent Pending ATT, so unless you can get the patent not to
issue, building a network based on it runs the risk of the patent troll.

I've said my piece on this. Where I've made errors or misspoken, I'm
happy to stand corrected. If I've offended anyone, I'm sorry.

Best wishes to all. May your packets flow, your sessions connect, and
your pagers remain silent.

20

3: Backbone routers can't reasonably filter on a bunch of /32s and also
forward traffic at wire speed.

yes they can. the size of the individual route doesn't matter to the
devices in question, the NUMBER of routes does... (as does the
associated change-rate of that number, but that's a story for another
day)

4: It would be much harder to get all the ingress networks, which
include all sorts of small local and regional ISPs, to join such a
scheme than it would be to get larger ISPS to do so, assuming item 3
above is not true.

some already do this though... not in quite the manner Ben's aiming to
do, but there are folks that accept BGP feeds in order to drop traffic
inside there network(s).

5: When one /32 is under DDOS, the rest of the hosts served by the same
links are also effectively DOSed, ergo renumbering them out of the DOSed
space, while painful, might be less painful than continuing to deal with
the DOS.

you have not had to deal with renumbering I presume? not a raft of
end-users (consumers nevermind businesses). Why is the assumption that
the surrounding space is a /24 relevant exactly? The aggregation
scheme used inside any particular network isn't necessarily '/24 per
pop/link/service-area'...

renumbering for DDoS isn't really a workable solution, save the
distinct case when you own the IP in question and services it provides
(and other ancillary bits/bytes related to said ip/device/thingy).

8: Disaggregation can be done now, with the tools currently available,
and requires no additional hardware, software, or legal agreements.

your point here is that perhaps instead of this scheme one would just
advertise the max-prefix-length (/24 currently) from a 'better' place
on your network and suck all the 'bad' traffic (all traffic in point
of fact) for the attacked destination via a transit/peer/place which
can deal with it properly?

This isn't a bad solution, and it gives you some control on the
traffic stream, it does have the penalty to everyone else of 'one more
route in the RIB/FIB'... which I think was Ben's vote against this
method. (also not a bad vote...)

anyway, the idea behind multi-as blackholing has been (and apparently
contunues to get) rehashed a few times over the last 5-8 years... good
luck!

-Chris