large BCP38 compliance testing

Mailman List Mirror (Read Only)
1

Hi,

To fix a lot of the DDOS attacks going on, we need to make sure BCP38 compliance goes up. Only way to do this I can think of, is large scale BCP38 testing. One way of doing this, is to have large projects such as OpenWRT, RIPE Atlas project, perhaps even CPE vendors, implement something that would spoof 1 packet per day or something to a known destination, and in this packet the "real" source address of the packet is included.

I have been getting pushback from people that this might be "illegal". Could anyone please tell me what's illegal about trying to send a packet with a random source address?

If we can get consensus in the operational world that this is actually ok, would that help organisations to implement this kind of testing. I could see vendors implement a test like "help verify network stability and compliance, these tests are anonymous" checkbox during the initial install, or something like this.

Why isn't this being done? Why are we complaining about 300 gigabit/s DDOS attacks, asking people to fix their open resolvers, NTP servers etc, when the actual culprit is that some networks in the world don't implement BCP38?

2

After posting this, I was pointed to http://spoofer.cmand.org . This seems like the exact thing I would like to test.

3

ntp monlist / dnssec abuse can provide ~30x amplification. So if you can find ten 1G links anywhere in the world which aren't protected with BGP38 filtering, you can initiate a mostly untraceable 300G DDoS.

This shouldn't stop us from finding, then naming and shaming operators who don't use bcp38, but we also need to maintain realistic expectations about how successful it's going to be.

It would probably be more productive to pressurise transit providers to enforce bcp38 on their customer links.

Nick

4

Le 02/10/2014 12:28, Nick Hilliard a �crit :

It would probably be more productive to pressurise transit providers to
enforce bcp38 on their customer links.

This. But let me ask you, how many transit provider actually implement
strict prefix-filtering ? I've seen many using a max-prefix as their
sole defense.

Now, let's consider what you want is to match an interface ACL to
prefixes received on a BGP session runing through the same interface.
Ain't that what uRPF-strict is all about ? What are the known downsides
to uRPF-strict ?

When buying from transits, you either update your IRR for automatic
perfix-filter generation on your transit's side, or start by a "BGP over
SMTP" session. While the former could generate ACLs from a template, the
latter will be prone to human error. And still, how many of us _really_
ensure their IRRs are always up-to-date ?

Next in line : IXPs. You never really know what routes will be available
or has to be filtered when 800+ AS, most with customers also using BGP,
starts talking to the same route-server. Or maybe, the route-server
could provide a flowspec AFI to send filters AND routes simultaneously.
Would you trust it ? Will your router have enough silicon-horse-power to
match both IP _and_ L2 headers at line-rate ?

BCP38 aims at spoof prevention by filtering as close to the source as
possible. Implementation on network's edge looks to me like a tricky
one. Sharing the load amongst CPE is the best practice, and could be
considered a requirement enforced by transit providers. Or shouldn't it ?

Best regards,

5

uRPF Strict mode is NOT a tool to use on the transit connections. It was built for the SP-Customer connections.

uRPF VRF mode _was_ built for the transit connections. You can take all the prefixes received from the peer and stick them into a VRF. You can then check all the incoming packet source addresses against that list. If there is no match, then it was not in the BGP advertisements.

6

This. But let me ask you, how many transit provider actually implement
strict prefix-filtering ? I've seen many using a max-prefix as their
sole defense.

Plenty do and have no back-end capability to handle this, other than email updates.

Now, let's consider what you want is to match an interface ACL to
prefixes received on a BGP session runing through the same interface.
Ain't that what uRPF-strict is all about ? What are the known downsides
to uRPF-strict ?

Your bgp announcement to your upstream is not guaranteed to be there all the time. E.g. if you're doing maintenance and stop announcing bgp to your upstream for inbound traffic, but still want to depend on it for outbound traffic, urpf will trash things.

urpf is only feasible for statically configured hand-offs.

When buying from transits, you either update your IRR for automatic
perfix-filter generation on your transit's side, or start by a "BGP over
SMTP" session. While the former could generate ACLs from a template, the
latter will be prone to human error. And still, how many of us _really_
ensure their IRRs are always up-to-date ?

This only happens when there is a reason to do so.

Next in line : IXPs. You never really know what routes will be available
or has to be filtered when 800+ AS, most with customers also using BGP,
starts talking to the same route-server. Or maybe, the route-server
could provide a flowspec AFI to send filters AND routes simultaneously.

IXPs are more difficult, but if your IXP is running a route server, they should be implementing strict prefix filtering. At least, this puts pressure on IXP participants to register their prefix at their local irrdb.

Would you trust it ? Will your router have enough silicon-horse-power to
match both IP _and_ L2 headers at line-rate ?

probably yes on most routers with dedicated hardware for this, but it will depend on the number of acl entries.

BCP38 aims at spoof prevention by filtering as close to the source as
possible. Implementation on network's edge looks to me like a tricky
one. Sharing the load amongst CPE is the best practice, and could be
considered a requirement enforced by transit providers. Or shouldn't it ?

urpf is appropriate for the ISP last hop. Static filters are suitable for the transit provider connecting that ISP to the rest of the network.

Nick

7

Hi,

To fix a lot of the DDOS attacks going on, we need to make sure BCP38
compliance goes up. Only way to do this I can think of, is large scale
BCP38 testing. One way of doing this, is to have large projects such
as OpenWRT, RIPE Atlas project, perhaps even CPE vendors, implement
something that would spoof 1 packet per day or something to a known
destination, and in this packet the "real" source address of the
packet is included.

    A proof of concept could be as simple as a basic BCP38 test
implemented into the OpenWRT/DD-WRT UI.

I have been getting pushback from people that this might be "illegal".
Could anyone please tell me what's illegal about trying to send a
packet with a random source address?

    You could reserve yourself an IP address in a subnet you own and use
that as a source IP for testing.

If we can get consensus in the operational world that this is actually
ok, would that help organisations to implement this kind of testing. I
could see vendors implement a test like "help verify network stability
and compliance, these tests are anonymous" checkbox during the initial
install, or something like this.

In short:

    . Test Client call the BCP38 Test Server for a Token;
    . Test Client send a test packet with that token in the payload;
    . Test Client call the BCP38 Test Server with the Token and the
server returns pass of fail which is displayed back in the CPE UI;

    While being over simplified, BCP38.org has some perl scripts from
last year NTP DDoS rush that actually does part of this.
   
    If the initial proof of concept get traction, a more complete BCP38
test set can be implemented, there is a few projects out there that
could be approached for it.

Why isn't this being done? Why are we complaining about 300 gigabit/s
DDOS attacks, asking people to fix their open resolvers, NTP servers
etc, when the actual culprit is that some networks in the world don't
implement BCP38?

    "most networks in the world"

    BCP38 compliance is the exception not the norm.

    PS: About that uRPF Convo, we could dump all that knowledges into
lets say... some comprehensive wiki page maybe =D That way when the
topic arise we could just link to it.

8

I'm not sure that's actually the case, practically-speaking.

NAT is an awful thing for many reasons, and it's negative in terms of its overall impact on security, but there's one actual benefit from it; it is effectively a form of anti-spoofing.

The prevalence of NAT on consumer broadband access networks means that those networks do not generally emit spoofed packets. The same goes for many SME networks, even though they oughtn't to be running NAT in front of their servers.

My guess is that the majority, if not all, of the reflection/amplification attacks we see are actually initiated from servers under the control of attackers and residing on hosting/co-location IDC networks which don't enforce anti-spoofing at the access, distribution, or peering/transit portions of their topologies. Some of these servers are tied into so-called 'booter' systems, whereas others are linked into more conventional C&C under the direct control of attackers, while still others are utilized to launch attacks 'by hand', as it were.

Those networks are unmanaged and are likely to remain so (or are so-called 'bulletproof' networks catering to criminals). Their peers/upstream transits likewise are not enforcing anti-spoofing on ingress, nor are they monitoring traffic originating from these networks as it ingresses their own networks (and in any event, the traffic volume of the spoofed packets on the attack source - reflector/amplifier leg is relatively small).

So, the problem is that those networks which are likely to implement the various topologically-appropriate at the various edges of their network are likely to have done so. And by definition, the endpoint networks where the spoofed traffic originates aren't likely to do so, nor are their immediate peers/upstream transits - or they would've done so long ago.

9

   BCP38 compliance is the exception not the norm.

I'm not sure that's actually the case, practically-speaking.

NAT is an awful thing for many reasons, and it's negative in terms of its overall impact on security, but there's one actual benefit from it; it is effectively a form of anti-spoofing.

The prevalence of NAT on consumer broadband access networks means that those networks do not generally emit spoofed packets. The same goes for many SME networks, even though they oughtn't to be running NAT in front of their servers.

    You are right on that point, I keep forgetting about the little man :frowning:

    My mindset was set on DDoS and not C&C/SPAM/etc.

My guess is that the majority, if not all, of the reflection/amplification attacks we see are actually initiated from servers under the control of attackers and residing on hosting/co-location IDC networks which don't enforce anti-spoofing at the access, distribution, or peering/transit portions of their topologies. Some of these servers are tied into so-called 'booter' systems, whereas others are linked into more conventional C&C under the direct control of attackers, while still others are utilized to launch attacks 'by hand', as it were.
   
    We had the same experience where you get a 1Mbps steam of DDoS
amplification start on the 15th and end abruptly on the 30th at 23h55m
(CC report cycle/reject is often around 15 days). Then on the 5th and
end 15 days later... for a few month in a row.

10

My point is that the ability to launch reflection/amplification DDoS attacks (as well as spoofed SYN-floods, and so forth) is dependent upon the ability to spoof packets, and that my hunch is that there're a relatively small number of networks from which the spoofed attack traffic is emitted.

11

This feels indeed like boiling the ocean, but what are the alternatives,
given that we are looking at a fundamental feature in the Internet
routing system - source address has no practical significance?

But on another side, how easy it is to comply? The best documentation I
found was �RIPE Anti-Spoofing Task Force HOW-TO�,
http://www.ripe.net/ripe/docs/ripe-431, but even that doesn't cover all
cases and needs updating.

Are there are other useful references, besides bcp38/84 and bcp38.info?

Andrei

12

The *real* problem isn't the testing.

It's the assumption that you can actually *do* anything useful with this data.
Name-n-shame probably won't get us far - and the way the US works, if there's a
large cartel of BCP38-compliant providers calling out the offenders by name,
you might encounter an offender that finds it cheaper to send a lawyer chanting
'restraint of trade!' or similar rather than actually fixing their problem....

13

In several jurisdictions in APAC, at least, truth is not a defense against *criminal* defamation charges, much less in civil suits.

And there are more than a few networks in APAC which haven't implemented source-address validation to the degree possible . . .

14

We have not seen support from other customers of our vendors for these features in RFI/RFP. It has taken us sometimes a year (or more) to get software fixes for uRPF related defects. The network performance can be impacted for all users due to the penalty by turning on uRPF as well, so it’s not even technically viable if you want line-rate from certain hardware sets. (See RFI/RFP).

I’ve tried to collect a list of other interested parties to include this in their purchasing process with 0 takers so have put this on the back burner and just kept measuring networks that permit spoofed packets instead.

It’s like any other things (e.g.: BGP hygiene), many people don’t invest the time/though/resources to cause the necessary impact.

- Jared

15

A lot of the discussion on BCP38 seems to be around providers who are unintentionally allowing IP spoofing.

What about providers who knowingly allow IP spoofing, because it's profitable?

There's a provider that basically caters to the DDOS-as-a-service industry by selling servers with unmetered connections, where they allow IP spoofing. (If you've ever looked into this at all, you know exactly who I'm talking about).

16

Ultimately, the only way to even possibly try to get a handle on this facet of the problem may be via lawsuits; in many jurisdictions, the burden of proof is lower for the plaintiffs in a civil case than it is for prosecutors in criminal cases.

The questions of evidence, standing, jurisdiction, provable damages, et. al. then come into play . . . and when those organizations are located in areas where the rule of law isn't particularly strong, it complicates matters further. There are precedents for extraterritorial legal suits, but unless there are assets that can be seized in the event of a verdict for the plaintiff, it's unclear how much actual impact they would have.

[Note: IANAL, nor do I play one on television. All of the above is uninformed speculation and may be completely wrong.]

17

What about providers who knowingly host massive spam operations, because
it's profitable? As in:

  http://www.spamhaus.org/statistics/networks/

We've been down this road before. Unless there is prompt, concerted,
collective action (as there was AGIS) then there is zero reason for those
behind such operations to do anything but keep collecting dirty money.

So it's all good and well to *know* where the problems are: that's
useful. But if the goal is to actually make the problems go away,
then that'll require fingers on keyboards implementing null routing,
firewalling, blacklisting etc.

---rsk

18

> What about providers who knowingly allow IP spoofing, because it's
> profitable?

What about providers who knowingly host massive spam operations, because
it's profitable? As in:

  Spamhaus: The Top 10 World's Worst Abuse Support ISPs

We've been down this road before. Unless there is prompt, concerted,
collective action (as there was AGIS) then there is zero reason for those
behind such operations to do anything but keep collecting dirty money.

So it's all good and well to *know* where the problems are: that's
useful. But if the goal is to actually make the problems go away,
then that'll require fingers on keyboards implementing null routing,
firewalling, blacklisting etc.

Or it will require legislation and I will assure that whatever is
written not be liked. On the other hand everyone one in the country
will be in the same boat.

19

There is, please join:

http://www.routingmanifesto.org/manrs/

Andrei

20

I concur with you -- strongly. Legislation is not the answer, because
(a) it only applies in limited jurisdictions and this is a global problem
and (b) it will inevitably be written by those with the deepest pockets,
see for example CAN-SPAM, crafted by and for spammers and their supporters.

But legislation isn't necessary. Within limits (prescribed by contractual
obligations) none of us are required to offer services to arbitrary
parties. We *choose* to do so, by default, all day every day because that's
why we have an Internet. But we're not *obligated* to do so: those services
may be withheld in full or part at any time for any reason (or even
without a reason).

And this is where I quote the best thing I've ever read on this mailing list:

  If you give people the means to hurt you, and they do it, and
  you take no action except to continue giving them the means to
  hurt you, and they take no action except to keep hurting you,
  then one of the ways you can describe the situation is "it isn't
  scaling well".

  --- Paul Vixie

Having observed, for example, the spam problem since its genesis, I can
unequivocally state that the *only* thing that has ever addressed the
problem (rather than merely addressing its symptoms) is SMTP blacklisting.
Everything else has been ineffective, misdirected, wishful thinking.

The same thing applies here: persistent, systemic sources of large-scale
abuse via BCP-38 noncompliance are either:

  1. Being operated by clueless, negligent, incompetent people
or
  2. Being operated by deliberately abusive people

There are no other possibilities. (Note: "persistent, systemic".
Transient, isolated problems happen to everyone and are not what I'm
talking about here.)

It's difficult to know which of those two are true via external
observation, but it's not *necessary* to know: the appropriate remedial
action remains the same in either case: stop giving them the means.

---rsk