Ingress filtering on transits, peers, and IX ports

Mailman List Mirror (Read Only)
1

We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.

This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.

It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.

We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:

* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :slight_smile:

The questions I wanted to ask the SP community are:

* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?

Thanks a lot,

-Brian

2

You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.

RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :slight_smile:

Here’s a Cisco brief on the topic:

https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding

I think all router vendors support this feature. Here’s a similar article by Juniper:

https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html

-mel beckman

3

Matt Harris​

Infrastructure Lead Engineer

816‑256‑5446

Direct

Looking for something?

Helpdesk Portal

Email Support

Billing Portal

We build and deliver end‑to‑end IT solutions.

4

That’s an interesting suggestion

There are 2 modes for uRPF. Loose and strict.

Which one would you recommend in this scenario and why?

There are many ways to solve this and definitely uRPF is one layer of defense. But, probably not the best alone. I advocate a 3 layers approach.

I’m curious to hear/read which uRPF would you recommend for this particular case.

Thanks

Jean St-Laurent

5

Hi Mel,

My understanding of uRPF is:

  • Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received

  • Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.

Strict mode won’t work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.

Loose mode won’t work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.

Or am I missing something?

Thanks,

-Brian

6

Hi, Brian

Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
by: RFC8704 Enhanced Feasible-Path uRPF).

    Ingress Access Lists require typically manual maintenance, but are
    the most bulletproof when done properly; typically, ingress access
    lists are best fit between the edge and the ISP when the
    configuration is not too dynamic if strict RPF is not an option,
    between ISPs if the number of used prefixes is low, or as an
    additional layer of protection

Ingress filters Best Practices
https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
*Don’t accept BOGON ASNs
*Don’t accept BOGON prefixes
*Don’t accept your own prefix
*Don’t accept default (unless you requested it)
*Don’t accept prefixes that are too specific
*Don’t accept if AS Path is too long
*Create filters based on Internet Routing Registries

And also BGP Best Current Practices by Philip Smith
http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf

Regards.

7

Specifically with regards to “Don’t accept your own prefix”, this poses an interesting challenge for the original notice sent out by the security researchers.

It is not uncommon today for various content networks (and others) to operate multiple “island networks” with a single ASN. For example, AS65001 operates in Los Angeles and New York, with no internal connectivity between them - only connectivity via the Internet (loop prevention disablement is done by every tier 1 provider upon request). Los Angeles advertises 192.168.10.0/24 and New York advertises 192.168.20.0/24 to the Internet, and both have relevant anti-spoofing filters (e.g. Los Angeles has an ingress filter to drop traffic with source IP of 192.168.10.0/24, and New York has an ingress filter to drop traffic with a source IP of 192.168.20.0/24). Traffic between Los Angeles and New York is not filtered as they need to legitimately pass traffic to each other over the Internet. This triggers a false positive detection by the system in question that sent the original notification email.

After discussing this with the people running this project and highlighting that this will be generating false positive data as part of their research (and probably quite a lot, this practice is fairly common), the response was to establish some form of tunnel between the AS islands over the Internet. Not realistic for a bunch of the content companies who practice this design pushing tens/hundreds of Gbps over the Internet (if not more). There seemed to be no interest in the discussion that the data being generated by this system is arguably flawed.

Tl;dr - definitely don’t accept your own prefix from the site it originated from, or other sites that have internal connectivity. But also don’t assume that an AS has a full-mesh of internal connectivity behind it and shouldn’t accept its own prefixes for any reason.

8

Aside from the BCPs currently being discussed for ingress filtering, I would be very interested in seeing what this traffic looked like from the perspective of your DNS servers’ logs.

I assume you’re talking about customer facing recursive/caching resolvers, and not authoritative-only nameservers.

Considering that one can run an instance of an anycasted recursive nameserver, under heavy load for a very large number of clients, on a $600 1U server these days… I wonder what exactly the threat model is.

Reverse amplification of DNS traffic returning to the spoofed IPs for DoS purposes, such as to cause the nameserver to DoS a single /32 endpoint IP being targeted, as in common online gaming disputes?

What volume of pps or Mbps would appear as spurious traffic as a result of this attack?

9

Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:

Considering that one can run an instance of an anycasted recursive
nameserver, under heavy load for a very large number of clients, on a $600
1U server these days... I wonder what exactly the threat model is.

A customer forwarded one of these notices to us - looked like it's about
recursive DNS cache poisoning. It's been a while since I looked
closely, but I thought modern recursive DNS software was pretty
resistant to that, and anyway, the real answer to that is DNSSEC.

I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email
from some group I've never heard of (and haven't AFAIK engaged the
community about their "new" attack, scans, or notices)... seems more
like shameless self promotion.

10

If I had a dollar for every ‘scary security alert’ email received in a NOC email inbox from a ‘security researcher group’ that is the results of a port scan, or some small subset of trojan infected residential endpoint computers attempting outbound connections on ($common_service_port), or similar…

11

I get stupid automated "abuse" notices all the time about being an evil haxx0r, which is actually just having egress proxy enabled on GGC.

The most crazy email I've had so far was saying that I "breached Section 4 of the Terms and Conditions of the domain" and that my as (the AS the GGC nodes are behind) is "to immediately cease and desist" followed by a bunch of BS about how their IP addresses are restricted and no crawl rights have been granted blah blah blah.

12

there's no "almost" about it: strict mode is unfeasible for both transit and IX ports.

Nick

13

While I can understand some reasons why people don't do it, I believe the proper thing to do in this case is have multiple ASNs - one for each island.

They obviously have distinct routing policy and thus qualify at least under ARIN policy AFAIK. With AS4, we don't have any imminent shortage of ASNs and don't need to be particularly stingy about allocating them as long as a need is met.

14

Hi Marcos,

Thanks for your reply. But I am looking for guidance on traffic filtering, not BGP prefix filtering.

I have looked at BCP 84, and it's a good overview of the methods available to an ISP. My questions are more operational in nature and are not covered by the document. Of the choices presented in BCP 84, what do folks really use? If it's an ACL, what challenges have there been with updates? etc.

-Brian

15

Hi Eric,

I shot a message over the folk who did the testing for more info about their test. If I’m able to find anything useful in our logs from their detail, I’ll post it to the list.

The message referenced DNS cache poisoning and DDOS amplification, so it seemed fairly general and more focused on whether ASes accepted spoofed traffic. They also referenced the new NXNSAttack, which I did not know about previously.

Thanks,

-Brian

16

Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:

Considering that one can run an instance of an anycasted recursive
nameserver, under heavy load for a very large number of clients, on a $600
1U server these days... I wonder what exactly the threat model is.

A customer forwarded one of these notices to us - looked like it's about
recursive DNS cache poisoning.

In part, yes. More generally, it's about allowing spoofed-source packets in your front door, appearing to be from your own network, and what a bad actor could do with them. The probes from the experiment were harmless. But if there were malicious intent, this access could facilitate cache poisoning, depending on your DNS server configuration.

It's been a while since I looked
closely, but I thought modern recursive DNS software was pretty
resistant to that, and anyway, the real answer to that is DNSSEC.

It is. But DNSSEC requires support both on the side of the resolver (validation enabled) and the authoritative server (zone signed). Adoption is still far from universal. There are efforts to improve that, but it can't be your only hope, in its current state.

But, perhaps more importantly, cache poisoning is not the only concern. Other vulnerable DNS (for example) configurations might be exploited by a single packet being received and acted on as "trusted".

I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email

Crafting wording in an alert email such that it should both be taken seriously and it doesn't sound too dramatic is hard. We have gotten many positive responses. But we've also gotten some *meh*. In the end we made a choice about whether individual reach-out was important and worth the effort, ahead of future publication and presentation. We decided that it was. Many operators have agreed with us. But I get that not everyone will feel the same about it.

from some group I've never heard of (and haven't AFAIK engaged the
community about their "new" attack, scans, or notices)

I suppose it depends on your definition of "engage the community". I think that's what we're doing right now. We're also no stranger to NANOG (though perhaps more of a lurker on the mailing list). But community is a much broader term. And anyway, there is some order to this whole thing, and broader announcements will come later.

Cheers,
Casey

17

Great for customer-facing interfaces, though.

18

You'll be surprised at the garbage you would drop that you can't return.

  - Jared

19

I too would like to know more about their methodology and actual tangibles ideally in the form of PCAPs.

20

Hi Bryan,

I too would like to know more about their methodology

We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.

Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.

and actual tangibles ideally in the form of PCAPs.

What do you mean by "tangibles in the form of PCAPs"?

Casey