BCP38 For BGP Customers

Mailman List Mirror (Read Only)
1

Hello -

I'm are currently working on getting BCP38 filtering in place for our BGP customers. My current plan is to use the Juniper uRPF feature to filter out spoofed traffic based on the routing table. The mentality would be: "If you don't send us the prefix, then we don't accept the traffic". This has raised some issues amongst our network engineers regarding multi-homed customers.

One of the issues raised was if a multi-homed BGP customer revoked a prefix from one of their peerings, but continued sending us traffic on the link then we would drop the traffic.

I would like to hear what others are doing for BCP38 deployments for BGP customers. Are you taking the stance of "if you don't send us the prefix, then we don't accept the traffic"? Are you putting in some kind of fall back filter in based on something like IRR data?

Thanks!

2

Hey Charles,
My recommendation would not be to run uRPF facing a BGP customer.

That said, you have two issues to address here: one is the acceptance of prefix advertisements, and the other is the acceptance of traffic.

uRPF does nothing to help with the former, and the gold standard there is generally considered to be RPKI. IRR based filtering is another reasonable way to filter prefix advertisements you receive, and several well-known IX’s and transit providers for example do just this.

The latter, acceptance of traffic, is a broader challenge. In essence, you don’t really have a good way to know what traffic is legitimate and what isn’t. My advice would be simply to watch for things you don’t expect, log them when they occur in significant quantity, and manually review incidents that are unexpected to understand why. If you cannot understand why, then you can work with the client sending the traffic to try to understand it, or block that specific traffic from that specific client. uRPF on a client circuit raises exactly the issues you’ve already raised. Many clients, even smaller ones, who choose to run BGP sessions with transit providers will wish to be able to employ common TE practices, and by deploying uRPF, you may very well be creating a nasty situation for them which potentially is also difficult for smaller shops without high end tooling in place to diagnose easily.

  • mdh

Matt Harris​

VP OF INFRASTRUCTURE

Follow us on LinkedIn!

matt.harris@netfire.net

816-256-5446

www.netfire.com

3

Are you taking the stance of “if you don’t send us the prefix, then
we don’t accept the traffic”?

If you were one of my upstreams, and you implemented that, you would very quickly no longer be one of my upstreams.

4

Once upon a time, Charles Rumford <charlesr@deft.com> said:

I would like to hear what others are doing for BCP38 deployments for
BGP customers. Are you taking the stance of "if you don't send us
the prefix, then we don't accept the traffic"? Are you putting in
some kind of fall back filter in based on something like IRR data?

In the case of Juniper, you can use the same prefix-list in your BGP
policy (you are applying a filter to your customers' BGP announcements,
right?) and the uRPF exception list.

5

As it should. This plan will break asymmetric routing which is an
ordinary part of multihoming. Moreover, it would not actually
accomplish BCP 38 since the customer would be able to falsify route
announcements. So, basically a complete fail.

For a small BGP customer who has no downstreams of his own, implement
static filters based on the address ranges you have personally
authenticated as belonging to the customer. PERSONALLY AUTHENTICATED.
This means a manual process. The customer will have to
administratively inform you when those address ranges change.

For large BGP customers who service many BGP downstreams, the bottom
line is that BCP 38 cannot be reasonably implemented. It's one of the
weaknesses in the system.

Regards,
Bill Herrin

6

For large BGP customers who service many BGP downstreams, the bottom line is that BCP 38 cannot be reasonably implemented. It's one of the weaknesses in the system.

Yes, from personal experience BCP 38 should never be implemented buy a transit provider as it will inevitably cause breakage on multi-homed downstream customers for little to no gain and a lot of customer anger. It should be implemented at the customer edge AS, so for a wholesale transit provider is more of a customer education situation. By all means use prefix lists to prevent your customer networks being received anywhere but directly from your customers to prevent them using your capacity without paying for it however.

7

Hi Tony,

Do not do this either as it will render your entire network
unreachable to your customer during an outage of their direct circuit.
Multihomed means you may legitimately receive their prefix
announcement from both their direct link and from your upstream
transit provider.

You CAN, tag announcements received directly from your customers with
a BGP community and then filter routes without that tag when offering
the announcement to your upstream transits. That will have the effect
you're looking for - preventing inappropriate free transit. This is
rarely necessary - unless your network is unusually complex the
additional AS path length of a rebroadcast announcement will generally
prevent such transrouting.

The problem tends to creep in when you have both reciprocal peers and
customers and then a customer's route announcement appears via the
peer. You have to make sure the announcement from the peer is neither
capable of being rebroadcast upstream nor capable of beating the
direct announcement when the direct announcement is present. That
takes some subtle work with BGP communities and route filtering. How
subtle? The routes from the peer may be more specific than the direct
routes.

Regards,
Bill Herrin

8

This may not exist yet, but what about a uRPF-like feature that uses RPKI, IRR, etc. instead of current BGP feed?

9

RPKI and IRR should be part of the prefix-list generation process, from there setup rpf-check with a fail-filter pointing to an ACL that allows source traffic matching the prefix-list and drops the rest. Although at that point you can just apply said ACL to the L3 interfaces supplying the BGP handoff.

Ryan

10

I also have this concern about Spoofing coming from Downstreams.

And after a lot of struggle I can say that using uRPF in strict mode per interface doing FIB lookup is not a good idea!
And I feel sad to have to say that.

I’ve spent a lot of time wrestling with this issue, and the measurement that matters most, which are support tickets, doesn’t show good results.

The best option I’ve found so far?
It is to use the same Prefix-List that you use to release the routes accepted in the BGP session in a Filter Policy applied in the interface with the Downstream.
Another important point to note is that you MUST NOT drop everything else that doesn’t match this Prefix-List.
(Yes, I know it’s hard to accept that…)
But put a bandwidth and PPS control on what doesn’t match the prefix-list, and block what exceeds.
And why do it this way?
Among other reasons, it is very common that Exceeded TTL responses come with source IPs for private use, or IP blocks that are for public use but not announced to the DFZ.

With this method of a Policy-Filter based on the same Prefix-List as BGP and a rate-limit that doesn’t match the prefix-list you won’t block 100% of the spoofed packets that might come from a downstream, but it certainly will. block an eventual DDoS vector coming from this interface.
It is worth remembering that your neighborhood router with Downstream has to support this type of filtering in high capacity. And it is almost certain that only hardware based filtering scenarios will handle this.

11

Hi Mike

This may not exist yet, but what about a uRPF-like feature that uses RPKI, IRR, etc. instead of current BGP feed?

There is rfc8704 that extends urpf
But I do not know of any commercial available solutions

Brian

12

There is work a tthe IETF on an addon to RPKI called ASPA. There is a draft that describes how the combiantion of ASPA and RPKI can be used to help with DDOS prevention.

There is also a working group at the IETF called SAVNET that is looking at what technological additions can be made to address the shortcomings in BCP 38. In fairness, there is distinct disagreement as to what those shortcomings are, and whether the ideas being presented can help. Input from more operators would be great. (For completeness, I am a co-chair of that working group.)

Yours,

Joel

13

I also have this concern about Spoofing coming from Downstreams.

+1

And after a lot of struggle I can say that using uRPF in strict mode per interface doing FIB lookup is not a good idea!

Maybe it's the lack of caffeine, but would someone please remind / enlighten me as to why uRPF is a bad idea on downstream interfaces?

N.B. Maybe I'm thinking more a varient of uRPF wherein if I have a route to the source from the interface in question. Thus not uRPF-strict (active route) nor uRPF-loose (route on any interface).

I've spent a lot of time wrestling with this issue, and the measurement that matters most, which are support tickets, doesn't show good results.

Will you please share a high level of what the technical problems are?

The best option I've found so far?
It is to use the same Prefix-List that you use to release the routes accepted in the BGP session in a Filter Policy applied in the interface with the Downstream.
Another important point to note is that you MUST NOT drop everything else that doesn't match this Prefix-List.

Please clarify if there are routes that would return the packets that would be dropped back to your router & downstream client.

I don't understand why you would want to allow packets that couldn't return the same path.

As for asymmetrically routed packets, I would still expect a return path to exist, even if it's not utilized.

(Yes, I know it's hard to accept that...)

#headScratching

But put a bandwidth and PPS control on what doesn't match the prefix-list, and block what exceeds.
And why do it this way?
Among other reasons, it is very common that Exceeded TTL responses come with source IPs for private use, or IP blocks that are for public use but not announced to the DFZ.

I'll argue -- possibly from a place of ignorance -- that there should not be any packets on the Internet at large (default free zone) to or from IPs not part of the Internet (DFZ).

I view the TTL Exceeded packets as errant in their non-globally-routed source address and as such should be dropped early / often.

Both "be liberal in what you accept and conservative in what you send" (Jon Postel) and "Brown M&M" (Van Halen) come to mind, as does the newer industry phrase "fail-fast".

14

Hi Joel, can you please point us to the IETF draft document that describes how a "combination of ASPA and RPKI can be used to help with DDoS prevention". I was not able to find it.
Thanks!
-Rich

    CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.

    There is work a tthe IETF on an addon to RPKI called ASPA. There is a
    draft that describes how the combiantion of ASPA and RPKI can be used to
    help with DDOS prevention.

    There is also a working group at the IETF called SAVNET that is looking
    at what technological additions can be made to address the shortcomings
    in BCP 38. In fairness, there is distinct disagreement as to what those
    shortcomings are, and whether the ideas being presented can help. Input
    from more operators would be great. (For completeness, I am a co-chair
    of that working group.)

    Yours,

    Joel

15

The Internet Draft is at:

Some slides that will be used to present thematerial on Friday are at:
16

Maybe it's the lack of caffeine, but would someone please remind /
enlighten me as to why uRPF is a bad idea on downstream interfaces?

Hi Grant,

Two words: asymmetric routing.

If the downstream network is architected in such a way that there's
more than one path in and out of their network then there's no way to
guarantee that any particular router believes the forward path to that
network goes to a particular next hop. It can currently map to any
next hop that goes in the direction of one of the valid paths. That
routing is completely independent of how next hops are selected in the
other direction. Packets can travel in one path and out another.

Reverse path filtering literally says don't accept a packet from
somewhere that isn't currently the next hop for that packet's source
address. When it's possible for the forward route to point a different
direction than the one from which valid packets are received, that is
the wrong thing to do. In fact, it breaks the network.

Useful automated reverse path filtering can ONLY be used when there is
exactly ONE valid path to which and from which packets can be
received. This is where strict mode uRPF actually works.

N.B. Maybe I'm thinking more a varient of uRPF wherein if I have a route
to the source from the interface in question. Thus not uRPF-strict
(active route) nor uRPF-loose (route on any interface).

Even if a particular router happens to have RIB entries for all the
valid paths to a host (a sketchy proposition at best), only one such
entry will be stored in the FIB where uRPF looks to make its filtering
decision.

As for loose mode, it's basically useless in a BCP38 discussion. Loose
mode only filters bogons. It doesn't prevent impersonation of any IP
address currently routed in the system and doesn't do anything at all
on a router with a default route.

Regards,
Bill Herrin

17

“Reverse path filtering literally says don’t accept a packet from
somewhere that isn’t currently the next hop for that packet’s source
address.”

FIB or RIB?

I knew of uRPF as available over an interface, per the routing table, not best path available. Or is that implementation dependent?

18

FIB. The RIB is never consulted on a per-packet basis. It's not built
to be fast enough to consult on a per-packet basis.

Regards,
Bill Herrin

19

Grant,

You’re thinking about it from the upstream perspective, where a route
could be accepted but depreferenced and thus not actively used.
Think about it from the downstream network’s perspective, though.
If you’re my upstream, and I don’t want to use your link for inbound
traffic, but I’d like to be able to send out some traffic over the link,
how can I advertise the prefix to you in a way that would both ensure
that you have it in your table locally, so that uRPF is happy, but also
to ensure no packets actually make use of that routing table entry?
Sure, you could tag the routes with ‘no-export’, but that only prevents
the prefix from propagating outward, it doesn’t prevent traffic on that
router from using the routing table entry. You can try adjusting your
MED, and hope the upstream doesn’t squash the MED back to a default
value it applies to all its customers. For the most part, you’re up against
a wall. You don’t know how your upstream’s route selection process is
stacked with respect to routes you advertise, so you have no certainty that
if you announce a prefix to them, it won’t potentially be used to carry all
your inbound traffic.
The only way to be sure that you won’t take inbound traffic on a link is to
not advertise the prefix at all across that link.

Why might this be necessary, you ask?

Imagine you’ve got links of different sizes on your network.
You have a 1G link to provider A
You have a 1G link to provider B
You have a 10G link to cheap provider C
You have a 10G link into a peering exchange

Somewhere beyond provider A, someone decides they don’t like one of your
customers, and sends a 5Gbps DDoS flow at you.

If you continue advertising that prefix to provider A, everyone going through
provider A will suffer, including all their customers. You have plenty of capacity
to take the inbound flow through the exchange point and through cheap provider C.

So, you stop advertising the prefix under attack to provider A and provider B, to ensure
the traffic doesn’t saturate your 1G links.

Inbound traffic happily shifts to the exchange point port and provider C’s port.
Life is good.

Oh no! Provider A has implemented uRPF, and now all their customers are unhappy
because they can’t reach your websites on that prefix, because the return traffic
is still flowing out the 1G link directly to provider A.
Trying to implement a source-based routing filter to redirect all traffic coming from
the prefix under attack destined to ISP A to instead go through ISP C is a pain in the
butt. So, you grudgingly stop accepting routes from ISP A, as that’s the only way to
make the uRPF pain stop.

Now, none of your traffic is flowing out the 1G link to ISP A; their customers are happy
again, because they can reach websites on your prefix that is under attack (via ISP C,
or the exchange point).

At the end of the month, you look at your contracts, and realize that you had to spend
a chunk of your limited engineering resources working around the upstream uRPF filter,
and ultimately ended up not sending traffic across a link you were paying for.

When renewal time comes, you decide it’s not worth the headache to pay for a link to
ISP A that you can’t reliably use, and that requires manual intervention to work around
whenever creative routing solutions are necessary. You don’t renew your contract with
ISP A.

As Mr. Bush might say,
“I recommend all my competitors implement this in their networks.”

Thanks!

Matt

20

>
> Are you taking the stance of "if you don't send us the prefix, then
> we don't accept the traffic"?
>

If you were one of my upstreams, and you implemented that, you would very
quickly no longer be one of my upstreams.

  Yes, I suffer from having two upstreams that each have a shared
transit supplier. They are most likely to only have a single best path
on their network and i can observe in the flow data it's not the one I
expect it to be.

  I'm not sure how that provider (3356) would make it happen.

  I can tell you that the uRPF that 7018 does made me not able to
utilize one of the providers for outbound traffic because they never
opened the proper ticket for routing that IP space until I had
side-escalated to some people that could help me after several months.

  Thankfully it's not a lot of bits but was still annoying to
diagnose and triage.

  - Jared