deploying RPKI based Origin Validation

Mailman List Mirror (Read Only)
1

Hi all,

I wanted to share with you that a ton of activity is taking place in the
Dutch networker community to deploy RPKI based BGP Origin Validation.
The mantra is "invalid == reject" on all EBGP sessions.

What's of note here is that we're now seeing the first commercial ISPs
doing Origin Validation. This is a significant step forward compared to
what we observed so far (it seemed OV was mostly limited to academic
institutions & toy networks). But six months ago Amsio (https://www.amsio.com/en/)
made the jump, and today Fusix deployed (https://fusix.nl/deploying-rpki/).

We've also seen an uptake of Origin Validation at Internet Exchange
route servers: AMS-IX and FranceIX have already deployed. I've read that
RPKI OV is under consideration at a number of other exchanges.

Other cool news is that Cloudflare launched a Certificate Transparency
initiative to help keep everyone honest. Announcement at:
https://twitter.com/grittygrease/status/1017224762542587907
Certificate Transparency is a fascinating tool, really a necessity to
build confidence in any PKI systems.

Anyone here working to deploy RPKI based Origin Validation in their
network and reject invalid announcements? Anything of note to share?

Kind regards,

Job

2

It's great to hear that this is catching up. To be honest, I haven't
kept up with the latest goings-on in this space for almost 1 year now,
so I hadn't heard about anyone implementing OV in an "Invalid = Drop"
manner.

Between 2014 - 2016, we (SEACOM, AS37100) deployed and operated (what
was then rpki.net) Dragon Research's RPKI CA and RP tools. I believe the
project has since moved over to GitHub:

https://github.com/dragonresearch/rpki.net

Anyway, the operational issue we ran into was due to our aggressive
policy to drop Invalids. The reason this became an issue was networks
that ROA'd just their aggregates, but either forgot to or decided not to
ROA the longer prefixes that were children of the aggregate.

So suddenly, you have a customer who is multi-homed; one connection was
to us, SEACOM, and the other was to another ISP not doing OV. Our policy
meant the customer was receiving far fewer routes from SEACOM vs. the
other provider, which took traffic away from us (and consequently, $$);
not to mention the NOC-related headache.

After 2 years of struggling to get community traction with OV based on
this policy, we decided to maintain the validation, but remove any
actions being ran against the validation result.

OV where Invalids are dropped is something that all (major, and some
regional, at the very least) ISP's need to enable for it to make a real
difference in terms of:

  * Encouraging all networks to participate, and

  * Reaching the desired outcome, i.e., to mitigate against route leaks
    and hijacks

We also discovered a number of bugs that myself, Randy, Rob, Philip,
Fakrul, Matthias, Jay and Andreas, and a few others at Cisco and Juniper
helped fix in code that shipped between 2016 - 2017.

I would really be keen to hear feedback from you or others that have
decided to deploy OV and drop Invalids. I'm more than happy to get back
on to this wagon in the interest of cleaning up the global BGP table.
But it needs mass...

Mark.

3

Anyway, the operational issue we ran into was due to our aggressive
policy to drop Invalids. The reason this became an issue was networks
that ROA'd just their aggregates, but either forgot to or decided not to
ROA the longer prefixes that were children of the aggregate.

So suddenly, you have a customer who is multi-homed; one connection was
to us, SEACOM, and the other was to another ISP not doing OV. Our policy
meant the customer was receiving far fewer routes from SEACOM vs. the
other provider, which took traffic away from us (and consequently, $$);
not to mention the NOC-related headache.

After 2 years of struggling to get community traction with OV based on
this policy, we decided to maintain the validation, but remove any
actions being ran against the validation result.

Have you considered applying "invalid == reject" on just transit/peering
sessions rather than customer sessions as an intermediate step? I bet
most misconfigurations or hijacks didn't come in via your customers.

I would really be keen to hear feedback from you or others that have
decided to deploy OV and drop Invalids. I'm more than happy to get
back on to this wagon in the interest of cleaning up the global BGP
table. But it needs mass...

Cool!

Kind regards,

Job

4

Yes, we did. The issue is some of our customers did ROA their
aggregates, but not the more-specifics. We didn't want to get into a
situation where we had to custom-design templates depending on what RPKI
mood the customer was in :-).

But yes, the majority of the issue was with routes learned from peers
and transit. That, though, still leaves the problem where you end up
providing a partial routing table to your customers, while your
competitors in the same market aren't. Most customers that aren't keen
on IPv6 or DNSSEC treat RPKI the same way - as a nuisance. So trying to
speak sense into them would be a more treacherous road to take than just
turning it off until we get wider support within the BGP operational
community.

Mark.

5

But yes, the majority of the issue was with routes learned from peers
and transit. That, though, still leaves the problem where you end up
providing a partial routing table to your customers, while your
competitors in the same market aren't.

Ouch.

Most customers that aren't keen on IPv6 or DNSSEC treat RPKI the same way - as a nuisance.

I can see that. :frowning:

So trying to speak sense into them would be a more treacherous road to take than just turning it off until we get wider support within the BGP operational community.

Please forgive the n00b question: But isn't that where carrying the prefixes through your network and conditionally advertising them to customers comes into play?

Or does that run into complications where you must also have the prefixes which don't validate routed in your core?

The reading I did on RPKI / OV yesterday made me think that it is possible to have validated routes preferred over unknown routes which are preferred over invalid routes. So I'd think that you could still have the routes through your core but conditionally advertise the prefixes to customers based on their desires.

I would appreciate it if someone would be kind enough to explain what I'm misunderstanding. Or better, point me to some better documentation to read myself.

Thank you from the peanut gallery.

6

The reading I did on RPKI / OV yesterday made me think that it is
possible to have validated routes preferred over unknown routes which
are preferred over invalid routes. So I'd think that you could still
have the routes through your core but conditionally advertise the
prefixes to customers based on their desires.

you get the option at input (from transit/peering edge say) to evaluate the
'rpki status' of a particular route, then set normal bgp attributes based
on that evaluation, so yes you can:
   valid == localopref 1000 && community-A
   unknown == localpref 80 && community-B
   invalid == localpref 1 && community-Z

but given:
   192.168.0.0/16 - valid
   192.168.0.0/17 - unknown
    192.168.0.0/24 - invalid

your routing system will still forward toward the 192.168.0.0/24 prefix
because 'longest prefix match'.
Job's plan, I think, is that you reject/drop/do-not-accept the 'invalid'
prefix(es) and hope that you follow another / proper path.
Perhaps Mark could send along ONLY the valid/unknown routes to his
customer, or some mix of the set based on what type of customer:
  super-sekure-customer - valid only
  sorta-sekure-customer - valid/unknown
  wild-wild-west-customer - all

it sounded like Mark didn't want to deal with that complexity in his
network, until more deployment and more requests from customers like;
  Customer: "Hey, why did my traffic get hijacked to paY(omlut)pal.com
yesterday?"
  Mark: "because you didn't ask for 'super-sekure-customer config? sorry?"

I could have misunderstood either mark or job or you.. of course.

7

That is exactly what I mean. Because of the golden rule "most-specific
always wins" (and parts of the AS_PATH are pretty easy to spoof) it
only makes sense to me to completely reject invalid routes.

Kind regards,

Job

In Junos speak:

[...]
set policy-options policy-statement IMPORT:PEER term RPKI-DROP-INVALID
from protocol bgp
set policy-options policy-statement IMPORT:PEER term RPKI-DROP-INVALID
from validation-database invalid
set policy-options policy-statement IMPORT:PEER term RPKI-DROP-INVALID
then validation-state invalid
set policy-options policy-statement IMPORT:PEER term RPKI-DROP-INVALID
then reject
[...]

8

you get the option at input (from transit/peering edge say) to evaluate the 'rpki status' of a particular route, then set normal bgp attributes based on that evaluation, so yes you can:

    valid == localopref 1000 && community-A
    unknown == localpref 80 && community-B
    invalid == localpref 1 && community-Z

ACK

but given:
    192.168.0.0/16 - valid
    192.168.0.0/17 - unknown
     192.168.0.0/24 - invalid

your routing system will still forward toward the 192.168.0.0/24 prefix because 'longest prefix match'.

*facePALM*

Thank you.

So the information would be carried across the network, but it still suffers from the same problem.

Job's plan, I think, is that you reject/drop/do-not-accept the 'invalid' prefix(es) and hope that you follow another / proper path.

Yep.

You would almost need separate logical networks / VRF to be able to prevent the longest prefix match winning issue that you reminded me of.

Perhaps Mark could send along ONLY the valid/unknown routes to his customer, or some mix of the set based on what type of customer:

   super-sekure-customer - valid only
   sorta-sekure-customer - valid/unknown
   wild-wild-west-customer - all

Yep. That's what I was thinking of.

it sounded like Mark didn't want to deal with that complexity in his network, until more deployment and more requests from customers like;

Fair.

   Customer: "Hey, why did my traffic get hijacked to paY(omlut)pal.com yesterday?"
   Mark: "because you didn't ask for 'super-sekure-customer config? sorry?"

I could have misunderstood either mark or job or you.. of course.

You understood me correctly.

Thank you for explaining what I was missing.

9

> but given:
> 192.168.0.0/16 - valid
> 192.168.0.0/17 - unknown
> 192.168.0.0/24 - invalid
>
> your routing system will still forward toward the 192.168.0.0/24 prefix
> because 'longest prefix match'.

*facePALM*

Thank you.

So the information would be carried across the network, but it still
suffers from the same problem.

well, consider the situation where Mark's mythical customer(s) are:
  custA: dual-homed + accept default (from both providers)
  custB: dual-homed

(and live in the 'total sekure world' TSW (tm))

CustA may not see the invalid /24 (nor the /17) but have no other path and
"randomly" choose Mark and his /17 + /24 world :frowning:
CustB simply drops packets (aka: what Job wants - again, I think)

So... if we had more CustB and less CustA ... maybe everywhere it's OK for
'Large Mark Providers' - LMP (tm) to provide such services? I've not looked
in a 'long time', but when I worked at a large ISP ~30-35% of our customers
did BGP with us, of that ~70+% just did it with us (dual / redundant links
to us). I think 'almost all' took a default from us too.. whether they used
that default I can't say.

I think getting to Job's world is a goal, I think living in Mark's is a
reality for a bit still.
(yes, you could ALSO do some game playing where the customer ports for TSW
were in a VRF with no 'bad' routes, but.. complexity)

> Job's plan, I think, is that you reject/drop/do-not-accept the
'invalid'
> prefix(es) and hope that you follow another / proper path.

Yep.

You would almost need separate logical networks / VRF to be able to
prevent the longest prefix match winning issue that you reminded me of.

yup, yea... complexity though :frowning:

> Perhaps Mark could send along ONLY the valid/unknown routes to his
> customer, or some mix of the set based on what type of customer:
>
> super-sekure-customer - valid only
> sorta-sekure-customer - valid/unknown
> wild-wild-west-customer - all

Yep. That's what I was thinking of.

> it sounded like Mark didn't want to deal with that complexity in his
> network, until more deployment and more requests from customers like;

Fair.

> Customer: "Hey, why did my traffic get hijacked to paY(omlut)pal.com
> yesterday?"
> Mark: "because you didn't ask for 'super-sekure-customer config?
sorry?"
>
> I could have misunderstood either mark or job or you.. of course.

You understood me correctly.

Thank you for explaining what I was missing.

sure thing! (err, this rpki/secure-routing business isn't really super
intuitive :frowning: )

10

Please forgive the n00b question: But isn't that where carrying the
prefixes through your network and conditionally advertising them to
customers comes into play?

Or does that run into complications where you must also have the
prefixes which don't validate routed in your core?

Carrying prefixes in the network is not an issue, valid or otherwise.

If you act on them as they enter the network in an aggressive manner,
then the other end of an eBGP session will not receive them. That's the
issue. Of course, that's how RPKI is supposed to work, but when you're
the only one doing it, you're shooting your own foot.

The reading I did on RPKI / OV yesterday made me think that it is
possible to have validated routes preferred over unknown routes which
are preferred over invalid routes. So I'd think that you could still
have the routes through your core but conditionally advertise the
prefixes to customers based on their desires.

Using LOCAL_PREF to (de)prefer routes based on their validation status
is an idea that has been used since 2014. But for me, it defeats the
purpose if you are going to go soft when trying to implement something
that requires this much resolve to clean up the Internet.

Mark.

11

I didn't want to pass on Invalid routes at all, to ensure that the
source operator of that route correctly signs it in the RPKI. However,
one can't make the horse drink.

Using LOCAL_PREF to determine the preference between Valid, Unknown and
Invalid routes is just pussy-footing around the feature, if I'm being
honest.

What's the saying... "Go big, or go home" :-).

Mark.

12

Exactly my preference, and exactly what we did for 2 years. But in
practice, customers don't really like this, nor does your CFO.

We need mass deployment for this to work effectively, and also a bit
more education for those that sign aggregates but not the more-specifics.

Mark.

13

Oooh, complexity - things we want to avoid :-).

Then again, we don't run the Internet in a VRF, so...

Mark.

14

I think getting to Job's world is a goal, I think living in Mark's is a
reality for a bit still.
(yes, you could ALSO do some game playing where the customer ports for TSW
were in a VRF with no 'bad' routes, but.. complexity)

This summarizes the current status of affairs quite accurately.

I'd like to get to the point where RPKI is widely deployed so that we
can all run a cleaner BGP. I don't think that waiting for all BGP
operators to enable RPKI and drop Invalids will be the solution. So if
the top 7 global operators decided to do it, and perhaps suffer the pain
of the effects for a few months, the rest of the community will be
inclined to follow suit.

Kind of like how only a few major operators really use RPSL, which
forces all BGP operators to keep some kind of updated IRR, even if they,
themselves, may not be RPSL users.

sure thing! (err, this rpki/secure-routing business isn't really super
intuitive :frowning: )

As always, the difficult bit is done, i.e., the protocol spec. is
clearly defined, there is code in routing software, and there are plenty
of options for Route Validation software.

But as always, the hard part is getting the community to implement, as
we've seen with IPv6 and DNSSEC.

Mark.

15

In the RIPE part of the world there is no excuse for not getting RPKI
correct because RIPE made it so easy. Perhaps the industry could agree on
enabling RPKI validation on all european circuits for a start?

Regards

Baldur

16

I think the first step (and what I'd consider to be a quick win) is if
we determined all the prefixes that are being designated Invalid, and
nail down how many of those are Invalid due to the fact that they are
more-specifics announced without a ROA, vs. the parent aggregate which
is ROA'd.

We would then ask the operators of those prefixes to either withdraw
them (easier, but unlikely) or sign them in the RPKI and create ROA's
for them (more work, but more likely). Going for the latter.

Once that is fixed, and even though the entire BGP world is not running
RPKI, those that are and are dropping Invalids would be 100% certain
that those Invalids are either leaks or hijacks.

I think that will get us 50% of the way there, with the other 50% would
now just be growing community participation in RPKI.

Thankfully, I believe all (or most) of the RIR's support a simple "click
of a button" to say "All prefixes up to a /24 or a /48 of the aggregate
should automatically be ROA'd if the aggregate, itself, is ROA'd". So it
shouldn't be a lot of work to get what is currently broken fixed. And
the beauty, we don't need everyone to participate in the RPKI today for
those that want the benefit right now to enjoy it so.

Mark.

17

I actually view it as a competitive advantage to carry a cleaner set of
routes compared to the providers with a more permissive (or lack of)
filtering strategy. Sometimes less is more.

Kind regards,

Job

18

* When you consider your addressable market 'clueful customers'.

19

Typically, I wouldn't disagree.

In practice, most customers only care about reachability, and not their
contribution to the Internet hygiene. A case of "They will look after
it", where "they" is not "me".

Mark.

20

> In the RIPE part of the world there is no excuse for not getting
> RPKI correct because RIPE made it so easy. Perhaps the industry
> could agree on enabling RPKI validation on all european circuits for
> a start?

I think the first step (and what I'd consider to be a quick win) is if
we determined all the prefixes that are being designated Invalid, and
nail down how many of those are Invalid due to the fact that they are
more-specifics announced without a ROA, vs. the parent aggregate which
is ROA'd.

I calculated this here few days ago
http://instituut.net/~job/rpki-report-2018.07.12.txt

Markus Weber from KPN is generating a daily report here and drew similar
conclusions: https://as286.net/data/ana-invalids.txt Markus scrapes all
routes from the AS 286 PEs and marks the routes for which no valid or
unknown alternative exists as "altpfx=NONE".

We would then ask the operators of those prefixes to either withdraw
them (easier, but unlikely) or sign them in the RPKI and create ROA's
for them (more work, but more likely). Going for the latter.

Or delete the incorrect RPKI ROA. Either way is fine.

Once that is fixed, and even though the entire BGP world is not
running RPKI, those that are and are dropping Invalids would be 100%
certain that those Invalids are either leaks or hijacks.

I think that will get us 50% of the way there, with the other 50%
would now just be growing community participation in RPKI.

Thankfully, I believe all (or most) of the RIR's support a simple
"click of a button" to say "All prefixes up to a /24 or a /48 of the
aggregate should automatically be ROA'd if the aggregate, itself, is
ROA'd". So it shouldn't be a lot of work to get what is currently
broken fixed. And the beauty, we don't need everyone to participate in
the RPKI today for those that want the benefit right now to enjoy it
so.

Perhaps the RIRs should start an outreach program to proactively inform
the owners of those 2,200 invalid route announcements to get them to
either fix or delete the RPKI ROA.

Kind regards,

Job