ARIN RPKI TAL deployment issues

Mailman List Mirror (Read Only)
1

Dear NANOG,

I'd like to draw attention to a very concerning development: it appears
that the ARIN TAL is not as widely deployed as other RPKI TALs (such as
RIPE or APNIC's TAL). This means that ARIN members are needlessly put at
higher risk.

Ben Cartwright-Cox performed RPKI research a few weeks ago where he
compared the (un)reachability of RPKI Invalid announcements using ARIN,
RPKI, APNIC and JPNIC space. The full write-up is available here:

    https://blog.benjojo.co.uk/post/are-bgps-security-features-working-yet-rpki

I quote Ben's assessment:

    """Using the data, we can also see that the providers that have not
    downloaded the ARIN TAL. Either because they were not aware that
    they needed to, or could not agree to the agreement they have with
    it.

    This is frustrating to watch. Since it means that ROA signing ARIN
    prefixes will be less secure than ROA signing others, until these
    restrictions are abolished. Even after that it will have a long term
    knock on effect as software updates take a long time to propagate to
    end networks."""

In other words, when creating RPKI ROAs for your resources, ARIN members
are getting *LESS* in return compared to say RIPE members.

As ARIN member I'd hope and expect the ARIN organisation to go above and
beyond to distribute their RPKI TAL as widely as possible. (Think:
proactively submitting the ARIN TAL to relevant open source projects,
making the TAL available for download without restrictions or
limitations).

At the NANOG 74 meeting David Whisnick will talk about legal barriers to
RPKI adoption and he'll offer suggestions for reform. I think Ben's
observation that the ARIN TAL is less widely deployed is a direct result
from the legal barriers that David identified.

    https://pc.nanog.org/static/published/meetings//NANOG74/daily/day_2.html#talk_1767

I fear that if no action is taken (e.g. when none of the RPKI Cache
Validators can include the ARIN TAL in their software distribution [1]),
we'll continue to see the gap between ARIN and the other RIRs widen.
Software developers have indicated that the RPA is problematic and
prevents BGP implementations from shipping "secure by default" software:
https://lists.arin.net/pipermail/arin-consult/2018-April/001087.html

When ARIN members create ROAs, I assume those members would also like
networks *outside* the ARIN region to honor those ROAs and help prevent
propagation of incorrect BGP announcements. The ARIN member and the
operator of the foreign network may not even have any languages in
common! I fear that limited deployment of the ARIN TAL is detrimental to
the business interests of resource holders in the ARIN region.

Kind regards,

Job

[1]: https://github.com/NLnetLabs/routinator/commit/b1e70746bb3768554fa439c5ced4e8b8484eeb00

2

Job -

Is it possible to ascertain how many of those who have not downloaded the ARIN TAL are also publishing ROA’s via RIPE’s CA?

/John

3

I'm sure we could extend the data set to figure this out. But given the
assymmetric relation between applying Origin Validation based on RPKI
data and publishing ROAs, the number will be between 0% and 100% and
over time may go up or down. So, out of curiosity, what is your
underlaying question?

(An example: a route server operator generally doesn't originate any BGP
announcements themselves, but route servers are in an ideal position to
perform RPKI based BGP Origin Validation.)

What I'm hoping for is that there is a way for the ARIN TAL to be
included in software distributions, without compromising ARIN's legal
position.

Perhaps an exception for software distributors would already go a long
way?

    "You can include the ARIN TAL in your software distribution as long
    as you also include an unmodified copy of the
    https://www.arin.net/resources/rpki/rpa.pdf file alongside it."

Kind regards,

Job

4

Job Snijders wrote :
(An example: a route server operator generally doesn't originate any BGP announcements themselves,
but route servers are in an ideal position to perform RPKI based BGP Origin Validation.)

Indeed. Also, an IX should have an RPKI validator accessible by its members, and the RPA makes that impossible.
IMHO, the RPA is too restrictive.

Michel.
TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...

5

Sounds reasonable to me but IANAL, nor an RIR, nor an IXP.

IXPs however do seem to be the sites of some number of recent mis-originations (putting it as charitably as possible).

Let’s try and make it harder for bad actors to do their mischief.

Thanks,
Tony

6

It would be informative to know how many organizations potentially have concerns about the indemnification clause in the RPA but already agree to indemnification via RIPE NCC Certification Service Terms and Conditions.

Thanks!
/John

John Curran
President and CEO
ARIN

7

Dear John,

>
>>>
>>> """Using the data, we can also see that the providers that have not
>>> downloaded the ARIN TAL. Either because they were not aware that
>>> they needed to, or could not agree to the agreement they have with
>>> it.
>>
>> Is it possible to ascertain how many of those who have not downloaded
>> the ARIN TAL are also publishing ROA’s via RIPE’s CA?
>
> I'm sure we could extend the data set to figure this out.

It would be informative to know how many organizations potentially
have concerns about the indemnification clause in the RPA but already
agree to indemnification via RIPE NCC Certification Service Terms and
Conditions.

This seems a matter of personal curiosity that perhaps distracts from
the problem at hand: the ARIN TAL is less widely deployed than the other
TALs.

I'm open to solutions or suggestions to get the ARIN TAL more widely
distributed, however I do think that inclusion in the RPKI Cache
Validators is a *key* element, so the ARIN TAL can be used after a
default installation of such software.

We really need to bring it back down to "apt install rpki-cache-validator"
to best serve the interests of the ARIN members. Imagine the Chrome
browser shipping without any of the TLS Root Certificates, or Unbound
without the DNSSEC root key!

Kind regards,

Job

8

It would be informative to know how many organizations potentially
have concerns about the indemnification clause in the RPA but already
agree to indemnification via RIPE NCC Certification Service Terms and
Conditions.

This seems a matter of personal curiosity that perhaps distracts from
the problem at hand: the ARIN TAL is less widely deployed than the other
TALs.

Job -

I would have to disagree regarding whether question of indemnification is a simply a distraction…
It has been raised by the operator community multiple times, and in fact, you indirectly reference
the same issue by quoting a sentence in the write-up within your post -

I quote Ben's assessment:

    """Using the data, we can also see that the providers that have not
    downloaded the ARIN TAL. Either because they were not aware that
    they needed to, or could not agree to the agreement they have with
    it.”"

I believe that you correctly characterize the situation; i.e:

      1) Either they were not aware they needed it (note this is trivial to fix with outreach/education and requires skills well within the range of anyone who is going to be doing routing validation based on RPKI data), or

      2) They could not agree to ARIN RPA agreement (for which the most cited reason is the indemnification clause, but perplexing given agreement to other indemnification clauses such as RIPE’s Certification services.)

Further analysis and characterization of those not using the ARIN TAL would help enormously in understanding which of the scenarios above is dominant, and prioritize efforts for addressing the situation.

Thanks!
/John

John Curran
President and CEO
ARIN

9

>> It would be informative to know how many organizations potentially
>> have concerns about the indemnification clause in the RPA but
>> already agree to indemnification via RIPE NCC Certification Service
>> Terms and Conditions.
>
> This seems a matter of personal curiosity that perhaps distracts
> from the problem at hand: the ARIN TAL is less widely deployed than
> the other TALs.

I would have to disagree regarding whether question of indemnification
is a simply a distraction… It has been raised by the operator
community multiple times, and in fact, you indirectly reference the
same issue by quoting a sentence in the write-up within your post -

>> I quote Ben's assessment:
>>
>> """Using the data, we can also see that the providers that have not
>> downloaded the ARIN TAL. Either because they were not aware that
>> they needed to, or could not agree to the agreement they have with
>> it.”"

I believe that you correctly characterize the situation; i.e:

1) Either they were not aware they needed it (note this is trivial to
fix with outreach/education and requires skills well within the range
of anyone who is going to be doing routing validation based on RPKI
data), or

My assessment (based on the data Ben collected) is that the current
outreach & education are failing to accomplish the goal of widely
distributing the ARIN TAL, and I fear this may get worse over time.

2) They could not agree to ARIN RPA agreement (for which the most
cited reason is the indemnification clause, but perplexing given
agreement to other indemnification clauses such as RIPE’s
Certification services.)

It may make sense to associate an implicit agreement (or perhaps a
license?) with the ARIN TAL to limit risks to the ARIN organisation.
"Use at your own risk"-style clauses are common and acceptable.

In section 5 ("prohibited conduct") of the RPA I read:

    "You shall not, directly or indirectly, disclose, share, divulge,
    link to, rebroadcast, provide access to or in any other way make
    available the TAL to any third party, except as permitted by the
    ORCP Service Terms."

I believe this prevents OpenBSD, Cisco, Juniper, etc... from shipping
their operating system with a copy of the ARIN TAL, and as far as I
understand is the reason that RIPE NCC's RPKI Cache Validator and the
NLNetLabs RPKI Validator don't include the ARIN TAL either.

Further analysis and characterization of those not using the ARIN TAL
would help enormously in understanding which of the scenarios above is
dominant, and prioritize efforts for addressing the situation.

I'd like to frame the discussion in context of what software developers
and distributors can do to be able to include the ARIN TAL (by default)
rather than focus on getting thousands of organisations to separately
download & install a TAL file. The latter approach doesn't scale. We're
both at NANOG next week, I'd love to sit down and have a cup of coffee.

Kind regards,

Job

10

…

It may make sense to associate an implicit agreement (or perhaps a
license?) with the ARIN TAL to limit risks to the ARIN organisation.
“Use at your own risk”-style clauses are common and acceptable.

Job -

We did look at using a specific set of terms (rather similar to the indeminifcation used by the RIPE in their RPKI Certification Services terms) via a referenced agreement for this purpose, but making that actually legally binding in the US legal environment is a bit challenging.

Absent a specific action (such as our present “browserwrap” approach of specifically downloading the TAL), US courts have not generally recognized mere usage of software as an indication of informed consent to enter into an agreement – hence the RPA download, and why so many things in the US require some form of explicit acknowledgement in order to proceed.

I’d like to frame the discussion in context of what software developers
and distributors can do to be able to include the ARIN TAL (by default)
rather than focus on getting thousands of organisations to separately
download & install a TAL file. The latter approach doesn’t scale. We’re
both at NANOG next week, I’d love to sit down and have a cup of coffee.

Quite understandable - I suspend we’ll hear some more about these issues during David Wishnick’s session, and coffee-fueled discussion afterwards is encouraged.

Thanks!
/John

John Curran
President and CEO
American Registry for Internet Numbers

11

It would be interesting to see how much further deployment would have occurred if ARIN made it’s TAL available similar to the other locations.

Thankfully we have active measurements that show that ARIN prefixes are less protected due to this. As someone that is (for personal reasons) now a voting member of ARIN, this is one of my primary concerns. My ARIN issued space is _less_ protected than if I were to have used another RIR. This devalues that investment.

Instead of asking for an experiment, John I challenge you to make the ARIN TAL available and help play a role in securing the IP space within your region. There is this mantra of Secure by Default that many people have learned since the open-relay, smurf amplification and other attack days. There’s a reason my password isn’t a dictionary word, etc.

If you make it easy to secure a website (eg: Lets Encrypt is a great example) there are now fewer self-signed certificates because it’s easier to do.

Why is ARIN making it so hard for it’s members to get the benefits of the global ecosystem for their RIR controlled space? What makes ARIN IP space so unique in this sense? As part of a global ecosystem it’s incumbent of many of us to do the right thing here and ARIN is increasing the friction on the part of everyone to do the right thing.

If I had to download the ARIN CA in order to interact with www.arin.net vs it being bundled in my browser store, would I be able to securely interact with ARIN?

Therefore, I once again challenge you as part of the leadership of this organization to make the ARIN IP space as protected as those issued by the other regions. Let the developers know that if they bundle the ARIN TAL they won’t face legal action. Help bring routing security to the same ease of use as places like LetsEncrypt do for those in the SSL/TLS ecosystem.

- Jared Mauch
(Representing my own self/WFPL-1)

12

John,

John Curran wrote :
2) They could not agree to ARIN RPA agreement (for which the most cited reason is the indemnification clause, but perplexing given agreement to other indemnification clauses such as RIPE’s Certification services.)

I would entertain that "could not agree to ARIN RPA" is why they don't use the TAL. I may not be representative, but I knew I had to download it.
And maybe you missed a third possibility :
3) Nobody really cares about the ARIN TAL because almost nobody has validated a prefix within the ARIN region therefore installing the ARIN TAL is almost useless :frowning:

We don't only have a problem withTAL deployment, we also have an adoption issue.
And possibly an egg-and-chicken issue : nobody deploys the TAL because nobody validates their prefixes, and vice versa.

Michel.

TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...

13

Actually there are prefixes in the ARIN region with ROAs, and one would presume that issuing the ROA means you want it to be validated as well. (Similar to hosting a website on SSL vs HTTP or even gopher://)

The intent is at least there, and similar to DNSSEC, publishing your DS record in the parent is part of that explicit configured intent.

Saying “nobody validates their prefixes” is patently false. You may not. I may not, but there are a number of networks that are and have advertised that they are.

I’m not saying you need to secure your network, but if you want to secure your routes and have an allocation from ARIN, you really need their TAL to be in the default trust store similar to how you have your PKI trust store in your OS, Browser, etc…

I need my local geographic RIR to care that their anchor is included by default and to make it clear that distributing the TAL is different from _using_ the TAL. Just because I have CA roots in my browser trust store does not mean I am using them all, but if I need to it will work.

On my Mac when I upgrade Xcode it often asks me to accept the License for what I downloaded. The same is true if you use gnu parallel, it outputs some wonderful legalese. There are many comparisons, which is why I’m asking that ARIN permit developers to make it easier for end-users to use the PKI material that makes the global ecosystem more complete and secure. If to start you have to edit the config file to say “I accept arin license for this”=yes would that work? We need that outreach and clarity because at present it’s not there by default and there is a communication gap between the various developers and ARIN.

Those that are issuing ROAs (or are soon to) depend on this. Like I said previously, I’m going to be talking to each ARIN candidate for election this fall about this very topic and what actions they intend to do to support global secure routing.

Michel, It would be a shame if you created a ROA and it could not be validated in some non-english speaking corner of the world that put your assets at risk due to this posture. The community needs secure by default for all regions and the barriers for ARIN IP space are a real and measured problem. It’s time to end this disparity as right now not all TALs are equal. They should be.

- Jared

14

Jared,

Jared Mauch wrote :
Saying “nobody validates their prefixes” is patently false. You may not. I may not, but there are a number of networks that are and have advertised that they are.

I did validate mine, but in the ARIN region I'm part of the only 2% that did, that's close enough to "nobody" for me, in context compared to RIPE numbers.

Michel, It would be a shame if you created a ROA and it could not be validated in some non-english speaking corner of the world that
put your assets at risk due to this posture. The community needs secure by default for all regions and the barriers for ARIN IP space
are a real and measured problem. It’s time to end this disparity as right now not all TALs are equal. They should be.

I agree, but it's not that simple.
The main issue I currently see with RPKI / ROA is not the ARIN TAL (altough I am directly affected) but the fact that nobody or almost nobody actually enforces RPKI. Most operators who are validating RPKI prefixes keep carrying them even when they are invalid, which makes the entire thing completely useless.
And yes I know, it's not that simple :wink:

And it may be shameless self-plugin, but I think we need to encourage experiments that actually try to enforce RPKI.

Michel.

TSI Disclaimer: This message and any files or text attached to it are intended only for the recipients named above and contain information that may be confidential or privileged. If you are not the intended recipient, you must not forward, copy, use or otherwise disclose this communication or the information contained herein. In the event you have received this message in error, please notify the sender immediately by replying to this message, and then delete all copies of it from your system. Thank you!...

15

Jared -

Indeed - In the process of complying with a different legal environment, ARIN sometimes has to behave differently than RIRs that are located elsewhere...

In order to protect the stability of the services we provide to all ARIN customers, we have those relying on ARIN’s certificate services indemnify ARIN from claims of damages in connection with their usage. Such indemnification isn’t unique to ARIN - RIPE has RPKI publishers indemnify RIPE, APNIC has any users of APNIC digital certificates indemnify APNIC, etc.

The significant difference for ARIN is that we operate under a different legal regime, and as a matter of US law, it appears that we cannot rely only upon terms and conditions published in our website as evidence of informed agreement; i.e. within the US legal framework, we need a specific act of acceptance in order to have a binding agreement.

We originally accomplished this binding via an explicit click-box acknowledgement and emailing the the TAL to agreeing party, then managed to evolve it over time to just the click-box acceptance of terms, and now are able to consider the act of simply downloading of the TAL itself as sufficient to constitute agreement to terms.

Different legal regimes result in different implementations, even when the terms (such as indemnification) are similar.

Thanks,
/John

John Curran
President and CEO
American Registry for Internet Numbers

16

Hi, John.

Indeed - In the process of complying with a different legal environment, ARIN sometimes has to behave differently than RIRs that are located elsewhere…

[…]

The significant difference for ARIN is that we operate under a different legal regime, and as a matter of US law, it appears that we cannot rely only upon terms and conditions published in our website as evidence of informed agreement; i.e. within the US legal framework, we need a specific act of acceptance in order to have a binding agreement.

Without venturing too far off topic, can you briefly compare this situation versus e.g. licensing of open source software? Often, such software is (apparently) licensed without express agreement - using bundled license files, comments inside source files, etc - and seems to accommodate the IPR and liability needs of developers and their supporting organizations. Is it ARIN’s understanding that this approach is not useful for RPKI data in the US, etc?

In any case, I also look forward to hearing the Overcoming Legal Barriers to RPKI Adoption talk next week (on Monday afternoon, IIRC), and I hope that the Q&A discussion (and evening follow-up) will be helpful.

Thanks,
-Benson

17

(I’m going to regret posting this later, but…)

18

Without venturing too far off topic, can you briefly compare this situation versus e.g. licensing of open source software? Often, such software is (apparently) licensed without express agreement - using bundled license files, comments inside source files, etc - and seems to accommodate the IPR and liability needs of developers and their supporting organizations. Is it ARIN's understanding that this approach is not useful for RPKI data in the US, etc?

Benson -

Excellent question.

First and foremost, an RIR agreement which provide indemnification (such as RIPE’s RPKI publisher terms, APNIC’s Certificate user terms, and ARIN’s RPA) provides an affirmative defense regarding liability claims; i.e. effectively we are able to point out at the very beginning of proceedings that parties using RPKI data per the respective agreement definitively have all of the associated liability from such use, not the RIR. This allows for a timely disposition by a judge of any liability claims against an RIR regarding such use, which is definitely not the case of a software license agreement.

In the latter case of a software license agreement, if an RIR should suffer an RPKI outage (e.g. RIPE Feb 2013 – https://www.ietf.org/mail-archive/web/sidr/current/msg05621.html), it will be necessary to show that any parties making claims of damages were harmed as the result an an ISP which had a duty to act with a certain level of care with regard to use of RPKI information and who failed in that duty, as opposed to the being the result of the RIR outage. Such an argument must be made to the satisfaction of a jury based on the preponderance of evidence – i.e. even though each ISPs would have signed an agreement to use the RPKI information “as is”, that would not preclude any case proceeding to trial and would not necessarily be sufficient for an RIR to avoid significant liability in the event of an outage and despite the clear disclaimer of “as is” provision of RPKI data.

In any case, I also look forward to hearing the Overcoming Legal Barriers to RPKI Adoption talk next week (on Monday afternoon, IIRC), and I hope that the Q&A discussion (and evening follow-up) will be helpful.

Indeed – note that your question regarding a comparison to “licensing of open source software” might also be asked during that Monday session in order to gain better insight from an actual attorney (rather than my offhand knowledge of such matters...)

Thanks!
/John

John Curran
President and CEO
ARIN

19

Chris -

The “Let’s encrypt” subscriber agreement (current version 1.2, 15 Nov 2018) includes "indemnify and hold harmless” clause, and parties affirmatively agree to those terms by requesting that ISRG issue a "Let’s Encrypt” Certificate to you.

(I don’t know whether that process is particularly more or less onerous technically than the effort to download the ARIN TAL.)

FYI,
/John

John Curran
President and CEO
ARIN

20

The process for lets encrypt is fairly straightforward, it collects some minimal information (eg: e-mail address, domain name) and then does all the voodoo necessary. If ARIN were to make this request of the developers of RPKI software, it would seem reasonable to have that passed to ARIN via some API saying “bob@example.com” typed “Agree” to the ARIN TAL as part of the initial installation of the software.

For me, this is about the friction involved in making it work and while the click-through page may not seem like a barrier, there are active measurements that demonstrate it is. It may take time to communicate to the existing set of operators running RPKI validators they are missing the ARIN TAL, but I would like to ensure that new deployments don’t make this same mistake.

I think this thread/communication is part of that. “Don’t forget about the extra step for ARIN”. It’s also “ARIN, please help make it easier to use your service”.

With Google Maps, etc.. I may have to create an API key, it comes in multi-lingual systems in non-roman alphabet support, etc. Being part of this global ecosystem and running an RIR comes with some extra effort compared to running a corner mom & pop shop. Our actions and decisions have global consequences to the safety and security of how your and my traffic is routed.

Please work with the developers for a suitable method to include the ARIN TAL by default. Come up with the click-accept legalese necessary.

Since you asked, here’s what they did with the CertBot that’s commonly used by Lets Encrypt:

    (The first time you run the command, it will make an account, and ask for an email and agreement to the Let’s Encrypt Subscriber Agreement; you can automate those with --email and --agree-tos)

    If you want to use a webserver that doesn’t have full plugin support yet, you can still use “standalone” or “webroot” plugins to obtain a certificate:

    ./certbot-auto certonly --standalone --email admin@example.com -d example.com -d www.example.com -d other.example.net

If you/ARIN could work closer with the developers of RPKI software to help make this happen that would be great. If you need introductions, I’m happy to help make them.

- Jared