AT&T/as7018 now drops invalid prefixes from peers

FYI:

The AT&T/as7018 network is now dropping all RPKI-invalid route
announcements that we receive from our peers.

We continue to accept invalid route announcements from our customers,
at least for now. We are communicating with our customers whose
invalid announcements we are propagating, informing them that these
routes will be accepted by fewer and fewer networks over time.

Thanks to those of you who are publishing ROAs in the RPKI. We would
also like to encourage other networks to join us in taking this step
to improve the quality of routing information in the Internet.

Thanks!

            Jay B.

FYI:

The AT&T/as7018 network is now dropping all RPKI-invalid route
announcements that we receive from our peers.

We continue to accept invalid route announcements from our customers,
at least for now. We are communicating with our customers whose
invalid announcements we are propagating, informing them that these
routes will be accepted by fewer and fewer networks over time.

Thanks to those of you who are publishing ROAs in the RPKI. We would
also like to encourage other networks to join us in taking this step
to improve the quality of routing information in the Internet.

Thanks!

Jay B.

Good move AT&T , thanks for taking this on

A round of applause to AT&T for leading the way!

Best regards,
Martijn

Dear Jay, AT&T,

The AT&T/as7018 network is now dropping all RPKI-invalid route
announcements that we receive from our peers.

Thanks for filtering us! :slight_smile:

AT&T doing origin validation combined with the peerlock-style AS_PATH
filters this makes for a pretty strongly protected path between you and
others.

We continue to accept invalid route announcements from our customers,
at least for now. We are communicating with our customers whose
invalid announcements we are propagating, informing them that these
routes will be accepted by fewer and fewer networks over time.

I think this is a sensible strategy.

Thanks to those of you who are publishing ROAs in the RPKI. We would
also like to encourage other networks to join us in taking this step
to improve the quality of routing information in the Internet.

Thank you for paving the way!

If you can share more about the experience in terms of load on the
support tiers in your organisation, or questions from peering partners,
that could perhaps be helpful information for others in their
preparations.

Kind regards,

Job

Jay & everyone AT&T: I just want to say thank you. Kudos to your team for implementing and management for having the intestinal fortitude to do so.

That's great! Do you guys have plans to publish ROAs for your own netblocks? If so, can you please share info on your process (tools, pitfalls, etc.)? Thanks!

This is the best news today! Great job!!

Cheers,
Melchior

Congrats!

Are you able to comment on what amount of routes are getting dropped?

Compton, Rich A writes:
> That's great! Do you guys have plans to publish ROAs for your own netblocks? If so, can you please share info on your process (tools, pitfalls, etc.)? Thanks!
>

Hi Rich,

We do have ROAs published for a not insignificant fraction of our
address space. For example (and cherry-picking the representation
most favorable to us) we're listed at #6 in the "25 Autonomous Systems
with the most Address Space VALID by RPKI" at this NIST RPKI tracker:

https://rpki-monitor.antd.nist.gov/#rpki_adopters

We will publish more ROAs over time. Thusfar we have been utilizing
ARIN's hosted model, but down the road ARIN's delegated model will be
in our future.

https://www.arin.net/resources/rpki/using_rpki.html

Thanks.

            Jay B.

valdis.kletnieks@vt.edu writes:
> > The AT&T/as7018 network is now dropping all RPKI-invalid route
> > announcements that we receive from our peers.
>
> Congrats!

Thanks!

> Are you able to comment on what amount of routes are getting dropped?

In round numbers, we dropped about 5000 invalid prefixes total between
ipv4 and ipv6. Roughly half of those prefixes were covered by
less-specific non-invalid routes, so connectivity should not have been
affected for those prefixes (assuming an announcement yields
reachability to all destinations within it). Flow analysis was
showing just a couple Gbps of traffic to all invalid routes all across
the country, and much less than that with those invalids having no
covering less-specifics.

            Jay B.

Job Snijders writes:
> Dear Jay, AT&T,
>
> > The AT&T/as7018 network is now dropping all RPKI-invalid route
> > announcements that we receive from our peers.
>
> Thanks for filtering us! :slight_smile:

Any time! :slight_smile:

> If you can share more about the experience in terms of load on the
> support tiers in your organisation, or questions from peering partners,
> that could perhaps be helpful information for others in their
> preparations.
>

A few reports of resulting connectivity loss have come in through
various channels: on Jared's outages mailing list, on IRC, through our
customer care ticket system, etc.

Thusfar I have been very pleased with the reactions folks have had
when we described how our policy change caused us to lose their
affected route announcement. Everyone so far has understood the
purpose of the RPKI, they understood why the affected route
announcements were deemed invalid and thus were dropped, and best of
all -- they understood what they needed to do to fix things.

We got some very good advice watching this video from your most recent
NLNOG day:

https://www.youtube.com/watch?v=vrzl__yGqLE

... but there is one place where I disagree with Niels. He advised
against lowering the local-pref of invalid routes. I agree that this
should not be anyone's target policy, but it is a useful step along
the way. To set invalid routes a lower local-pref, one needs to
establish RTR sessions from routers to relying party servers, and to
configure a policy that takes validation state into account. The
policy here can also set community based on the validation state,
which can help with flow-based traffic analysis. Then, when you are
comfortable operating RPs that talk RTR to your routers and you're
ready to implement a meaningful policy, it's a simple matter of
changing from:

if validation-state = invalid
then
   local-pref = $LOWER
   community = foo
fi

to:

if validation-state = invalid
then
   drop
fi

In short: C'mon in! The water's fine! :slight_smile:

Thanks.

            Jay B.

Well done!

Mark.

We got some very good advice watching this video from your most recent
NLNOG day:

https://www.youtube.com/watch?v=vrzl__yGqLE

... but there is one place where I disagree with Niels.

You’re of course welcome to do so :slight_smile:

He advised
against lowering the local-pref of invalid routes. I agree that this
should not be anyone's target policy, but it is a useful step along
the way. To set invalid routes a lower local-pref, one needs to
establish RTR sessions from routers to relying party servers, and to
configure a policy that takes validation state into account.

I agree that this is a good approach for taking first steps into the RPKI world and I would not discourage a lower local preference as a first stage. As long as we’re on the same page about invalid == reject being the intended end result.

In short: C'mon in! The water's fine! :slight_smile:

As a competitive swimmer I couldn’t agree more!

Congrats Jay, this is awesome news!

Compton, Rich A writes:

That's great! Do you guys have plans to publish ROAs for your own netblocks? If so, can you please share info on your process (tools, pitfalls, etc.)? Thanks!

Hi Rich,

We do have ROAs published for a not insignificant fraction of our
address space. For example (and cherry-picking the representation
most favorable to us) we're listed at #6 in the "25 Autonomous Systems
with the most Address Space VALID by RPKI" at this NIST RPKI tracker:

NIST RPKI Monitor

I’m interested to hear what is preventing you from creating ROAs for all of your announcements.

We will publish more ROAs over time. Thusfar we have been utilizing
ARIN's hosted model, but down the road ARIN's delegated model will be
in our future.

https://www.arin.net/resources/manage/rpki/hosted/

What are your main drivers for wanting to move to the delegated model?

Cheers!

-Alex

… but there is one place where I disagree with Niels. He advised
against lowering the local-pref of invalid routes. I agree that this
should not be anyone’s target policy, but it is a useful step along
the way.

For initial deployment, this can seem attractive, but remember that one of the benefits an ROA gives is specifying the maximum prefix length. This means that someone can’t hijack a /23 with a /24.

Lowering local pref on invalid means you’re no longer protected (just generating alerts) because longer prefix length always beats local preference.

Once you are confident that you’re not dropping anything valuable, the local preference rule should move to dropping the route (not the traffic!) from being installed.

M

they can if they forge the source ASN. RPKI helps against misconfigs rather than intentional hijackings.

Nick

Only if you specify a a minlen of /23 and a maxlen of /24 and you only
announce a /23. Which you should not.

Some networks have AS_PATH filters in place that prevent accepting a
spoofed ASN behind an EBGP session that is not authorized to announce
the spoofed ASN. Secondly, there also is a group of networks that
assign the same local preference for all routes received via peering -
meaning that the use of a spoofed ASN will make the AS_PATH one hop
longer. In other words: everyone should peer directly with the
destination networks that matter to them. This is not news of course.
:slight_smile:

I agree some attacks in some cases may still get through, but I've
come to think that ASN spoofing is far less of an issue than I
originally thought it would be.

Kind regards,

Job

How does one register their routes in the Rpki? If the routes are in the Arin database under the proper company name is that sufficient? *Ducks*

TJ and all,

To participate in RPKI with ARIN, your organization would need to have Internet number resources directly registered and those Internet number resources must be covered under a Registration Services Agreement (RSA) or a Legacy RSA. In addition, participation in RPKI will require that you have an ARIN Online account, linked to an Admin or Tech Point of Contact (POC) on the Organization Identifier (Org ID) that contains the Internet number resources to be certified.

Our “Resource Public Key Infrastructure (RPKI)” is a great jumping off point to get started with certifying your Internet number resources.

If you would like any assistance verifying your eligibility for RPKI participation or would like additional information on getting started with RPKC, please call our Registration Services Helpdesk at 703.227.0660. Our hours of operation are Monday – Friday, from 7:00 am to 7:00 pm eastern time.

To be clear, I don’t believe they are dropping all routes which don’t validate (have no ROAs), only routes where the prefix matches an existing ROA and the origin AS in the AS PATH does not match.

Owen