Public shaming list for ISPs announcing other ISPs IP space by mistake

The italian courts seem to have told ISPs there to block ThePirateBay (bittorrent tracker), and this evening (CET) LLNW (AS22822) originated
88.80.6.0/24 via 6762 (telecom italia) to what I presume is most of Europe.

Basically same thing that happened when people tried to block YouTube a few months back (afghanistan?).

How do we hinder this in the short term? I know there are a lot of long term solutions that very few is implementing, but would the fact that these mistakes are brought up into the (lime)light by a public shaming list make ISPs shape up and perform less mistakes?

I am still waiting for a response from LLNW NOC on the issue.

Sure. I'd also like to see providers actually just shut
off customers that originate stuff like ms-sql slammer
packets still. But it keeps flowing. I'm sure there are
smurf amps and other badness still going. codered anyone?

  these are all issues, but operational? depends.
If LLNW is not being filtered by telecom italia, time for
6762 to fix that. If they persist, will you depeer them
as a security risk until they clean up their act?

  I'm still amazed at the AS_PATHs that appear
out there and the providers that can't figure out how to
route.

  Why AS174 would listen to 3549 routes from AS12713
is beyond me, but it's there.[1]

221.134.222.0/24 1280 174 12713 3549 2914 9498 9583

  - jared

1 - http://puck.nether.net/bgp/leakinfo.cgi
  - http://puck.nether.net/bgp/stats.cgi?days=3

The italian courts seem to have told ISPs there to block ThePirateBay
(bittorrent tracker), and this evening (CET) LLNW (AS22822) originated
88.80.6.0/24 via 6762 (telecom italia) to what I presume is most of
Europe.

Basically same thing that happened when people tried to block YouTube a
few months back (afghanistan?).

How do we hinder this in the short term? I know there are a lot of long
term solutions that very few is implementing, but would the fact that
these mistakes are brought up into the (lime)light by a public shaming
list make ISPs shape up and perform less mistakes?

I am still waiting for a response from LLNW NOC on the issue.

  Sure. I'd also like to see providers actually just shut
off customers that originate stuff like ms-sql slammer
packets still. But it keeps flowing. I'm sure there are
smurf amps and other badness still going. codered anyone?

  these are all issues, but operational? depends.

I beg to differ, this is absolutely operational.

If LLNW is not being filtered by telecom italia, time for
6762 to fix that. If they persist, will you depeer them
as a security risk until they clean up their act?

De-peering won't help if someone is propagating it as a transit customer route. Filtering the prefix is all you can do.

I just discovered (via bgplay) that 22822 first originated the prefix via 5511 (france telecom), then it was withdrawn a while later, and then originated via 6762, and then withdrawn again. An hour or so between these events.

We all know we don't filter our peers (there is no operationally sane way of doing this today), so the question is how to make ISPs filter their customers more sanely.

We have prefix-filters on our customer bgp sessions, so that should be fairly safe, but I see no good way of doing this towards peers as there is no uniform way of doing this, and there is no industry consenus how it should be done.

The italian courts seem to have told ISPs there to block ThePirateBay
(bittorrent tracker), and this evening (CET) LLNW (AS22822)
originated
88.80.6.0/24 via 6762 (telecom italia) to what I presume is most of
Europe.

Basically same thing that happened when people tried to block
YouTube a
few months back (afghanistan?).

How do we hinder this in the short term? I know there are a lot of
long
term solutions that very few is implementing, but would the fact that
these mistakes are brought up into the (lime)light by a public
shaming
list make ISPs shape up and perform less mistakes?

I am still waiting for a response from LLNW NOC on the issue.

  Sure. I'd also like to see providers actually just shut
off customers that originate stuff like ms-sql slammer
packets still. But it keeps flowing. I'm sure there are
smurf amps and other badness still going. codered anyone?

  these are all issues, but operational? depends.

I beg to differ, this is absolutely operational.

  So, I should shut down or depeer networks that
continue to originate the crap to me? (packets, announcements).

If LLNW is not being filtered by telecom italia, time for
6762 to fix that. If they persist, will you depeer them
as a security risk until they clean up their act?

De-peering won't help if someone is propagating it as a transit customer
route. Filtering the prefix is all you can do.

  Taking this example, if I were to depeer 6762 becuase
they can't keep their routing table clean to me, I suspect
they would look at how to fix the issue. I could just filter their
as-path globally until they contact me to resolve the issue.

  I'm not saying I would actually do that, but there is
a question of what level of action should be taken to resolve
these issues, and a timescale for their resolution. I've found
some networks excellent to work with, and others "we'll stop
leaking to you in a few days once we finish escalating
the issue to our tier-n NOC in XXX city".

  Honestly, I find that to be kinda lazy considering how
critical the routing infrasturcture is for our survival as an
industry.

P.S. Obligatory BCP38 shout-out, even though it's not exactly on-point.

  I can't agree with this more, If you're not doing u-RPF on your
customer interfaces (t1, ds3, ge, fe, colo, etc..) you should be. The only
excuses are broken software or incapable hardware from your vendor.

  Sadly those last two seem to impair the ability to take these
basic network security requirements into account for a network of any
size.

  It'll help reduce the possible attack home-base for various spoofing
attacks (including some DNS one, did you hear about it?)

  - Jared

Can't IANA give $100000 stupidity tax or revoke AS for people that do this?

Read your peering contract with the other ISP. It should cover what to do
if this happens.

What? you don't have a peering contract with the other ISP. Well I guess there is no requirement to keep the peering session established if the peer does stuff you don't want on your network.

If it hurts when you do something, why do you keep doing it?

Can't IANA give $100000 stupidity tax

perhaps those of us in glass houses should not suggest a major throwing
of stones? :slight_smile:

  Sure. I'd also like to see providers actually just shut
off customers that originate stuff like ms-sql slammer
packets still. But it keeps flowing. I'm sure there are
smurf amps and other badness still going. codered anyone?

  these are all issues, but operational? depends.

I beg to differ, this is absolutely operational.

  So, I should shut down or depeer networks that
continue to originate the crap to me? (packets, announcements).

Saying something is Operational (and on-topic for nanog) does not mean you should de-peer them.

That said, I will not stop you from de-peering a network who can't keep its table clean. Your network, your decision.

If LLNW is not being filtered by telecom italia, time for
6762 to fix that. If they persist, will you depeer them
as a security risk until they clean up their act?

De-peering won't help if someone is propagating it as a transit customer
route. Filtering the prefix is all you can do.

  Taking this example, if I were to depeer 6762 becuase
they can't keep their routing table clean to me, I suspect
they would look at how to fix the issue. I could just filter their
as-path globally until they contact me to resolve the issue.

You wield a much bigger hammer than 99.999% of the people here, and you know it.

  I'm not saying I would actually do that, but there is
a question of what level of action should be taken to resolve
these issues, and a timescale for their resolution. I've found
some networks excellent to work with, and others "we'll stop
leaking to you in a few days once we finish escalating
the issue to our tier-n NOC in XXX city".

  Honestly, I find that to be kinda lazy considering how
critical the routing infrasturcture is for our survival as an
industry.

While I doubt "shame" will work in all but the most extreme cases, I believe brokeness does, eventually have an impact. Let's just hope that impact is not blunted by (for instance) monopoly power, so that "voting with your wallet" will force network to fix things.

Too bad we know monopoly power is blunting most of the effects. :frowning:

P.S. Obligatory BCP38 shout-out, even though it's not exactly on-point.

  I can't agree with this more, If you're not doing u-RPF on your
customer interfaces (t1, ds3, ge, fe, colo, etc..) you should be. The only
excuses are broken software or incapable hardware from your vendor.

  Sadly those last two seem to impair the ability to take these
basic network security requirements into account for a network of any
size.

  It'll help reduce the possible attack home-base for various spoofing
attacks (including some DNS one, did you hear about it?)

Just thought I'd say "BCP38" again.

two things:

  1) I didn't mean to call out any specific provider, we all
have challenges. Sorry to my friends at Cogent that may have been
offeneded.

  2) I think some people have been a bit too lax in enforcing
their peering policies on this topic. Letting something leak for a few
hours may not matter much for some small business or corner of the world.
Leaking something important, or being nasty with it could be really bad.
Imagine instead of spoofing some nameserver, annoucing the space and
being rogue long enough to push out some huge TTL. Take whitehouse.gov
out for the next 30 days..

  Would make life interesting. I can think of other badness to do
but won't enumerate it here.

  - Jared (dinner time!)

My thoughts on the prefix filtering issue would be that we need some kind of system that works along the same principles as DNSSEC and SPF, ie a holder of IP space can publish that they would like everybody to filter in a certain way for announcements for that perticular prefix, and then the other end can do so if they want to. This needs to be automatic and quick, ie a change in RADB policy should be reflected in the real world in minutes (preferrably) or hours (more realistically perhaps) and not in days or months.

This might already be in place, I don't know other than RIPE, but in RIPE you can register a route object with a certain IP space, and IP space can have multiple route objects. The thing here is that to implement this policy in IOS creates *huge* rulesets that are really cumbersome and cluttery. Also, people tend not to rebuild these very often, so for customer routes, doing a handover between ISPs when moving PI space might involve outages and days of limited connectivity. Also, change of policy doesn't isn't reflected unless a route is cleared (soft though) which involves re-hashing a lot of routes very often if filters are updated often.

I also think that the information in RADB doesn't really contain everything I would need in it, so it might need to be extended, but this is easier than getting a new framework into our routers (routing policy server?) that might work in near realtime. Is there work being done that could realistically be implemented anytime soon? Does benefit outweigh the potential catastrophies that might happen when rolling out and running such a system?

Perhaps it's too late for IPv4 in this aspect, but it might be feasable for IPv6? Fewer prefixes and (hopefully) no break-outs, would mean PA blocks could be filtered hard, and if larger ISPs do hard filtering based on RADB, ISPs getting into IPv6 would need to get their prefixes properly registered there before getting IPv6 working to any extent.

Anyhow, as people are continuing to use null routes to enforce regulatory demands (likely cause for the latest outages) this problem will most likely escalate.

This would also help with <http://eng.5ninesdata.com/~tkapela/iphd-2.ppt> problem I guess.

Point taken :wink:

How do we hinder this in the short term? I know there are a lot of long
term solutions that very few is implementing, but would the fact that
these mistakes are brought up into the (lime)light by a public shaming
list make ISPs shape up and perform less mistakes?

My thoughts on the prefix filtering issue would be that we need some kind
of system that works along the same principles as DNSSEC and SPF, ie a
holder of IP space can publish that they would like everybody to filter
in a certain way for announcements for that perticular prefix, and then
the other end can do so if they want to. This needs to be automatic and
quick, ie a change in RADB policy should be reflected in the real world
in minutes (preferrably) or hours (more realistically perhaps) and not in
days or months.

  There was an idea years ago about utilizing a domain (rdi.int) for
this.

  eg:

  dig any 267.rdi.int.

This might already be in place, I don't know other than RIPE, but in RIPE
you can register a route object with a certain IP space, and IP space can
have multiple route objects. The thing here is that to implement this

  Herein is the value, the RIR (RIPE) is also the holder of the policy.
With ARIN, this is not the case, there is RADB and a number of other RR's
that are out there for varying reasons, some personal and some business.

policy in IOS creates *huge* rulesets that are really cumbersome and
cluttery. Also, people tend not to rebuild these very often, so for
customer routes, doing a handover between ISPs when moving PI space might
involve outages and days of limited connectivity. Also, change of policy
doesn't isn't reflected unless a route is cleared (soft though) which
involves re-hashing a lot of routes very often if filters are updated
often.

  I think in this web 2.0 world, everything you're speaking of
can be a challenge but not be impossible. The problem I see is there are
no good tools. Take http://prefix.pch.net/ for example.

  This can help you audit the routes that are going to be placed
in a prefix-list. How do you integrate something like this into your
business policy? Have customers submit a web form for their routes? It's
easy when your customer is AS267, but what if your customer is something
larger like telstra?

Perhaps it's too late for IPv4 in this aspect, but it might be feasable
for IPv6? Fewer prefixes and (hopefully) no break-outs, would mean PA

  I think it's a matter of having a semi-uniform industry policy
that is generally agreeable. I don't want to see the ANS-days case where
you could not route without RADB and some fancy scripts probing their bay
networks devices with snmp sets.

  But I do agree that we need a better toolset built. Now
the question is, who can/will do this? How can it be shared?

  If I can make this backend uglyness called "RADB/irrd" invisible
to my customers, will that help?

blocks could be filtered hard, and if larger ISPs do hard filtering based
on RADB, ISPs getting into IPv6 would need to get their prefixes properly
registered there before getting IPv6 working to any extent.

Anyhow, as people are continuing to use null routes to enforce regulatory
demands (likely cause for the latest outages) this problem will most
likely escalate.

This would also help with <http://eng.5ninesdata.com/~tkapela/iphd-2.ppt>
problem I guess.

  Yeah, the challenge here is those that are unwilling to take
action threaten the entire industry as it only takes a few bad actors
to disrupt the network currently, and if you do it correctly, who is
going to trust the infrastructure that we operate?

  - Jared

Saying something is Operational (and on-topic for nanog) does not mean
you should de-peer them.

  If it's active and persistent, it would qualify as operational.
Then I can classify the risk. I'm openly wondering if there should be
more aggressive "turn the bad stuff off" happening.

That said, I will not stop you from de-peering a network who can't keep
its table clean. Your network, your decision.

  I'm still seeing persistent leaks, generally over 10k/day that
are unresolved after a year of collecting this data.

You wield a much bigger hammer than 99.999% of the people here, and you
know it.

  I'm not posting as my employer, nor purporting to represent them,
but at the same time, wonder what the impact would be if there were more
consistent actions taken by networks when there was badness,
either routing leak or otherwise.

While I doubt "shame" will work in all but the most extreme cases, I
believe brokeness does, eventually have an impact. Let's just hope that
impact is not blunted by (for instance) monopoly power, so that "voting
with your wallet" will force network to fix things.

  I certainly agree on the impact. If there were clear
and predictable reactions to the brokeness, would people actually
take actions to repair the problem?

  eg:

200.1.15.0/24 2914 6762 27648 3561 5511 6505 27782

  What If I were to respond with a bgp notify (invalid as-path)
to 6762 when they send this route to 2914? Doesn't matter if they're
a customer or a peer, i may not want to get 3561 routes from you.

Just thought I'd say "BCP38" again.

Router#conf t
Router(config)#interface customer0/1
Router(config)# ip verify unicast source reachable-via rx

  - Jared

ok, i can not hold my tongue. sorry.

might there be a formally rigorous approach to this problem? we keep
having it. perhaps there is something solid and real we could do, as
opposed to temp hack after temp hack.

randy

-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: http://getfiregpg.org

owFdVAtsFFUUbRdaZRIoQfkUxLzwaRHWbgtIC0RA8YNfsEBAIJTZmbe7r52ZN857
s8tWgSIoUiqoAUUQlE8aAxYrgUpbqKCQEotSykeCKRLoR2qBQspXqN43uy3iJvub
ue/cc849dz7q2inGFbtZH3r58It/SbFFnZd5E9OGp2WkD89IHT1iZPoon9/0Z3Hd
fIFoeNLWKR+bsqVSA+lhRPwGtWRDwW4kM0SQLoeRQTkKyEGMeAAjPN/EFsFQgbJ0
yngWoj4UpjZSqUQMpwRNnYIUquu2QXg4JUWSvDZ3o1AgjELU1lQjmSNGdcwDxPAj
jeRg5KOWLmtaGFn4LZtYcF1SbMahyGIeE4tPDqwY4cyDuYI4jRCyZNWLqDcbK5wJ
vjJKjiBgHRs8WQJcJI4DoKcdEHn9JmLYChIFM0kiEfayheG0gUOoo042VOcWA0+Q
bQqS4qwbEY4IQ4qGZUsLS4zLHKsoql2hBjBVQK/zL4rlMS0aJCqACnqYcaEc9LN2
IpIOlWA2MTi8OyQJfcLAKDgxoJmmIUAD7gyY+B5oAzPA7P64/uuOW4rIZoxQA4Cx
6IVYwOYqDYE8gxPNwepgGgTbfATwHJ5ghuTHBrZALEM+onEYCgoRHkCWyeAopZok
MVnHyC9ItDvPUlIQaQ+ImHx0Ph1GOXMTlEGYZGAeolYOg1o3yhaWtJcTy0KeBwRF
eFEDOxn1wheFOgijLJmgzY0gdVAjQwcneoKYCA5TLGJyRxGSbfBN5mAJ0JQkAAIc
kc0ImOASySwkA2YMp+G+KnrCT2ww23IMg41A4CKoDRBTEsAq1rDfARaMDFv3OrNn
1LZgciIsQkuEIxGIgioxchzCUqSlQwWpxOcjiq2JDXIuEz7eSa0mW34A7bBMVmWT
R0j+b7uEUZCVaAcTU1PD0R6QJ4uLIz6qaTTkLD2kILqpkRgZFCkBCiEFRopsM1hw
UXKfpQIiqM+HfBbVkYItJ8IWtbnYLyVgEci7bEjwWu56rHNMrCsmPs4lHjsxUpfu
7U+p2pXdY75SFkw+sun0cy1fr8lM/nxYy8XqtsqB42d7PIfVYxcaugz+YntLQlLL
kJwjeyuaTTm9sltXd9PBM32nHm1a96d747WaksYt+/6+caX+6oTzK0mBJOUdLFnU
q/fD5rvo6MWGTzJLty8Z8PLc87HNavbapJrcn/tfdI2cdS63Z1v5zSmZcROWZyT0
Oj33UlvtvYlLdi3Ox+xkuPX4zpl3d1c9m/HmjTv6zazc1Y2XTzTUlI1uXVx6+OSq
4s+qV2zdv35m0TjfvarypfUHnnu6esGovWOK59zZU53Xv++oc0m329K7zPhDrs3O
rnlm4uC6hvz0D07FVaQeK/zuajCnZMSXiSvrfqnaOzmp6PrQ+cWDzu6c/cqR3AsD
4u9OOr79UM9riwt6lO7Zb259Ku/x0etKEwb3PacU9tbDtn0w0A03DfNuejVYeaso
Z/rYK7OW5GZe/SF/w+59q6f1KziwZmBqn22tg25nTGupO3PiypyU4K45aYGCIWWP
bk4oWVRc2Hxt16eb0/plNZsnvlk7fE1i2q1izyOX3rlZGfj2cn3F1NCCspcS/Xzc
6t/jY8sXxldU9yx86P15bzTX//jkoaE/uV6L37cjY9l7qQs7xbW2VpV5l45Z9XZt
UfeJeuPrfda6zuaFq3L92+zG8tRt10/N80+/x71jd1yy19ecbdqy7vtFckz6zBnm
6X+S5i37sFeP5/NPbazb0Gnhiid8v/72Lw==
=TITY
-----END PGP MESSAGE-----

apologies for the encrypted email, pgp acting up..

bottom line: the irr is a hack, not a formal solution.

rand

I don't think the IRR is so much a hack (it's a tool), but we're lacking the process and infrastructure to vet/validate that a given ASN is *authorized* to originate a prefix, and all of the policy bits (which the IRR has if you use it) associated with which ASNs should propagate the prefix, etc...

We're lacking the authority and delegation model that DNS has, I think?

-b

think?

Some of the RIR's have been working diligently on this stuff and you can
see proposals, at least in the ARIN region, where there is a concerted
effort at credentialing allocations. If you read between the lines and
know the people and their funding sources, you can assume that there is
a high level of attention to this, contrary to what we read in the press
about DHS et. Al.

Some of this activity is likely to tie into RIR transfer process i.e.
the titling of ip address allocations. At least I think so.

Disclaimer: This is a personal opinion only and I invite mailbox flames
correcting.

Best,

-M<