Verio Decides what parts of the internet to drop

IMHOTS

Apparently for their convenience Verio has decided what parts of the Internet I
can get to. With no notification. This was (eventually) posted to the BSDI
mailing list when some of us were cut from access to the site we need to
maintain our OS.

For this I pay them.

Doug Denault

Apparently for their convenience Verio has decided what parts of the
Internet I can get to.

verio does not accept from peers announcements of prefixes in classic b
space longer than the allocations of the regional registries.

we believe our customers and the internet as a whole will be less
inconvenienced by our not listening to sub-allocation prefixes than to have
major portions of the network down as has happened in the past. some here
may remember the 129/8 disaster which took significant portions of the net
down for up to two days.

the routing databases are not great, and many routers can not handle ACLs
big enough to allow a large to irr filter large peers. and some large peers
do not register routes.

so we and others filter at allocation boundaries and have for a long time.
we assure you we do not do it without serious consideration or to torture
nanog readers.

With no notification.

verio's policy has been constant and public.

randy

doug@safeport.com wrote:

IMHOTS

Apparently for their convenience Verio has decided what parts of the Internet I
can get to. With no notification. This was (eventually) posted to the BSDI
mailing list when some of us were cut from access to the site we need to
maintain our OS.

Actually, this is a fairly common routing policy. I think you will find
other networks do the same thing. It is an entirely sane thing to do, and
it helps force people to aggregate their routing announcements properly.
Also, it protects you against fat-fingers that people sometimes do (one time
UUnet announced 50k extra /24s in classful B space).

Alec

Normally I agree with Randy (cough) but:

> Apparently for their convenience Verio has decided what parts of the
> Internet I can get to.

verio does not accept from peers announcements of prefixes in classic b
space longer than the allocations of the regional registries.

Simply put, thats dumb. I can't imagine a technical reason for this (CPU
and/or memory), so it must be politcal.

we believe our customers and the internet as a whole will be less
inconvenienced by our not listening to sub-allocation prefixes than to have
major portions of the network down as has happened in the past. some here
may remember the 129/8 disaster which took significant portions of the net
down for up to two days.

I believe that if I have a customer who is multihomed between me and
another provider, his punch-throughs to the non-address-space-providing
provider should be heard. It's called 'global routability.'

the routing databases are not great, and many routers can not handle ACLs
big enough to allow a large to irr filter large peers. and some large peers
do not register routes.

There are ways to get around this (as-path filtering, maximum-paths, etc)
that aren't as nazi as one would hope, but will prevent stupidity and
provide sanity checking.

so we and others filter at allocation boundaries and have for a long time.
we assure you we do not do it without serious consideration or to torture
nanog readers.

Heh.

> With no notification.

verio's policy has been constant and public.

But unfortunate. Will they announce a customer-announced /24?

> > Apparently for their convenience Verio has decided what parts of the
> > Internet I can get to.
>
> verio does not accept from peers announcements of prefixes in classic b
> space longer than the allocations of the regional registries.

Simply put, thats dumb. I can't imagine a technical reason for this (CPU
and/or memory), so it must be politcal.

  Your pager didn't go off when the routing table had 100k prefixes
in it, I take it.

  This is a Good Thing(tm).

> we believe our customers and the internet as a whole will be less
> inconvenienced by our not listening to sub-allocation prefixes than to have
> major portions of the network down as has happened in the past. some here
> may remember the 129/8 disaster which took significant portions of the net
> down for up to two days.

I believe that if I have a customer who is multihomed between me and
another provider, his punch-throughs to the non-address-space-providing
provider should be heard. It's called 'global routability.'

  The people who "purchased" this space, didn't realize that such
routing policies exist, and it is not the problem of someone trying to reach
them, it's the problem of the person who is using address space that
was not originally assigned to them.

> the routing databases are not great, and many routers can not handle ACLs
> big enough to allow a large to irr filter large peers. and some large peers
> do not register routes.

There are ways to get around this (as-path filtering, maximum-paths, etc)
that aren't as nazi as one would hope, but will prevent stupidity and
provide sanity checking.

  Maximum paths deals primarily with ibgp

  as-path filtering? How will this help?

  Oh yeah, I'll as-path filter my peers, and then have even
more reacability issues.

> so we and others filter at allocation boundaries and have for a long time.
> we assure you we do not do it without serious consideration or to torture
> nanog readers.

Heh.

> > With no notification.
>
> verio's policy has been constant and public.

But unfortunate. Will they announce a customer-announced /24?

  Yes.

  They can't guarentee that peers will listen to it though.

  Your pager didn't go off when the routing table had 100k prefixes
in it, I take it.

  This is a Good Thing(tm).

Au contriar, monfrair (sp?). I was among the first to call Vinnie.

> I believe that if I have a customer who is multihomed between me and
> another provider, his punch-throughs to the non-address-space-providing
> provider should be heard. It's called 'global routability.'

  The people who "purchased" this space, didn't realize that such
routing policies exist, and it is not the problem of someone trying to reach
them, it's the problem of the person who is using address space that
was not originally assigned to them.

You misinterpreted.

Multihomed customer gets a /24 of my announced /16. He's announcing that
/24 to his other provider; since it is more specific the other provider
will always win (BGP 101). So, for it to work, I need to allow a punch
through of a /24 to my peers. And for it to _really_ work, people would
have to listen to the /24 from both us and the other provider to our
multihomed customer.

> There are ways to get around this (as-path filtering, maximum-paths, etc)
> that aren't as nazi as one would hope, but will prevent stupidity and
> provide sanity checking.

  Maximum paths deals primarily with ibgp

Well, thats patently wrong. I don't know how else to respond to this.

  as-path filtering? How will this help?

It will prevent redistribution of a person who announces * to you. It
won't fix everything (including the 7007 debacle, but thats a whole
another story), but it will fix most fsck-ups.

  Oh yeah, I'll as-path filter my peers, and then have even
more reacability issues.

Tell Sprint, Agis, and others. Unless they changed since my last dealing
with them.

> But unfortunate. Will they announce a customer-announced /24?

  Yes.

  They can't guarentee that peers will listen to it though.

Well, it's a start.

jared:

Your pager didn't go off when the routing table had 100k prefixes in
it, I take it.

i read about the incident on this mailing list. and, more importantly,
our customers did not feel it except as inability to get to the networks
which implemented routing policy via pager.

alex rubenstein:

I believe that if I have a customer who is multihomed between me and
another provider, his punch-throughs to the non-address-space-providing
provider should be heard. It's called 'global routability.'

it is not ours to say what should be heard, i.e. what our peers accept.
it is ours to say what we announce. and indeed for verio customers we
are willing to announce long prefixes from other providers' spaces.
heck, for multi-homed customers, we are willing to announce longer
prefixes which punch holes in our own larger space, allowing them to
more easily play load balancing games.

randy:

the routing databases are not great, and many routers can not handle
ACLs big enough to allow a large isp to irr-filter large peers. and
some large peers do not register routes.

alex rubenstein:

There are ways to get around this (as-path filtering, maximum-paths,
etc) that aren't as nazi as one would hope, but will prevent
stupidity and provide sanity checking.

many things which might work for a small 42nd tier isp do not scale to a
tier one provider. and i share jared's scepticism that these hacks work
for even the teenies, i.e. the 129/8 disaster was from a direct peer.

randy:

verio's policy has been constant and public.

alex rubenstein:

But unfortunate. Will they announce a customer-announced /24?

jared:

Yes. They can't guarentee that peers will listen to it though.

bingo. and not to announce a customer route would preempt our peers'
ingress route filtering policy choices.

randy

For the record BSDI has renumbered the critical parts of their network.

For the record BSDI has renumbered the critical parts of their network.

great. if there is anything reasonable we can do to help them, they should
please holler.

randy

Alex Rubenstein
Sent: Thursday, December 02, 1999 3:03 PM

Normally I agree with Randy (cough) but:

> > Apparently for their convenience Verio has decided what
parts of the
> > Internet I can get to.
>
> verio does not accept from peers announcements of prefixes
in classic b
> space longer than the allocations of the regional registries.

Simply put, thats dumb. I can't imagine a technical reason
for this (CPU
and/or memory), so it must be politcal.

Nah, it's tradition.

> we believe our customers and the internet as a whole will be less
> inconvenienced by our not listening to sub-allocation
prefixes than to have
> major portions of the network down as has happened in the
past. some here
> may remember the 129/8 disaster which took significant
portions of the net
> down for up to two days.

I believe that if I have a customer who is multihomed between me and
another provider, his punch-throughs to the
non-address-space-providing
provider should be heard. It's called 'global routability.'

I actually have this situation right now. MHSC Colorado offices are in
Colorado Springs, USWorst territory. Our CA offices are in the East SF Bay
Area (down the street from LLNL). Our own IP block is a /24, from our
upstream provider. Technically, we are multi-homed and have an ASN. However,
no one listens to it. This is not slam against Verio, since Sprint doesn't
listen to it either. They are only two of many that have such policies (as
we found out later). What we wound up doing is establish a SSH VPN between
our offices and the CO office uses our CA assigned IP numbers as NAT'd IP
behind a USWorst IP, using a USWorst connection. Outbound packets, to the
general Internet, go directly via the USWorst IP, but return packets come in
over the VPN from CA. Yeah, it sux.

It's a PITA and not the cleanest of methods, but until all the backbones
quit filtering /24s it's what we have to do. The other alternative (and
we've considered it) is to obtain a much larger space directly from ARIN and
burn the unused space. Then we could remove the last bit of static routing
and use BGP4 as we should.

It's a PITA and not the cleanest of methods, but until all the backbones
quit filtering /24s it's what we have to do. The other alternative (and
we've considered it) is to obtain a much larger space directly from ARIN and
burn the unused space. Then we could remove the last bit of static routing
and use BGP4 as we should.

Wouldn't it be nice if backbones got around to simply charging for annoucements
and quit this arbitrary filtering?

Tony

Yeay, verily.

We would be happy to pay fees within reason. The SSH VPN takes a lot of
maintenance and handing it over to a SysAdmin has been troublesome since it
is a very non-standard means of doing this. Open-source SSH VPNs are a great
hack, but [lack of] maintainability is the downfall of ALL hacks.

Wouldn't it be nice if backbones got around to simply charging for
annoucements and quit this arbitrary filtering?

thanks geoff. :slight_smile:

and how would charging for announcements have ameliorated the 129/8
disaster? ahhh, when they tried to announce those 50k /24s, the check
would have bounced!

randy

> Wouldn't it be nice if backbones got around to simply charging for

Without trying to talk about this specific class B incident, what would
the "official" line be if a customer with their own class B designed to
announce half with one transit provider on the West Coast, and the other
half with another transit provider on the East Coast?

I don't think I have heard of customers being required to maintain
contiguous networks.

Deepak Jain
AiNET

That depends. Many operators of /24s would be happy to pay, within reason.
This would provide plenty of cash to upgrade routers. Right now I am looking
at ~$1000/Gbps from various colo providers, for a site that is expected to
go over 1Tbps (Yes, that's a Tera-bit per second), in 18 months. The site,
with Dev/QA/Stage/Production, could easily burn a /24, but no more than
that. (One of our requirements is a provider with LOTS of dark-fiber and
cold-potato routing, as a result.) We are looking into distributing the load
geographically, which also covers Big-D disasters. Now we have a
multi-homeing problem unless we use the same provider in both locations.
Business-wise, this is not acceptable, to be locked-in, in this way.

Considering the amount of money involved, do you still doubt that my client
would be willing to pay reasonable fees, to announce their /24? Don't you
think that the presence of this cash would cover the check? We've already
established that the only technical issue is the capital expense ($cash$)
required to upgrade backbone routers.

Why not use CIPE?

As for BSDI, is it possible for them to trade their /17 in to ARIN for a
more likely routable one?

On filtering, if Verio's filter policy is so long standing, is there a
reason Sprint is the only NSP with a filter policy listed at
http://www.nanog.org/filter.html ?

If filter policies like Sprint's and Verio's really are common among the
big backbone providers, that would seem to make it relatively pointless to
attempt to multi-home with provider supplied IP space. We have a customer
interested in multihoming for reliability, but they use so little IP
space, even assigning them a /24 would be a stretch. Since the space we
let them use is in 209.x.x.x, it seems their route wouldn't get far. So
how do they make their internet connection more reliable? Use IP space
from each of their providers and play games with DNS and NAT?

> Wouldn't it be nice if backbones got around to simply charging for
> annoucements and quit this arbitrary filtering?

thanks geoff. :slight_smile:

and how would charging for announcements have ameliorated the 129/8
disaster? ahhh, when they tried to announce those 50k /24s, the check
would have bounced!

Thanks Sean! :wink:

Let us please be very clear here. There are multiple problems.

First, there is the problem of overall size of the routing table (and
proportional costs in BGP processing and convergence time, etc.).

Second, there is a reasonable requirement for sanity checking and
authentication in announcements.

Third, there is an alleged correlation between prefix length and route
flap.

We currently have one hammer (filtering) that has been used to drive the
above machine screw, ice piton, and cup hook, respectively.

I was not and am not suggesting that anyone stop all filtering. I am
suggesting that a sane prefix settlement scheme would allow us to dispense
with the filtering policies that are currently in place and would allow the
backbone to globally distribute any prefix, regardless of prefix length, if
only the originator has paid enough money and informed people first. You
want to inject a /32? Go right ahead. Send your check to your provider
and it can be made to happen.

This is the only sane mechanism to divide up a limited resource (global
prefix slots) amongst an otherwise unlimited demand (domains with
prefixes). Now I don't claim to be a policy wonk and I don't have the
slightest idea of how to make this equitable and fair for all, but I do
know that this would result in an outcome that would give us routing tables
far smaller than our current 67k entries.

I'll also note that this would also decrease the pressure on the address
space. No need to go get a /19 if I can get my /23 globally advertised.

The sanity checking and policy checking requirements would indeed require
filtering, but perhaps this is simply an extension of filtering any
advertisements that you haven't received payment for. This will prevent
you from accidentally receiving 50k of anything, not just /24's.

The correlation with route flap should be re-examined. I suspect that this
is no longer a driving force and is more than adequately compensated for by
having flap damping parameters that scale geometrically with the prefix
length.

Regards,
Tony

Thanks Sean! :wink:

and what the heck are you doing at this hour, yakov. get a life! :slight_smile:

Second, there is a reasonable requirement for sanity checking and
authentication in announcements.

s/reasonable requirement/desperate need/

given this much else is possible.

rany

If filter policies like Sprint's and Verio's really are common among the
big backbone providers, that would seem to make it relatively pointless to
attempt to multi-home with provider supplied IP space. We have a customer
interested in multihoming for reliability, but they use so little IP
space, even assigning them a /24 would be a stretch. Since the space we
let them use is in 209.x.x.x, it seems their route wouldn't get far. So
how do they make their internet connection more reliable? Use IP space
from each of their providers and play games with DNS and NAT?

  Yes, or get their provider to give them a chunk of swamp space (which
a few have tucked away), multihome with the same provider, or find a way
to justify a routable allocation.
  None of these are, IMO, very good solutions.

  Austin