Broken Internet?

[Roeland_M.J_Meyer]

From: Patrick Greenwell [mailto:patrick@cybernothing.org]
Sent: Tuesday, March 13, 2001 11:29 AM

to change the fact that these alternative root server
networks exist and
that the Internet still works, mostly(as I'm sure you'd agree
it's always a little broken.)

That is an understatement (a little broken). I have just been introduced to
one of those broken areas, the hard way.

Given:
1. Prefix filtering at /20.
2. Most small busineses limited to /24, by policy/procedure.
3. Multi-homing requirements for multi-office businesses (many SOHO's).
4. Impending business failure of many DSL ISPs.
5. Total lack of responsibile behavior among DSL access providers.

It is next to impossible for a small business to have reliable internet
connectivity without moving into a large co-lo. Even if they can afford the
multiple T1's, they can't get portable IP addresses that will be advertised
reliably. Many of them need, at most, a pair of /24's and ARIN, knowing
this, will not issue them portable blocks larger than /24 without severe
justification requirements.

Many of you might think that is okay, but what if their upstream dies off
(as recently happened to MHSC). In the current day and age, business stops
until they get reconnected. This disconnect is at minimum, 4-6 weeks, under
the best of circumstances. As one vendor recently pointed out in their
adverts, most businesses, down for more than 14 days, will never survive.
More importantly, such an outage flat-lines the revenue picture for that
entire fiscal quarter, for the unlucky victim.

What we have today is a manufactured dependence on a single upstream
provider and no way to multi-home. Even co-lo boils down to single-home
dependency.

Yes, there are a bunch of hacks to work around this problem. But, that is
exactly what they are ... hacks. They are not something I could build a
sustainable business around.

Any business needs:
1. to be able to change upstream providers without having to renumber.
2. to be able to change access providers without having to suffer
multi-month down-times.
3. to be able to have its net-block(s) visible regardless of which ISPs they
are currently using.

Currently the only ones that can do that are those that;
1. Are large enough to justify a /20 (begging the question of how they got
that large).
2. Can afford their own datacenter.

It looks like our technical solutions are raising unreasonable barriers to
entry for small businesses.

[Peter_Francis]

<snip>

Any business needs:
1. to be able to change upstream providers without having to renumber.

Why? Intelligent use of DNS and dhcp make renumbering only a minor inconvenience.

2. to be able to change access providers without having to suffer
multi-month down-times.

Mission/business critical services should be in a co-lo anyway and not off a DSL line.

3. to be able to have its net-block(s) visible regardless of which ISPs they
are currently using.

How do you propose doing this without growing the routing table 1-2 orders of magnitude?

Currently the only ones that can do that are those that;
1. Are large enough to justify a /20 (begging the question of how they got
that large).
2. Can afford their own datacenter.

It looks like our technical solutions are raising unreasonable barriers to
entry for small businesses.

No. Co-lo your website and "intranet". Get two T1's that same provider via two different entry points/carriers to your office (if possible) and you should be about as rock solid you could expect for $2-3000/month or there abouts.

Peter

[_Greg_A_Woods]

[ On Tuesday, March 13, 2001 at 12:52:41 (-0800), Roeland Meyer wrote: ]

Subject: Broken Internet?

Even co-lo boils down to single-home dependency.

It doesn't have to.

Yes, there are a bunch of hacks to work around this problem. But, that is
exactly what they are ... hacks. They are not something I could build a
sustainable business around.

For _small_ businesses it is extremely trivial to multi-home (i.e. to
truly multi-home all their network-visible servers). Well there's one
small trick that requires each host have decent support for something
like IP Filter that's capapble of re-directing packets based on source
address. (I'll post a technical description the trick I use with IP
Filter if enough people don't think it's obvious how it works. There
have also been hacks by others to the BSD networking stack to allow
multiple default routes and to do source-routing kinds of tricks.)

With a small amount of planning and skill it's possible to make this
kind of real multi-homing fully functional through the DNS (and even to
enjoy some load-balancing as a result).

For most any _small_ business this works very well (been there, done
that, would even do it with my machines here at home if Rogers@Home
didn't charge as much as they do for IP addresses). Conveniently about
the time your network gets big enough that this scheme gets too hard to
manage, you're up to the size where network multi-homing via BGP,
etc. is possible.

Any business needs:
1. to be able to change upstream providers without having to renumber.

Why? If you're _small_ then renumbering is relatively easy! It's the
big guys (who didn't use DHCP from the start) who have a hard time
renumbering.

2. to be able to change access providers without having to suffer
multi-month down-times.

If you're multi-homed then all your providers have to go down before
you'll suffer any down-time that's not your own doing.

The real issue is with lead times on ordering local loops, etc. If
you've already got them in place because you are already connected to
multiple providers and are doing host-based multi-homing then you don't
have to worry.

3. to be able to have its net-block(s) visible regardless of which ISPs they
are currently using.

By properly multi-homing all your servers (and not networks via routing)
there's no issue about net-block visibility, BGP peering, or the like.
You simply use as many/few IP addresses from each provider as you need
to multi-home all your servers, and they aggregate them into their own
routes as necessary.

Same thing goes for co-locating multiple identical servers in multiple
locations.

Currently the only ones that can do that are those that;
1. Are large enough to justify a /20 (begging the question of how they got
that large).
2. Can afford their own datacenter.

Yes, exactly. They're the only ones who really need network
multi-homing (which is such a poor phrase to describe what it is).

Everyone else can afford to multi-home their servers one way or
another.

It looks like our technical solutions are raising unreasonable barriers to
entry for small businesses.

I think not.

I fully agree that Internet-based businesses critically require multiple
network access points. However since this can be done trivially with
either multiple co-located servers, or properly multi-homed servers,
there's no reason to consider /20 netblocks, etc., to be barriers of any
sort.

[asrnanog]

1. Prefix filtering at /20.

This varies.

I've enjoyed pretty good -- though not total, for obvious reasons (hi
Randy! -- success announcing /24's and /23's out of provider-issued
IP space. Just make sure your upstreams are:

     - sufficiently redundant themselves.
     
     - announcing a shorter-length prefix that your space falls under,
       so that you can achieve some level of reachability, should
       people filter what you're announcing.

     - not insolvent.

2. Most small busineses limited to /24, by policy/procedure.

Small businesses, like any business, should be allocated as much space
as they can justify as per current registry guidelines. This might be
a lot more or less than a /24, depending upon many factors.

3. Multi-homing requirements for [...] many SOHO's

Do these need to involve BGP?

There are various solutions that don't. While less than optimal,
they're fully acceptable for the ghetto office/home office crowd.
Obtaining pipes to multiple providers, sticking mail/DNS servers on
each 'net, and NAT'ing out whatever pipe is operational, falls under
this category. And outsourced backup MX and DNS needn't cost you a
dime.

5. Total lack of responsibile behavior among DSL access providers.

Every industry has its bad apples. The DSL biz is certainly no
exception, as has been proven many times. Characterizing all DSL
access providers as harsh and irresponsible isn't really fair.

It is next to impossible for a small business to have reliable internet
connectivity without moving into a large co-lo.

You sure?

Even if they can afford the multiple T1's, they can't get portable
IP addresses that will be advertised reliably.

So, wait for ARIN to offer micro allocations. Or find a /24 or two
out of swamp space to recycle. Or find some other way around this.

most businesses, down for more than 14 days, will never survive.

Right. And chances are businesses with carefully planned
infrastructure will never be down this long, unless some major
catastrophe occurs, in which case 'net connectivity will likely be the
least of their concerns.

More importantly, such an outage flat-lines the revenue picture for that
entire fiscal quarter, for the unlucky victim.

Survival of the fittest, I guess.

Any business needs:
1. to be able to change upstream providers without having to renumber.

Implications of routing table growth aside, I'm not sure I understand
why you consider this to be essential.

2. to be able to change access providers without having to suffer
multi-month down-times.

Yes, conventional means of last-mile telco loop delivery can be slow
at times. So can other steps of the provisioning process. There are
some viable alternatives today, and more on the horizon. Plan
accordingly.

It looks like our technical solutions are raising unreasonable
barriers to entry for small businesses.

Unpleasant? Perhaps. Unreasonable? No.

-adam

[I_Am_Not_A_Lawyer_Si]

Roeland Meyer wrote:

What we have today is a manufactured dependence on a single upstream
provider and no way to multi-home. Even co-lo boils down to single-home
dependency.
...
It looks like our technical solutions are raising unreasonable barriers to
entry for small businesses.

I beg to differ.

I presented a _technical_ solution back in '92-93 at IETF -- numbering
allocations based on local exchanges. Deering presented another --
numbering based on metropolitan areas. Either eliminated dependence
on a single upstream, and made it simple to switch.

Instead, we have the non-technical solution -- provider-based
allocations. Why? Contrary to Greenwell's assertion, capital does NOT
seek a "market based" solution. Successful markets assume competition
and low barriers to entry. Capital seeks best return on investment.
Best return requires monopoly advantage.

Either of our proposals would have improved competition, but
competition was not what the large providers wanted. The large
providers funded ARIN.

The _technical_ solutions required regional cooperation between local
providers to carry all local (non-transit) traffic directed to the
exchange(s), much as the NSFnet (back when this was the regional-techs
list). Such cooperation has been in short supply.

WSimpson@UMich.edu
    Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32

[Daniel_Senie2]

Peter Francis wrote:

<snip>
>
>Any business needs:
>1. to be able to change upstream providers without having to renumber.

Why? Intelligent use of DNS and dhcp make renumbering only a minor inconvenience.

>2. to be able to change access providers without having to suffer
>multi-month down-times.

Mission/business critical services should be in a co-lo anyway and not off a DSL line.

I don't advise use of DSL regardless, but why is a colo better than a
hardened facility owned by a company, with off-grid power, and multiple
DS-3 lines? Just because that company only needs 200 public IP
addresses, why should they be unable to multi-home?

It's entirely possible to build a mission critical data center better
than the average colo, and certainly more secure than many colos.

There's a TECHNICAL issue here in HOW to implement multihoming
successfully. We have a policy issue at ARIN, APNIC and RIPE which is
keeping the issue from becoming one which people pay enough attention
to. If it were in our faces more, perhaps better solutions would be
proposed and implemented.

>3. to be able to have its net-block(s) visible regardless of which ISPs they
>are currently using.

How do you propose doing this without growing the routing table 1-2 orders of magnitude?

We can't. The point, though, is that the Internet needs to have a GOOD
way to support multihoming. We presently DO NOT have a good mechanism
for this. The IPv6 approach to this does not appear workable either.

This is a problem for the IETF, not NANOG, though, to solve. Getting
people to understand there IS a problem needing a solution appears to be
more than half the battle.

>
>Currently the only ones that can do that are those that;
>1. Are large enough to justify a /20 (begging the question of how they got
>that large).
>2. Can afford their own datacenter.
>
>It looks like our technical solutions are raising unreasonable barriers to
>entry for small businesses.

No. Co-lo your website and "intranet". Get two T1's that same provider via two different entry points/carriers to your office (if possible) and you should be about as rock solid you could expect for $2-3000/month or there abouts.

Great. So when this one upstream provider screws up, you're still dead.
When there's a routing table problem and that upstream's advertisement
for your block isn't seen by 1/2 the world, you're dead.

We HAVE built an environment where businesses are forced into such
situations UNLESS they are lucky enough to have grabbed IP address space
early in the life of the 'net, or are big companies. Colo isn't always
the answer.

[Peter_Francis]

<snip>

I don't advise use of DSL regardless, but why is a colo better than a
hardened facility owned by a company, with off-grid power, and multiple
DS-3 lines?

This discussion started when someone questioned whether the "difficulty" of multi-homing was a barrier to entry for SMALL businesses. I can think of no definition of SMALL that includes the ability to build a "hardened facility" with "off-grid power" and "multiple DS-3 lines.

Come on now. If you have that kind of capital then you might as well just go out and buy a small hosting company. This gets you enough usage to meet the minimum requirement for a portable CIDR block plus income from the hosting.

I'm tired of people waving the "I must be multi-homed" flag around without actually looking at where the highest risk points of failure are and focusing their resources there first.

For a SMALL business with < $50,000/year to spend on infrastructure you can get yourself well up into the 99th percentile of uptime withthe colo/T1 model. Then you can go spend the rest up your time and money building a business that actually works. Any SMALL business that doesn't have a solid enough relationship with its customers to survive the < 1% chanced outage has a bogus business model in the first place.

If you really want to be careful about things get two T1's, one back into your colo-site and one to another provider. Keep your DNS ttl's low, say 10 minutes, and run a secondary nameserver and backup server for your site off the non-colo-provider's T1 address space. Use dhcp for your office LAN and run a resolver with 2 nic cards, one talking to each T1. You get the picture. You are now way out beyond the 99th percentile at the cost of keeping one decent sys admin on staff.

Peter

[_Greg_A_Woods]

[ On Tuesday, March 13, 2001 at 19:54:00 (-0500), Daniel Senie wrote: ]

Subject: Re: Broken Internet?

We can't. The point, though, is that the Internet needs to have a GOOD
way to support multihoming. We presently DO NOT have a good mechanism
for this. The IPv6 approach to this does not appear workable either.

That's because this is a problem that has never existed, not ever.

Proper *real* multi-homing has *ALWAYS* worked and it's technically an
excellent way to achieve redundant connectivity for a "small" network.
(other risks related to "all your eggs in one basket" type of physical
infrastructure aside, and they can be put aside for many businesses
because if the bricks&mortar part is destoryed the business can't
survive anyway....)

Given the various simple little tricks I mentioned you don't even need
to put multiple interfaces in every server.

[Stephen_Sprunk2]

Thus spake "Peter Francis" <peter@softaware.com>

>Any business needs:
>1. to be able to change upstream providers without having to
>renumber.

Why? Intelligent use of DNS and dhcp make renumbering only a
minor inconvenience.

Renumbering PCs is a trivial task. Reconfiguring hundreds (or
thousands) of routers, firewalls, etc. to account for the moved PCs is
not trivial. Renumbering servers is not trivial.

>2. to be able to change access providers without having to
>suffer multi-month down-times.

Mission/business critical services should be in a co-lo anyway
and not off a DSL line.

Keep in mind that Fortune 100 companies with multiple DS3s in several US
locations are in the same boat wrt renumbering. Most don't qualify for
portable addresses by ARIN's rules.

Also, try convincing someone like AmEx or Citibank that they should put
their servers under someone else's physical control -- that'll be good
for a laugh. Sure, that's extreme, but where exactly do you draw the
line on who's "important" enough to host their own servers?

>3. to be able to have its net-block(s) visible regardless of which
>ISPs they are currently using.

How do you propose doing this without growing the routing table
1-2 orders of magnitude?

If they're only announcing one or two routes (reasonable if RIR policy
were more sane), it would *decrease* the routing tables by an order of
magnitude.

>Currently the only ones that can do that are those that;
>1. Are large enough to justify a /20 (begging the question of how
>they got that large).
>2. Can afford their own datacenter.
>
>It looks like our technical solutions are raising unreasonable
>barriers to entry for small businesses.

No. Co-lo your website and "intranet". Get two T1's that same
provider via two different entry points/carriers to your office (if
possible) and you should be about as rock solid you could expect
for $2-3000/month or there abouts.

Trust all of your server availability and corporate connectivity to a
single ISP? The only point of failure you've (hopefully) eliminated is
the local loop. And, if you depend on back-end servers to feed your
coloed web servers (likely), that local loop is still essential. And
now you're paying for rack space and it's a pain to do maintenance.
Wonderful.

Peter

S

[Jim_Mercer2]

IPO a new network carrier and do leveraged buyouts of a couple backbones,
then your small business can have the same facilities as any mega-corp.

geez, decent network infrastucture costs a decent chunk of money and requires
a decent amount of requisite know-how.

if you don't have all of these, then you gotta do the best with what you can
get.

a friend of mine used to spout "Cheap, Fast, Good, pick two".

[_Greg_A_Woods]

[ On Thursday, March 15, 2001 at 17:09:14 (-0600), Stephen Sprunk wrote: ]

Subject: Re: Broken Internet?

Renumbering PCs is a trivial task. Reconfiguring hundreds (or
thousands) of routers, firewalls, etc. to account for the moved PCs is
not trivial. Renumbering servers is not trivial.

For _small_ networks (where this discussion started) even manual
reconfiguration of all the hosts (including servers) in an office, on a
floor, or even in a small building, would take less time than this
discussion has gone on for!

Keep in mind that Fortune 100 companies with multiple DS3s in several US
locations are in the same boat wrt renumbering. Most don't qualify for
portable addresses by ARIN's rules.

In essence all that matters are the public servers and hosts. In theory
none of an organisations internal network will be affected in any way by
renumbering or multi-homing issues. ARIN's rules are just fine no
matter how big your internal network is. If you're running multiple
high-speed connections in multiple locations then your organisation
should have the skill set necessary (or the ability to hire it) to
manage renumbering any given location on demand. If you're doing stupid
things and putting private internal hosts on public networks then you're
asking for all kinds of troubles, not just renumbering and multi-homing
issues.

Also, try convincing someone like AmEx or Citibank that they should put
their servers under someone else's physical control -- that'll be good
for a laugh. Sure, that's extreme, but where exactly do you draw the
line on who's "important" enough to host their own servers?

This isn't about telling people whether they're allowed to host their
own servers or not -- that's irrelevant. Everyone's completely free to
make whatever choice they find most suitable for their circumstances
(though often the average person will make drastically wrong risk
assessments surrounding these issues and will thus inevitably make the
wrong decision).