Enterprise Multihoming

On another list we've been having multihoming discussions again and I
wanted to get some fresh opinions from you.

For the past few years it has been fairly common for non-ISPs to
multihome to different providers for additional redundancy in case a
single provider has problems. I know this is frowned upon now,
especially since it helped increase the number of autonomous systems and
routing table prefixes beyond what was really necessary. It seems to me
that a large number of companies that did this could just have well
ordered multiple, geographically separate links to the same provider.

What is the prevailing wisdom now? At what point do you feel that it is
justified for a non-ISP to multihome to multiple providers? I ask
because we have three links: two from Sprint and one from Global
Crossing. I'm considering dropping the GC circuit and adding another
geographically-diverse connection to Sprint, and then removing BGP from
our routers.

I see a few upsides to this, but are there any real downsides?

Flame on. :slight_smile:

Thanks,
John

Who defines what is "really necessary"? What is your understanding
of "really necessary" when it comes to the desire to be commercially
and technically independent of your suppliers?

It's this discussion again.

Regards,
Daniel

Many/most of my external connectivity problems are provider-related rather
than circuit-related. Having two circuits to a single provider doesn't help
when that provider is broken. I'm not saying that multi-ISP BGP-based
multi-homing is risk-free, but I don't see multi-circuit single-provider as a
viable alternative.

IMHO you do not need a justification. If you think multiple links to the same provider don't buy you what you need (e.g. if the ISP has severe problems with its internal network multiple links do not buy you anything. Same holds when your ISP goes south which still happens now and then these days) go for real multihoming.

Arnold

John Neiberger wrote:

I see a few upsides to this, but are there any real downsides?

Connecting to single AS makes you physically resilient but logically dependent on single entity, be that a provisioning system, routing protocol instance, etc. Depending on your requirements, the option of having somebody redistribute all their BGP routes into ISIS or OSPF might not worth looking forward to.

Pete

Mutli-homing a non-ISP network or system on multiple carriers is a good way to maintain independent links to the internet by means of different peering, uplinks, over-all routing and reliability. My network on NAIS is currently multi-homed through AT&T. I use a single provider as both of my redundant links via 100% Fiber network. Even though this is cheaper for me, all it takes is for AT&T to have some major outage and I will be screwed. If I have a backup fiber line from say, Global Crossing, then it doesn't matter if AT&T takes a nose dive, I still have my redundancy there.

That is why most non-ISPs hold multihoming via different providers as their #1 choice.

Greg

John Neiberger wrote:

At what point do you feel that it is
: justified for a non-ISP to multihome to multiple providers?

If the business model allows for the downtime caused by putting all your
internet connectivity in one bucket.

james

Well, I think this, in many cases, boils down to being able to pick
the right provider.

I mean, some providers go belly-up from time to time. Others are
designed/run better.

For a major provider, complete outage of all of its customers is such
a big thing they'll want to avoid it always. If it happens, for a
brief moment, once in five years (for example), for most companies
that's an acceptable level of risk.

Date: Thu, 11 Mar 2004 18:21:03 +0200
From: Petri Helenius

Depending on your requirements, the option of having somebody
redistribute all their BGP routes into ISIS or OSPF might not
worth looking forward to.

Couldn't quite parse this, but it sounds scary.

Eddy

There is another thing - if you are multi-homed, and want to switch providers, it is pretty seamless and painless - no renumbering, no
loss of connection, etc., as you always have a redundant path.

Sure -- though many ISPs will probably let you keep the address space,
even if you switch away completely -- as long as you pay them enough
(or the other ISP to route it).

Bad practice, but has happened a lot, and probably still does :slight_smile:

FWIW, even if you are multihomed, that does not in and of itself
require that you "own" address space. Public AS number is often
enough (and even private will do, but that leads to other kind of
mess.)

John Neiberger wrote:

On another list we've been having multihoming discussions again and I
wanted to get some fresh opinions from you.

Whilst the topic's under discussion may I present myself as a lightning
rod :slight_smile: by asking:

(a) Has anyone here used any of the 'basement multi-homing in a box'
products such as Checkpoint's ISP Redundancy feature?

http://www.checkpoint.com/products/connect/vpn-1_isp_redundancy.html
(The 'VPN-1' brand is slightly misleading - it's a generic firewall.)

This allows edge networks to multihome between separate ISPs. When it was
first mentioned around the office I explained that it couldn't possibly
work, and my colleagues explained to me that I was full of it and that the
product is on the market and in use. (It has subsequently been lab'd here
and seemed to work between our main link (UUnet) and a humble BT DSL line.)
As far as I understand it, it's a form of NAT - the device keeps track of
which session's packets are going where and spreads traffic around. If one
ISP goes down it'll fail over to the other link.

(b) I suspect the answer will be a vehement 'no!' -- if so, why? Obviously
this won't scale terribly well at the service provider level but for edge
networks - what's wrong with it?

Obviously this only works for outbound sessions but there are plenty of
large enterprises happy to keep the majority of inbound services (web etc)
off in a nice secure hosting centre where real netops will use BGP for real
multihoming.

cheers

\a

E.B. Dreger wrote:

Date: Thu, 11 Mar 2004 18:21:03 +0200
From: Petri Helenius

Depending on your requirements, the option of having somebody
redistribute all their BGP routes into ISIS or OSPF might not
worth looking forward to.

Couldn't quite parse this, but it sounds scary.

I�m refering to the most popular way of causing an IGP meltdown. Obviously there are other ways, like software defects to make your IGP go mad. But when your upstream�s IGP does that, you want to have provider B to switch over to. It probably has gotten better when the Internet has matured but a few years back when I was more involved in day-to-day operations it was a few times a year when excersizing this option was the best course of action.

Pete

Jay Ford wrote:
[snip]

Many/most of my external connectivity problems are provider-related rather
than circuit-related. Having two circuits to a single provider doesn't help
when that provider is broken. I'm not saying that multi-ISP BGP-based
multi-homing is risk-free, but I don't see multi-circuit single-provider as a
viable alternative.

FWIW, I've had almost the exact opposite experience. Almost all of our
connectivity problems have been circuit issues. Two T1s to the same ISP
at one site has saved us from a lot of pain. OTOH, we also do have some
ISP diversity, though we haven't needed it nearly as much as redundant
circuits.

YMMV. HAND.

Date: Thu, 11 Mar 2004 20:31:52 +0200
From: Petri Helenius

I�m refering to the most popular way of causing an IGP
meltdown. Obviously there are other ways, like software
defects to make your IGP go mad. But when your upstream�s IGP
does that, you want to have provider B to switch over to.

Okay. I was unsure if you were referring to a clueless
downstream bloating their IGP, or a clueless transit network
redistributing downstream routes.

Eddy

I think its too easy, thats the problem. For <$1000 (excluding bandwidth/ccts)
you can buy a box, connect to your two providers, get an ASN and IPs and you're
away. Compare to the telephone network, to 'multihome' you need to get licenses,
allocations of numbers and codes thats not so easy, get some SS7 kit and do your
data builds.. you're talking quite a lot more money and certainly a lot more
difficult technically. Perhaps we should make the Internet more difficult :slight_smile:

I dont agree that connecting to two+ upstreams makes you better. In my
experience end networks have a couple of orders of magnitude more downtime than
a PoP in any reasonably large ISP. Ie the percentage theoretical improvement is
small.

In addition you seriously increase the complexity of your system, chances are
you're using the cheapest kit you could find (or at least cheaper and smaller
than what I would use).. its not great at BGP and may fall over when you get a
minor DoS attack, you probably generate flaps quite a bit from adhoc changes and
if you're announcing a /24 then thats going to get you dampened quickly.. so you
actually create a new weakest link. Also most of the corporates I've dealt with
take defaults rather than full tables.. so if the provider does have an issue
you still forward the traffic, theres no failover of outbound routing.

Even if you spend (waste) the money on some decent gear, you're on your own and
when a problem occurs the ISPs are going to be less helpful to you (not by
choice, I mean they dont have control of your network any more.. there knowledge
of whats causing problems is limited to the bit that they provide to you), so
chances are your problems may be more serious and take longer to diagnose and
fix.

IMHO avoid multihoming. You will know when you are big enough and you *need* to
do it, if you're not sure or you only want to do it cause you heard everyone
else is and its real cool then I suggest you dont.

Steve

Stephen J. Wilcox wrote:

IMHO avoid multihoming. You will know when you are big enough and you *need* to do it, if you're not sure or you only want to do it cause you heard everyone else is and its real cool then I suggest you dont.

There _is_ another element that I tried to point to yesterday.

If you are on record for making arguments about how there are better
ways to spend the money, and your boss's boss gets replaced by a
kid with all the tap-dance skills needed to sell smoke, flash and
sizzle, what you become is "unemployed".

And somebody half your age or less at less than your salary puts in
the new OCn's (n = 3-12) and all the rest.

Being right is important, but ...

I think its too easy, thats the problem.

Hoping that I don't sound too much like Bill Clinton, that depends on what you mean by "it." If "it" is multihoming, with your own ASN, to two providers, your raise some valid points.

Is there an intermediate alternative before you go all out? Yes, I think so, assuming your current provider has multiple POPs. Let me examine some of your points if we consider RFC 1998-style multi-POPping (I just invented that highly technical term) using PA address space.

For <$1000 (excluding bandwidth/ccts)
you can buy a box, connect to your two providers, get an ASN and IPs and you're
away.

Alternatively, another POP link, and preferably another router. If you are more concerned with loop failures than router failures, not a completely unreasonable assumption, you could get away with one router that has multiple interfaces, and spend some of the savings on backup power -- possibly a backup power supply in addition to the UPS, such as a Cisco RPS on their smaller routers. While you'll probably take a performance hit, or if you can reduce to critical traffic on an outage, you might get away with a second smaller router.

I dont agree that connecting to two+ upstreams makes you better. In my
experience end networks have a couple of orders of magnitude more downtime than
a PoP in any reasonably large ISP. Ie the percentage theoretical improvement is
small.

Like everything else, It Depends. My experience is that access links fail more often than provider routing systems, especially with a clueful provider. Since you can't guarantee that your physical connectivity to two different ISPs doesn't involve a shared risk group in the lines, there are still some things you may not be protected against.

One option, depending on the plant in your area, is that if you are considering a second router, consider putting it in a nearby building, reachable by WLAN (if you are minimizing costs), where that building minimally has different ducts to the telco end office, and ideally goes to a different end office. Not always possible, but to be considered. Longer-range wireless (radio or optical) links get more expensive.

In addition you seriously increase the complexity of your system, chances are
you're using the cheapest kit you could find (or at least cheaper and smaller
than what I would use).. its not great at BGP and may fall over when you get a
minor DoS attack, you probably generate flaps quite a bit from adhoc changes and
if you're announcing a /24 then thats going to get you dampened quickly..

That's a motivation for PA address space, where the provider aggregate is less likely to be small and easily damped.

so you
actually create a new weakest link. Also most of the corporates I've dealt with
take defaults rather than full tables.. so if the provider does have an issue
you still forward the traffic, theres no failover of outbound routing.

Again looking at intermediate solutions, there are always partial routes such as customer routes of the provier.

Even if you spend (waste) the money on some decent gear, you're on your own and
when a problem occurs the ISPs are going to be less helpful to you (not by
choice, I mean they dont have control of your network any more.. there knowledge
of whats causing problems is limited to the bit that they provide to you), so
chances are your problems may be more serious and take longer to diagnose and
fix.

Again, an operational advantage of multiPOPping and working with one carrier, although you aren't going to be protected against insanity of their BGP/

IMHO avoid multihoming. You will know when you are big enough and you *need* to
do it, if you're not sure or you only want to do it cause you heard everyone
else is and its real cool then I suggest you dont.

MHO would be to look at "multihoming" as a spectrum of solutions rather than a binary choice of single-provider-single-link versus multiple-provider. In given situations, you might also want to look at DSL or cable for diversity, tunneling to an ISP since the broadband provider is unlikely to be willing to speak BGP. Even dialup/ISDN, sometimes for critical workstations, has its place.

Shameless plug: I do go through these options in my book, Building Service Provider Networks (Wiley). Even there, though, I only run through the alternatives. You will still have to make your own cost-benefit decisions based on business policy, budget, clue level and cost of alternatives.

As Marshall noted multi-homing gives you the ability to switch providers
easily. This ability also gives you leverage with your network providers
since vendor lock-in does not exist.

This is a strong business case for multihoming and is one the financial
types understand and appreciate.

In a prior incarnation I worked for a distributor who had a online
ordering system. Our telcom coordinator got a "great" deal on bundled
internet service and telephony from a unnamed vendor. Due to the peering
arrangements the carrier had major customers were unable to place orders
in a timely fashion.

I set up a new AS and set up multihoming with another carrier and made our
customers happy again. Subsequently said carrier had an outage which took
down our link to them for 7 weeks. Since this was an internal problem at
our provider multiple links to this carrier would not have benefited us in
the least. A multihoming strategy also allows you to select providers who
provide connectivty to your business partners and customers which is
another win for obvious reasons.

                            Scott C. McGrath