RE: Non-ISP companies multi-homing?

Mailman List Mirror (Read Only)
1

Nice lecture, thanks. But I saw always when thinking about multihoming some other problems also.

2] Multihomed to different ISP without BGP, supposing I need only one /24. It will be PA or PI addressing space ??
With respect to CIDR aggregation effort, make it sense to require /24 PI address block?? How will route the second ISP
my PA from the other ISP (he will de-aggregate the block of the second provider to announce more specific prefix??) ??

2] Multihomed to different ISP with BGP , supposing I need only one /24. Again, make it sense to have my own AS
and /24 PI block ?? Supposing to have IBGP to one ISP (to avoid assigning of independent AS)
and EBGP to the second ISP, again will my router announce the /24 inside of the first ISP address block to the EBGP peer ??

2. Multi-homed. Enterprise connects to more than one ISP. May or may not
use BGP. Wishes to protect against problems in the ISP routing system, and
will accept additional complexity and router requirements to get this. May
also have differing service agreements for Internet access for different
divisions.

2

Nice lecture, thanks. But I saw always when thinking about multihoming
some other problems also.

In the taxonomy I proposed, I tried to stay at the level of the customer
requirement, rather than the specific details of addressability. The cases
you cite are legitimate, but are in my mind at the next level of detail --
how one actually implements multihoming.

I'll make some general comments. I need to think a bit whether these would
be logically at a more detailed level of the taxonomy, or are in an
implementation taxonomy of their own.

Even beyond BGP capabiities (of either the enterprise or the ISP), your
examples are going to be affected by the particular ISPs' (and their
upstreams) aggregation and prefix filtering policies.

2] Multihomed to different ISP without BGP, supposing I need only one /24.
It will be PA or PI addressing space ??

From the taxonomy standpoint, it could be either. Current registry

policies from RFC 2050 would generally say PA.

With respect to CIDR aggregation effort, make it sense to require /24 PI
address block?? How will route the second ISP
my PA from the other ISP (he will de-aggregate the block of the second
provider to announce more specific prefix??) ??

2] Multihomed to different ISP with BGP , supposing I need only one /24.
Again, make it sense to have my own AS
and /24 PI block ??

Supposing to have IBGP to one ISP (to avoid assigning of independent AS)
and EBGP to the second ISP, again will my router announce the /24 inside
of the first ISP address block to the EBGP peer ??

Interesting approach. In general, the ISPs I know would be reluctant to
run iBGP with a customer, unless they had total control of all BGP
speakers. If I understand you correctly, the enterprise would have to tag
its advertisements to the second ISP with the ASN of the first, since the
enterprise doesn't have its own. Again, I think most ISPs would be
reluctant to give up this amount of control.

3

I think most of the companies running redundant links now have their own
address space and ASN. We got our primary address blocks back when a
company could still do that. I think there's going to have to be some
way to address that with semi-portable AS' in the near future though, as
more criticality transitions to the Net.

That, or people will start buying up service providers to get address blocks,
then they'll own the routers, and work out their iBGP issues "internally".
Not that that works for smaller companies who want it, but if you're a
multi-billion dollar corporation, it's an option (yes, it should scare you).

I know at least one tier 1 has started offering seperate wireline into
different NAPs in the DC area, which is about as good as you can get without
going to two providers. They want a lot of money for it though, and the
gains of a second provider are much more cost-effective from a strict
redundancy standpoint.

I don't know how we can get a combination of aggragate routing and
multi-homing to scale correctly, but I think it's becomming more
important that we do so.

Paul

4

>Supposing to have IBGP to one ISP (to avoid assigning of
>independent AS) and EBGP to the second ISP, again will
>my router announce the /24 inside of the first ISP
>address block to the EBGP peer ??

Interesting approach. In general, the ISPs I know would
be reluctant to run iBGP with a customer, unless they had
total control of all BGP speakers. If I understand you
correctly, the enterprise would have to tag its
advertisements to the second ISP with the ASN of the
first, since the enterprise doesn't have its own. Again,
I think most ISPs would be reluctant to give up this
amount of control.

Without the ISP having total control over the customer
router, a misconfiguration of filters on the customer side
could easily cause the customer to be a valid (and 1 hop)
path in the tables from ISP A to ISP B. Doesn't sound
like a possibility I would be willing to have hanging over
my head.

5

Without the ISP having total control over the customer
router, a misconfiguration of filters on the customer side
could easily cause the customer to be a valid (and 1 hop)
path in the tables from ISP A to ISP B. Doesn't sound
like a possibility I would be willing to have hanging over
my head.

Well, since my bandwidth is necessary for my business, I think I'd be
much more concerned about becomming the valid route than my upstreams, if
they get better routing through me, it's not necessarily a bad thing
for them unless they're concerned about me snarfing traffic.

Plus, you can filter out what you send to me if you're my upstream. That
means you'll need a misconfigured router on your side *and* one on mine.
I don't know your competency, but I'm fairly certain of mine ;). I put a
lot more time, effort and care into choosing a provider than you do into
choosing a customer.

I don't think it's as big of an issue, other than the obvious
effects of router filtering performance, and the chance that the upstream
could hose his filters when he goes to listen for routes to me from
external sources if he's already got major paranoia filters. Hopefully,
he's got that filtered to only happen from my other peering points though.

It's not rocket science, but it does take some care in set-up. You have
as much chance of getting control of my gateway routers as you have of
turning into a purple poodle. I'd purchase Yet Another Service Provider
and route a tier lower before I'd play that game. I've got a lot more to
lose than my upstreams.

Paul

6

You wrote:

>
> Without the ISP having total control over the customer
> router, a misconfiguration of filters on the customer
> side could easily cause the customer to be a valid (and
> 1 hop) path in the tables from ISP A to ISP B. Doesn't
> sound like a possibility I would be willing to have
> hanging over my head.

Well, since my bandwidth is necessary for my business, I
think I'd be much more concerned about becomming the
valid route than my upstreams, if they get better routing
through me, it's not necessarily a bad thing for them
unless they're concerned about me snarfing traffic.

They've also got to worry about your bandwidth, which
could become a big issue depending on the size of the two
providers involved.

Plus, you can filter out what you send to me if you're my
upstream. That means you'll need a misconfigured router
on your side *and* one on mine. I don't know your
competency, but I'm fairly certain of mine ;). I put a
lot more time, effort and care into choosing a provider
than you do into choosing a customer.
Paul

In the particular scenario being discussed, which routes
would you want from your upstream? You might want full
routes for the ability to actually choose best path, and
then the upstream providers loose control over what you
are sending where.

A and B can both filter what the customer sends to them
based on network, and then the problem is solved.
Unfortunately, this does not always give customers the
flexibility they are looking for.

I'm sure you know exactly what you are doing, but not
every Joe that a provider takes on does. My point is only
that this is a situation that I would not want to bring
upon myself.

7

Paul,

you clearly know what you are doing. But it's amazing how many
organizations don't understand fundamental global routing concepts, and
believe waving money at ISPs will make them do what they want even if that
makes no sense.

I've been doing design seminars for the pre-/post-sales tech support
organizations of several national-level carriers. In a recent class, the
students brought up a problem with one of their accounts, which I shall
call Major Clueless Bank (MCB).

Said bank wanted to offer consumer banking over the Internet. All their
direct connectivity came from my client, National Service Provider (NSP-1),
at several geographically dispersed points. By my taxonomy, single-homed,
multi-linked.

MCB desired to level the load over their various server farms and links to
NSP-1. They had fixated on BGP as the way to do what they thought they
wanted to do, which was to affect the MED passed to peers of NSP-1 based on
loading of their servers. They also wanted to affect NSP-1's interior
routing so they could advertise more specific routes to each of their
server farms, again based on _their_ load. Several million a year in
revenues were involved.

IMHO, on looking at what they were trying to do, it wasn't even a routing
problem. What they wanted was probably best done with DNS load control.
They simply did not realize that what they wanted in routing would have
marginal effect on the direct peers of NSP-1, and none on non-adjacent AS.
Their fundamental mental model was an enterprise network where they were in
control. And their next level of detail assumed everything could be
controlled with IP routing.

The concept that other traffic flowed in NSP-1, and that they could not
control the routing of other AS with whom they had no business
relationship, simply didn't penetrate.

So if the ISP has to set general policies,they need to protect themselves
against the NCBs of the world. Paranoid filtering isn't enough if the
customer is demanding something not possible. A part of making multihoming
practical is managing customer expectations and educating enterprise
network designers (or encouraging them to _have_ designers).

Howard

I see this again and again.

8

You wrote:
> >
> > Without the ISP having total control over the customer
> > router, a misconfiguration of filters on the customer
> > side could easily cause the customer to be a valid (and
> > 1 hop) path in the tables from ISP A to ISP B. Doesn't
> > sound like a possibility I would be willing to have
> > hanging over my head.
>
> Well, since my bandwidth is necessary for my business, I
> think I'd be much more concerned about becomming the
> valid route than my upstreams, if they get better routing
> through me, it's not necessarily a bad thing for them
> unless they're concerned about me snarfing traffic.

They've also got to worry about your bandwidth, which
could become a big issue depending on the size of the two
providers involved.

If they've oversold their provisioning, then yes, they would, but I can't
see how other than that they would. Perhaps I'm missing something? In
my particular case, my upstreams are UUNet and BBN, and I've been
particularly happy with the current arrangement.

In the particular scenario being discussed, which routes
would you want from your upstream? You might want full
routes for the ability to actually choose best path, and
then the upstream providers loose control over what you
are sending where.

I get full routes from my peers. That doesn't mean they send me traffic
based on destination addresses outside of those specifically linked to my
AS. Why would they route traffic destined to someone else through my
path if they were paranoid about me polluting things? I'd expect them to
no do that as much as I expect them to not accept routes advertised by
me that aren't in the address blocks I've specified. Maybe I'm missing
something here, but it seems pretty cut and dried, and other than the
filtering/CPU issues I don't see a major downside. Certainly my
upstreams didn't seem to have a problem implementing it, and it's saved
us bigtime a number of times since we started it.

I'm sure you know exactly what you are doing, but not
every Joe that a provider takes on does. My point is only
that this is a situation that I would not want to bring
upon myself.

I can understand that. In my case, it was a couple of years in coming,
but we'd planned for it at the start, and gotten agreements from the
providers to do it during circuit upgrades. I'd have dropped a provider
who wouldn't have agreed, since I had it as a critical need which it took
a while to get funded, and to get management to buy in to.

Long-term, I'm more concerned with the route aggragation problems once
other folks start jumping on the bandwagon than I am with any particular
application, mine included. Not just because I'm carrying full tables,
but because CIDR was a necessary evil, and we're basically moving towards
negating its advantages.

Paul

9

MCB desired to level the load over their various server farms and links to
NSP-1. They had fixated on BGP as the way to do what they thought they
wanted to do, which was to affect the MED passed to peers of NSP-1 based on
loading of their servers. They also wanted to affect NSP-1's interior
routing so they could advertise more specific routes to each of their
server farms, again based on _their_ load. Several million a year in
revenues were involved.

We went through that phase with senior management in one or two of our
divisions. It's surprising how some folks interpret "It doesn't do that"
to mean "Ask me again".

IMHO, on looking at what they were trying to do, it wasn't even a routing
problem. What they wanted was probably best done with DNS load control.

Distributed Director is looking better all the time, if only they'd drop
the price down to semi-managable. *sigh*

They simply did not realize that what they wanted in routing would have
marginal effect on the direct peers of NSP-1, and none on non-adjacent AS.
Their fundamental mental model was an enterprise network where they were in
control. And their next level of detail assumed everything could be
controlled with IP routing.

Bwaahahahah but I *do* control the Internet :wink:

The concept that other traffic flowed in NSP-1, and that they could not
control the routing of other AS with whom they had no business
relationship, simply didn't penetrate.

So if the ISP has to set general policies,they need to protect themselves
against the NCBs of the world. Paranoid filtering isn't enough if the
customer is demanding something not possible. A part of making multihoming
practical is managing customer expectations and educating enterprise
network designers (or encouraging them to _have_ designers).

Good point. I really just wanted to get a combination of things across,
firstly that it's doable, and I think we probably are doing it in the most
logical way, second of all, the routing infrastructure needs to change or
routing aggragation will break, and lastly that even though it isn't
always true, it is possible that the ISP is the least "victimized" in an
incorrect set-up.

But then, I think you've all got the easy jobs, since I have to deal with
most of the same issues (over 130 business units will do that), as well
as Appletalk, IPX and all the st00pid MS network garbage :wink: [1]

Paul "Arcserve backup is killing one of my internal 7513s" Robertson

[1] Yes, it's a troll, save the list follow-ups and flame directly

10

You wrote:

> You wrote:
> > >
> > > Without the ISP having total control over the
> > > customer router, a misconfiguration of filters on
> > > the customer side could easily cause the customer
> > > to be a valid (and 1 hop) path in the tables from
> > > ISP A to ISP B. Doesn't sound like a possibility I
> > > would be willing to have hanging over my head.
> >
> > Well, since my bandwidth is necessary for my
> > business, I think I'd be much more concerned about
> > becomming the valid route than my upstreams, if they
> > get better routing through me, it's not necessarily a
> > bad thing for them unless they're concerned about me
> > snarfing traffic.
>
> They've also got to worry about your bandwidth, which
> could become a big issue depending on the size of the
> two providers involved.

If they've oversold their provisioning, then yes, they
would, but I can't see how other than that they would.
Perhaps I'm missing something? In my particular case, my
upstreams are UUNet and BBN, and I've been particularly
happy with the current arrangement.

I think I see where the miscommunication lies. We were
discussing ISP's running IBGP sessions with multi-homed
customers. giving you the ability to announce routes to
another provider tagged with my AS is what makes me
nervous.

Are you announcing routes to BBN as AS 701? or to UUNet
from AS 1?

Besides that, becoming a valid shortest path between two
providers that do more traffic between them than your link
to either of them can handle IS dangerous for them,
because it restricts their other customers' ability to
talk to each other.

If one of my customers had 'router bgp 7019' somehwere in
their router configs, I wouldn't sleep well at night.

Peace,

  Gordon

11

Hmm. I could /maybe/ see allowing that as a seperate
  IBGP community, but even then it's dangerous.

  It's easy to get an ASN (even though it takes a while;
  I think the 'NIC still does those manually), so there's
  really no reason not to require multi-homed customers
  to do so if they want to talk BGP with you.

  But, seperate real ASN or seperate community, you'll
  still want to filter incoming announcements from the
  customer -- just as they'd probably want to filter
  announcements from you.

12

I think you mean confederation, JD.

Alec

13

You wrote:

>
> But, seperate real ASN or seperate community, you'll
> still want to filter incoming announcements from the
> customer -- just as they'd probably want to filter
> announcements from you.

I think you mean confederation, JD.

Alec

--

Don't think he did, Alec. Using communities would make it
much easier to filter the routes to the customer than
using confederation. I don't think there's any need to
implement confedrations here. Sounds like headaches I
don't need. Communities would allow you to filter very
specifically only routes coming from the router.

The real problem here is that the ISP with the EBGP
session still depends on the ISP with the IBGP session to
do things correctly, unless customer routes are filtered
at a network level -- Something I've never liked doing,
but always felt was necessary.

How can I have a setup that is flexible enough to satisfy
my customer (and my workload) but safe for me? I've had
customers running OSPF with one of my routers that was
redistributing OSPF into BGP, and it was probably one of
the stupidest mistakes I've ever made. Screwed me when
some dumbass decided he could use whatever networks he
wanted on the Sun they were running gated on.

14

Don't think he did, Alec. Using communities would make it
much easier to filter the routes to the customer than
using confederation. I don't think there's any need to
implement confedrations here. Sounds like headaches I
don't need. Communities would allow you to filter very
specifically only routes coming from the router.

Well, comparing a 'real AS to a separate community' doesn't really
sound right to me. Replacing community with confederation would make
more sense, although I do see your point. However I believe JD's
point is that it isn't _necessary_ to get a separate ASN if you've got
a small downstream who doesn't care about having his AS visible to the
outside world.

The real problem here is that the ISP with the EBGP
session still depends on the ISP with the IBGP session to
do things correctly, unless customer routes are filtered
at a network level -- Something I've never liked doing,
but always felt was necessary.

Unfortunately it is, as the AS7007 disaster illustrated all too
clearly.

How can I have a setup that is flexible enough to satisfy
my customer (and my workload) but safe for me?

MCI has a route registry that you send updates to just like the RADB
(the RADB and MCI RR actually exchange data). I believe MCI then
builds network-based access lists based on that database.

I've had customers running OSPF with one of my routers that was
redistributing OSPF into BGP, and it was probably one of the
stupidest mistakes I've ever made.

NONONONONO! Speaking IGP with customers bad!

Screwed me when some dumbass decided he could use whatever networks
he wanted on the Sun they were running gated on.

Yep, there's the problem. BGP was designed to be an inter-domain
routing protocol, and should be used as such. Unfortunately we need
some sort of network-level control over what a customer sends
upstream. Implementing some sort of automated scheme (like the MCI RR
for example) is IMO the only scalable way of doing so.

Alec

15

I think I see where the miscommunication lies. We were
discussing ISP's running IBGP sessions with multi-homed
customers. giving you the ability to announce routes to
another provider tagged with my AS is what makes me
nervous.

Ah, ok, that's the miscommunication.

Are you announcing routes to BBN as AS 701? or to UUNet
from AS 1?

Nah, they wouldn't let me, I'd have to fix UUNet's TCO, and the first thing
to go would be that damn ATM switch, *then* I'd have to fix BBN, and there
ain't that much time in the world.....

Paul

16

What's interesting to me is the few brave souls who are geographically
dispersed corporations using their own WAN for multihoming. If you already
have leased lines between your offices and each office has its own internet
access through a local provider, you can protect against failures of the
local providers by rerouting over the WAN leased-lines to another office
and popping out on *their* local provider's link.

Very cost effective, since you already are paying for the "backup link"
anyways as part of your enterprise network.