BGP deployment and peering questions

Mailman List Mirror (Read Only)
1

Hi,

I have a couple of questions concerning BGP deployment. I would
appreciate any feedback from folks knowledged in the deployment.
Thanks very much.

1. What is the maximum no. of peers a core-BGP peers with externally?
   What is a good average or median number? How does this vary with
   Tier-1 BGP speakers vs. Tier-2 BGP speakers? Also, What is an
   average no. of peers a BGP border router multi-homes with? (Do not
   include Border routers with a single ISP peer - only the multi-homed
   border routers)

2. I understand, an AS by itself does not originate more than
   10,000 (UUnet being the one with this many) subnets. But,
   I believe, when you peer with a tier-1 ISP BGP speaker, you
   will get AS Paths for the entire 90,000+ routes (or whatever
   the maximum core routing tabel size is) exchanged at BGP
   connection setup time. On the other hand, I believe, the number
   of routes exchanged to be much less when you peer with a tier-2 BGP.
   What is a resonable average size of routing entries you could
   expect from a tier-2 ISP (and even a Tier-1 ISP, for that matter)?

3. Do yo have an estimate of memory requirements for some of the core
   routers (peering with tier-1 ISPs or tier-2 ISPs)? Is there a
   relation with the number of BGP peers?

Thanks.

Regards,
suresh

2

1. What is the maximum no. of peers a core-BGP peers with externally?
   What is a good average or median number? How does this vary with
   Tier-1 BGP speakers vs. Tier-2 BGP speakers? Also, What is an
   average no. of peers a BGP border router multi-homes with? (Do not
   include Border routers with a single ISP peer - only the multi-homed
   border routers)

I'm not sure what the maximum number supported by various OSes is, but most
people seem to limit it to around 30-50 per router. Of course, the realisitic
limit depends on router CPU and memory hardware more than anything else - a
Cisco 3640 isn't going to be able to handle as much as a 12000GSR.

An "average" is meaningless - there are many rotuers out there that are
multihomed to only two or three ISPs and therefore only have a handful of
BGP sessions.

2. I understand, an AS by itself does not originate more than
   10,000 (UUnet being the one with this many) subnets. But,
   I believe, when you peer with a tier-1 ISP BGP speaker, you
   will get AS Paths for the entire 90,000+ routes (or whatever
   the maximum core routing tabel size is) exchanged at BGP
   connection setup time. On the other hand, I believe, the number
   of routes exchanged to be much less when you peer with a tier-2 BGP.
   What is a resonable average size of routing entries you could
   expect from a tier-2 ISP (and even a Tier-1 ISP, for that matter)?

Any ISP shuld give you the entire internet routing table - 90k+ prefixes.
However, your router will only use a certain number of those as "best" routes.
The number of "best" routes per ISP depends entirely on who the ISPs are, and
doesn't (necessarily) have any relation to what Tier the ISPs are.

3. Do yo have an estimate of memory requirements for some of the core
   routers (peering with tier-1 ISPs or tier-2 ISPs)? Is there a
   relation with the number of BGP peers?

A Cisco 3640 with 64Mb will (Or at least did) just about handle a BGP
feed from two or three peers. Memory requirements (And CPU requirements)
increase with the number of peers, but 192Mb should be plenty for most
applications.

3

See below...

> 1. What is the maximum no. of peers a core-BGP peers with externally?
> What is a good average or median number? How does this vary with
> Tier-1 BGP speakers vs. Tier-2 BGP speakers? Also, What is an
> average no. of peers a BGP border router multi-homes with? (Do not
> include Border routers with a single ISP peer - only the multi-homed
> border routers)

I'm not sure what the maximum number supported by various OSes is, but most
people seem to limit it to around 30-50 per router. Of course, the realisitic
limit depends on router CPU and memory hardware more than anything else - a
Cisco 3640 isn't going to be able to handle as much as a 12000GSR.

An "average" is meaningless - there are many rotuers out there that are
multihomed to only two or three ISPs and therefore only have a handful of
BGP sessions.

A GSR with 256MB will handle many full views. I haven't tested to the
limits, but 50 would be a comfortable number. IOS based routers store
additional views relatively efficiently. I'm sure someone with Cisco can
give a better number, but from observation, if a full view take n memory,
than each additional full view takes about .1n memory.

On a Juniper, this relationship is much more linear. That's why most
Junipers come with much more memory - 256mb will go south after a dozen
full views. 768mb is recommended (and is what they ship with now,
BTW).

> 2. I understand, an AS by itself does not originate more than
> 10,000 (UUnet being the one with this many) subnets. But,
> I believe, when you peer with a tier-1 ISP BGP speaker, you
> will get AS Paths for the entire 90,000+ routes (or whatever
> the maximum core routing tabel size is) exchanged at BGP
> connection setup time. On the other hand, I believe, the number
> of routes exchanged to be much less when you peer with a tier-2 BGP.
> What is a resonable average size of routing entries you could
> expect from a tier-2 ISP (and even a Tier-1 ISP, for that matter)?

Any ISP shuld give you the entire internet routing table - 90k+ prefixes.
However, your router will only use a certain number of those as "best" routes.
The number of "best" routes per ISP depends entirely on who the ISPs are, and
doesn't (necessarily) have any relation to what Tier the ISPs are.

Routing policies (i.e. what you filter), have a much greater effect on
routing table size as perceived by a downstream

> 3. Do yo have an estimate of memory requirements for some of the core
> routers (peering with tier-1 ISPs or tier-2 ISPs)? Is there a
> relation with the number of BGP peers?

A Cisco 3640 with 64Mb will (Or at least did) just about handle a BGP
feed from two or three peers. Memory requirements (And CPU requirements)
increase with the number of peers, but 192Mb should be plenty for most
applications.

Not any more. We have had customer try this configuration and go to malloc
hell. 128mb is the minimum for a full view these days. A Riverstone RS8000
or a Cisco 3640 will handle several full views nicely with 128mb. Remember
the 3640 won't go above 128mb, though. A 3660 or a SSR makes a nice CPE
box if you need 256mb.

--
Ryan O'Connell - <ryan@complicity.co.uk> - http://www.complicity.co.uk

I'm not losing my mind, no I'm not changing my lines,
I'm just learning new things with the passage of time

Daniel Golding

4

256MB is not much anymore. Today, we see ~1.6million paths and ~100,000
routes. with a full iBGP mesh in the core, and current memory leaks from
ciscos IOS (know issues) the GSRs are running out of memory.

too bad we cant replace our GSRs with 7200s as they can hold 512M of ram
:slight_smile:

Christian

5

[snip various points that I agree with]

> A Cisco 3640 with 64Mb will (Or at least did) just about handle a BGP
> feed from two or three peers. Memory requirements (And CPU requirements)
> increase with the number of peers, but 192Mb should be plenty for most
> applications.

Not any more. We have had customer try this configuration and go to malloc
hell. 128mb is the minimum for a full view these days. A Riverstone RS8000
or a Cisco 3640 will handle several full views nicely with 128mb. Remember
the 3640 won't go above 128mb, though. A 3660 or a SSR makes a nice CPE
box if you need 256mb.

Unless Cisco have changed something, a 3640 can only take 64Mb of memory max.
You'd need a 3660 for 128Mb.

You can get away with a Cisco 3640 for full BGP, but it does depend on the
cards you have installed. NM-1A-T3 is a memory hog for example, but if
you've only got NM-2E2Ws you're probably OK. You might also need to tweak
iomem, but later IOS releases do this for you. 11.2 might be a better
choice for BGP-speaking 3640s are the memory requirements are lower.

6

Yes, you're quite right of course. I should know that, I have 3640s with 96Mb
in. (Must have more Caffiene before NANOGing...) And, of course, it's 256Mb
for a 3660.

7

Hi,
  
  I have a couple of questions concerning BGP deployment. I would
  appreciate any feedback from folks knowledged in the deployment.
  Thanks very much.

warning
do not consider myself knowledged in bgp deployment.
however, caida's skitter project
http://www.caida.org/tools/measurement/skitter
has more data on topology/connectivity (by far)
than anything i've encountered
  
and some of our analyses may not
be totally useless to you
(andre did all the work below;
i just translated it to english...)

  Pyda:
  1. What is the maximum no. of peers a core-BGP peers with externally?
     What is a good average or median number? How does this vary with
     Tier-1 BGP speakers vs. Tier-2 BGP speakers? Also, What is an
     average no. of peers a BGP border router multi-homes with? (Do not
     include Border routers with a single ISP peer - only the multi-homed
     border routers)

see:
http://www.caida.org/~broido/bgp/asoutdeg.html

  2. I understand, an AS by itself does not originate more than
     10,000 (UUnet being the one with this many) subnets. But,
     I believe, when you peer with a tier-1 ISP BGP speaker, you
     will get AS Paths for the entire 90,000+ routes (or whatever
     the maximum core routing tabel size is) exchanged at BGP
     connection setup time. On the other hand, I believe, the number
     of routes exchanged to be much less when you peer with a tier-2 BGP.
     What is a resonable average size of routing entries you could
     expect from a tier-2 ISP (and even a Tier-1 ISP, for that matter)?
  
http://www.caida.org/~broido/bgp/stubas.html

(viz could be better on some of the graphs,
but you'll get the point)

btw wrt your UUcomment
i think 701 originates about 2300 prefixes
(at least on 29 november seen from routeviews,
w another 74 announced by 701 and by other ASes... to routeviews, anyway)
(if you count 702-705,7046,284,etc, it was another few K maybe.
not more than 5K total even including uucanada.)
another 1K or so might be announced by 701
from stubs behind 701etc, haven't done that analysis

  3. Do you have an estimate of memory requirements for some of the core
     routers (peering with tier-1 ISPs or tier-2 ISPs)? Is there a
     relation with the number of BGP peers?
  
think someone else got that one
(good thing)

k and andre

//
  to the pure geometer the radius of curvature is
  an incidental characteristic -
  like the grin of the cheshire cat.
  to the physicist it is an indispensable characteristic.
  it would be going too far to say that to the physicist i
  the cat is merely incidental to the grin.
  physics is concerned with interrelatedness such
  as the interrelatedness of cats and grins.
  in this case the cat without a grin and the grin
  without a cat are equally set aside as purely
  mathematical phantasies.
    -- sir arthur eddington, the expanding universe
//