Stephen,
I think this is the key point. Its common sense that peering
with the downstreams will improve user quality of service by
both reducing latency and taking unnecessary points of failure
out of the network.
Is it really common sense? If so, is the common sense correct?
In fact, there is a lot of recent work that suggests that there
can be a very poor (and as it turns out poorly understood)
interaction between richness of interconnection and BGP dynamics;
this is due, at least in part, to amplification and coupling
effects that appear in some large systems. So many argue that
that given the current set of protocols (i.e. BGP and its
implementations), increased topological richness beyond some
threshold can actually hurt robustness and reliability. And just
to be clear about this, this is not a statement about peering
policies themselves (I'm explicitly not commenting on that), but
rather about our current understanding of some of the dynamics
that exist in today's Internet.
I've been trying to capture some of this in the following
document (with the able help of Randy, Tim, and many others):
http://www.ietf.org/internet-drafts/draft-ymbk-arch-guidelines-03.txt
On the topic of interconnection richness and its (possibly
unanticipated) effects, Craig and Abha's early work on this is
maybe the canonical reference. For something a little more
recent, see "What is the Sound of One Route Flapping", Timothy
G. Griffin, IPAM Workshop on Large-Scale Communication Networks:
Topology, Routing, Traffic, and Control, March, 2002.
In any event, I guess the bottom line here is that sometimes what
looks like common sense (or even what we have a tendency to call
"conventional wisdom") may just be wrong.
Dave
Preaching to the ministers here:
I would like to see more data. I don't think a network with large
aggregates (some who can not peer with tier 1s due to current
policies) has much impact on the global routing structure.
The primary problem is the noise of smaller announcements popping
on and off magnified by multihoming punching holes in large aggregates.
Small announcement show more churn because they are more granular.
They expand the route table thus slowing convergence.
Scientific investigation and public sharing of data can help
networks build policies based on service criteria.
Many large networks are reluctant to share internal data that could
help in the broader analysis of stability issues.
Where is the threshold? Can I turn it into a policy?
How much does topology effect stability? Hierarchal design tends
to mitigate instability when it can be localized to a small segment
of your network infrastructure.
Flat designs tend to ring like a bell when instability is introduced.
I think we held the world record for flapping at NAP.NET in 95-96.
That was a flat design executed during a time when the Cisco architecture
and software could not keep up with the growth and churn rate. The
inclusion of algorithms that enhanced oscillation ringing (and since has
been fixed in IOS) did not help.
Have you ever seen an InterNAP route flap? Its good for around two minutes
or 120 traceroutes of pure humor, with a different loop across a different
backbone in a different city with every invokation.
Extensive peering relationships don't generally cause a breakdown of BGP,
which is probably the reason that we have settled into using that system.
Extensive transit relationships on the other hand, like those used by the
"optimized routing" crowd to try and take advantage of all the "richness
of paths" out there which aren't being used efficiently, break BGP very
very quickly (in my experience at any rate).
That makes sense ... many full routing tables is fare worse than
many partial routing tables. If my last resort was buying from a
Tier 1 after peering out most of my traffic I would prefer "paid
peering" or "partial transit". ... and one can always not listen to
routes that have multiple non optimized paths via transit
connections.
If you have 10 ways to an ASN and 3 or four stable and clean diverse
routes exist ... why chew up memory and CPU listening to the poor
routes?
[snip]
The primary problem is the noise of smaller announcements popping
on and off magnified by multihoming punching holes in large aggregates.
Small announcement show more churn because they are more granular.
They expand the route table thus slowing convergence.
Point: there's a body of data that indicates "multihoming" is not the
culprit. There's a lot of needless de-aggregating that has little or
nothign to do with multihoming, and mostly to do with lack of clue.
Both WRT limiting the scope of provider-based so-called "traffic
engineering" (CF ptomaine drafts) and that folks not using large tracts
of space can return blocks and get blocks that actually *fit* their
need.
Unfortunely there's a few companies/consultants whose business plan
requires them to graze on the commons and get all in a huff when any of
us tell them they're filtered because they are causing incremental damage
to our networks. Get over it kids; stable and deterministic behavior is
required for IP to work optimally.
Stability uber alles,
Joe
That makes sense ... many full routing tables is fare worse than
many partial routing tables. If my last resort was buying from a
Tier 1 after peering out most of my traffic I would prefer "paid
peering" or "partial transit". ... and one can always not listen to
routes that have multiple non optimized paths via transit
connections.
How will that work? Your last resort would then have to buy paid peering from
all non-transit networks (tier 1's), which means they have to be a kind of tier
1 themselves then?
But - BGP only propogates the single best route, BGP automatically removes the
"multiple non optimized paths" and if they're non-optimized they will never be
best and hence never cause flaps within or downstream of your network.
If you have 10 ways to an ASN and 3 or four stable and clean diverse
routes exist ... why chew up memory and CPU listening to the poor
routes?
I cant imagine a production router would have 10 ways tho, as above, any non
optimum routes will not be propogated past the eBGP.
... and assuming you arent taking transit then you rely upon your peerings as
the ONLY means of connectivity to their networks and their customer networks
which means multiple interconnect points and your still going to be receiving
multiple BGP routes to the same destination...
Note, I dont think I'm conflicting with the comments below which are referring
to route flaps at the source which therefore cross all networks and all bgp
routers, thats a separate issue.
Steve