Network Topology

Whenever possible, use multiple sites. A couple of lower tech sites
will generally outperform one expensive site.

Here's my take on this. A lot of this is from personal experience.

In the beginning, a lot of smaller ISP's/hosting companies (at least in
our neck of the woods) start out with a single T1 to the internet. This
seems to be up most of the time. However, you are dependent on the
reliability of your upstream provider. Which may be good or not. Also,
during backhoe season this is definately not good.

The next logical step is to go to a pair of T1's to two different
upstreams, taking pains to insure that the path is geographically
diverse, so that a fiber cut or some "disaster" in one part of the country
shouldn't affect both connections.

From there you have two options. The first is to add additional T1's to

different providers. At this point, the law of diminishing returns has
probably taken effect and any minor benefit to having 3 T1's is probably
outweighed by trying to semi-balance traffic between all three providers.
In fact, you'll probably find that it's almost impossible to get 3 T1's to
not be significantly unbalanced. In addition, you chew more router
memory and cpu by having the router deal with 3 (or more) views instead of
just 2. I realize memory and CPU is cheap nowadays, but I seem to
always be able to find some knob which I'd rather use than another BGP

The other option is to get thicker Pipes to your 2 upstreams. This can
either be muxed T1's to get 3.0 or 4.5 or more mb/s or some sort of FT3,
depending on cost. My opinion is that this seems to be the more
beneficial of the two options. You can scale each pipe up as the
bandwidth on that pipe grows. This is also better internet wise as you
end up being able to just say to the router "pick the optimum path"
instead of saying "prefer a sub-optimum path so that things are
well-balanced here". In addition, it permits flows greater than a single
T1 would permit.

There is one exception to my "only 2 upstream" opinion. That is that
there comes a point where you find that a significant part of your
bandwith is going through one of your upstreams to another provider. At
that point, it may pay to get a pipe directly to that provider and either
take only customer routes from them or accept some subset of routes such
as those which must travel through them to get to their destination.
This exception also applies when it becomes cost effective to just "peer"
with those providers who will peer with you.

If you'd like to discuss this further not-on-list, feel free to e-mail me

- Forrest W. Christian (