The questions stand

Seems to depend on where you sit. John is only a few hops away (on the
same 'Tier 1' provider?) as Karl, so traffic is dependant on only three
parties maintaining a clean network. Across another backbone (via public
view sage.aspire.net) it's not so clean, probably due to packet loss

No webserver active on common ports at sage.aspire.net, so I traced from
here to there:

[OverKill]:~$ traceroute sage.aspire.net
traceroute: Warning: Multiple interfaces found; using 209.41.244.2 @ eth0
traceroute to sage.aspire.net (207.233.170.6), 30 hops max, 40 byte packets
1 eth1-core0.Columbus.EnterZone.Net (209.41.244.1) 0.464 ms 0.402 ms
0.378 ms
2 core1-eth0-ENTERZONE.Columbus.fnsi.net (209.115.127.21) 4.123 ms
2.925 ms 1.905 ms
3 border1-atm6.Vienna.fnsi.net (206.183.239.90) 31.108 ms 28.097 ms
34.240 ms
4 br1.tco1.alter.net (192.41.177.248) 44.899 ms 30.415 ms 31.101 ms
5 335.ATM2-0-0.CR2.TCO1.Alter.Net (137.39.74.73) 44.533 ms 46.970 ms
46.335 ms
6 412.atm11-0-0.gw2.tco1.alter.net (137.39.22.13) 35.126 ms 50.597 ms
29.909 ms
7 diginetusa-gw.customer.ALTER.NET (157.130.34.26) 52.351 ms 51.069 ms
29.571 ms
8 rt1b-ser0-VA001.diginetusa.net (207.233.201.22) 59.098 ms 60.608 ms
57.744 ms
9 sage.aspire.net (207.233.170.6) 42.377 ms 48.387 ms 46.359 ms

--- sage.aspire.net ping statistics ---
64 packets transmitted, 64 packets received, 0% packet loss
round-trip min/avg/max = 44.1/50.0/67.9 ms

Looks clean to me.

across the Tier 1 provider that they use (UUNet) or loss at an exchange
point between UUNet and Napnet to get to Karl's network.

Oh, Gawd! I have never, in my most bizzare nightmares contemplated
obtaining bandwidth from UUNet. They don't even have a POP in Columbus
according to the Worldcomm/UUNet sales puke who approached me at our co-lo
facility last week.

We obtain our bandwidth from Fiber Network Solutions, Inc. (FNSI.NET).
Karl from what I see is getting bandwidth or transit at least from GOOD.NET:

[OverKill]:~$ traceroute www.mcs.net
traceroute: Warning: Multiple interfaces found; using 209.41.244.2 @ eth0
traceroute to www.mcs.net (205.164.8.12), 30 hops max, 40 byte packets
1 eth1-core0.Columbus.EnterZone.Net (209.41.244.1) 0.478 ms 0.400 ms
0.381 ms
2 core1-eth0-ENTERZONE.Columbus.fnsi.net (209.115.127.21) 1.522 ms
1.454 ms 1.367 ms
3 aads.good.net (198.32.130.49) 13.534 ms 13.805 ms 13.705 ms
4 mcsnet.chicago.good.net (207.98.189.130) 14.305 ms 16.059 ms 14.862 ms
5 www.mcs.net (205.164.8.12) 14.279 ms 13.941 ms 14.021 ms

Pings:

--- www.mcs.net ping statistics ---
52 packets transmitted, 52 packets received, 0% packet loss
round-trip min/avg/max = 13.6/15.2/23.4 ms

From an OAR.NET connected machine, it looks like it transits through NAP.NET:

[tvo@noldor tvo]$ traceroute www.mcs.net
...blah blah blah...
7 oeb3-atm6-0.columbus.oar.net (199.18.202.13) 37.002 ms 38.028 ms
37.028 ms
8 tlp3-atm1-0.toledo.oar.net (199.18.202.53) 45.021 ms aads.nap.net
(198.32.130.39) 49.466 ms 50.494 ms
9 chi-h0.mcs.net (206.54.225.150) 50.948 ms 57.74 ms 51.727 ms
10 www.mcs.net (205.164.8.12) 50.387 ms 50.473 ms 50.216 ms
[tvo@noldor tvo]$

Pings:
--- www.mcs.net ping statistics ---
67 packets transmitted, 67 packets received, 0% packet loss
round-trip min/avg/max = 58.5/62.2/81.6 ms

(Note: The above connection is via ISDN so, I'll be nice and attribute the
latency to that VS OAR.NET's architecture.)

Despite our best intentions, packet loss and delay will always exists from
our networks to certain destinations, no matter whether we use 'Tier 1'
providers or companies that are creating private NAPs of their own. And
how it looks from me to you doesn't mean much, in the long run - how it
looks across the board from you to your customer's destinations is what
makes a difference. If that's accomplished with a small backbone
provider, a big player like MCI, or some local 'private nap' company
shouldn't make any difference to anyone but yourself.

In all honesty, the only time I have experienced packet loss to any
destination, it has been isolated and attributed to
(heat/cold/incompetence/emergency, unscheduled maintenance - PICK ONE,
seems like the MAE operators do) at one of the MAEs or some idiot with too
much backhoe and too little clue.

If you are accepting packet loss as "normal and unavoidable", your provider
is making excuses vs isolating and eliminating problems in their network or
in their interconnect at the MAEs.

If, on the other hand, you expect to get what you pay for and deliver it to
your customers without having to make excuses for problems in your
upstreams network, perhaps I can interest you in a circuit or co-location.

If you're one of the folks who is caught up on the Tier issue, I can sell
you a circuit from FNSI.NET as well.

As for how we look from the "outside":

MAE-East Looking Glass Results
  1 mae-east.fnsi.net (192.41.177.11) 0 msec 0 msec 4 msec
  2 core1-hssi7.Columbus.fnsi.net (206.183.239.89) [AS 6259] 28 msec 28
msec 32 msec
  3 ENTERZONE.Columbus.fnsi.net (209.115.127.22) [AS 6259] 28 msec 76 msec
36 msec
  4 www.enterzone.net (209.41.244.11) [AS 6259] 28 msec 36 msec 32 msec

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echoes to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/36/48 ms

Sprintnap Looking Glass Results
  1 phl2-core2-f4-0.atlas.digex.net (165.117.51.37) 0 msec 0 msec 0 msec
  2 dca1-cpe2-h10-0-0.atlas.digex.net (165.117.52.121) 8 msec 4 msec 4 msec
  3 dca1-cpe1-fa4-0-0.atlas.digex.net (165.117.224.10) 8 msec 8 msec 4 msec
  4 iad1-core1-p5-0.atlas.digex.net (165.117.52.230) 8 msec 4 msec 4 msec
  5 mae-east.fnsi.net (192.41.177.11) 16 msec 12 msec 4 msec
  6 core1-hssi7.Columbus.fnsi.net (206.183.239.89) [AS 6259] 36 msec 40
msec 44 msec
  7 ENTERZONE.Columbus.fnsi.net (209.115.127.22) [AS 6259] 36 msec 32 msec
44 msec
  8 www.enterzone.net (209.41.244.11) [AS 6259] 48 msec 44 msec 48 msec

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echoes to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 36/36/40 ms

MAE-West Looking Glass Results
1 mae-west.fnsi.net (198.32.136.111) 0 msec 4 msec 0 msec
  2 core1-hssi8.Columbus.fnsi.net (206.183.239.93) [AS 6259] 66 msec 68
msec 66 msec
  3 ENTERZONE.Columbus.fnsi.net (209.115.127.22) [AS 6259] 64 msec 60 msec
60 msec
  4 www.enterzone.net (209.41.244.11) [AS 6259] 60 msec 60 msec 64 msec

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echoes to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 60/60/60 ms

PAIX Looking Glass Results
  1 sjc3-core2-fa0-0-0.atlas.digex.net (165.117.53.194) 4 msec 0 msec 0 msec
  2 sjc1-core2-h5-0.atlas.digex.net (165.117.50.121) 4 msec 4 msec 0 msec
  3 sjc1-core1-fa3-0-0.atlas.digex.net (165.117.50.145) 4 msec 4 msec 4 msec
  4 pacbell.fnsi.net (198.32.128.5) 4 msec 8 msec 4 msec
  5 core1-hssi8.Columbus.fnsi.net (206.183.239.93) [AS 6259] 72 msec 78
msec 70 msec
  6 ENTERZONE.Columbus.fnsi.net (209.115.127.22) [AS 6259] 72 msec 70 msec
74 msec
  7 www.enterzone.net (209.41.244.11) [AS 6259] 70 msec 70 msec 70 msec

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echoes to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 80/80/84 ms

ROUTE-SERVER.CERF.NET Looking Glass Results
  1 h1-0.san-hq3.cerf.net (192.215.254.1) [AS 1740] 4 msec 4 msec 4 msec
  2 h1-0.san-hq3.cerf.net (192.215.254.1) [AS 1740] 0 msec 0 msec 0 msec
  3 h2-0-0.san-bb3.cerf.net (134.24.29.61) [AS 1740] 0 msec 4 msec 4 msec
  4 atm1-0-155M.san-bb1.cerf.net (134.24.32.6) [AS 1740] 4 msec
    atm6-0-1-155M.sjc-bb2.cerf.net (134.24.29.94) [AS 1740] 16 msec 16 msec
  5 mae-west.nap.net (198.32.136.13) 20 msec 20 msec 20 msec
  6 core0-a0-7.chi1.nap.net (207.112.247.146) [AS 5646] 100 msec 92 msec 92
msec
  7 core0-fe8-1.chi1.nap.net (207.112.247.153) [AS 5646] 108 msec 96 msec
108 msec
  8 aads.fnsi.net (198.32.130.64) [AS 6196] 208 msec 116 msec 104 msec
  9 ENTERZONE.Columbus.fnsi.net (209.115.127.22) [AS 6259] 100 msec 116
msec 116 msec
10 www.enterzone.net (209.41.244.11) [AS 6259] 104 msec 104 msec 120 msec

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echoes to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 100/105/120 ms

And from (http://zeus.lyceum.com/cgi-bin/nph-bgp-routes)

  1 Serial0.BR1.ATL2.lyceum.net (206.71.7.18) 8 msec 4 msec 4 msec
  2 ge001.rt1.ATL.netrail.net (207.153.92.1) [AS 4006] 8 msec 8 msec 8 msec
  3 gs012.rt1.DCA.netrail.net (205.215.63.13) [AS 4006] 44 msec 40 msec 44
msec
  4 gf020.rt2.DCA.netrail.net (205.215.4.2) [AS 4006] 44 msec 40 msec 40 msec
  5 mae-east.fnsi.net (192.41.177.11) [AS 701] 44 msec 52 msec 48 msec
  6 core1-hssi7.Columbus.fnsi.net (206.183.239.89) [AS 6259] 84 msec 80
msec 80 msec
  7 ENTERZONE.Columbus.fnsi.net (209.115.127.22) [AS 6259] 72 msec 72 msec
76 msec
  8 www.enterzone.net (209.41.244.11) [AS 6259] 72 msec 76 msec 72 msec

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echoes to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 72/74/76 ms

And just for grins and giggles, lets look from a few more countries:

CSL GmbH
  1 csl.D.net.DTAG.DE (194.245.101.62) 4 msec 4 msec 4 msec
  2 D-ag1.D.net.DTAG.DE (194.25.7.77) 4 msec 4 msec 4 msec
  3 D-gw1.D.net.DTAG.DE (194.25.122.118) [AS 3320] 8 msec 8 msec 8 msec
  4 K-gw1.K.net.DTAG.DE (194.25.120.150) [AS 3320] 8 msec 8 msec 8 msec
  5 F-gw3.F.net.DTAG.DE (194.25.120.229) [AS 3320] 12 msec 12 msec 12 msec
  6 TysonsC-gw1.USA.net.DTAG.DE (194.25.6.186) [AS 3320] 120 msec 108 msec
112 msec
  7 Penns-gw1.USA.net.DTAG.DE (194.25.6.193) [AS 3320] 108 msec 116 msec
108 msec
  8 sl-bb1-pen-2-0.sprintlink.net (207.143.32.174) [AS 4999] 116 msec 116
msec 116 msec
  9 sl-bb11-pen-0-1.sprintlink.net (144.232.5.9) [AS 1239] 116 msec 116
msec 116 msec
10 sl-bb7-pen-4-0-0.sprintlink.net (144.232.5.58) [AS 1239] 116 msec 116
msec 116 msec
11 sl-bb5-chi-1-0-0.sprintlink.net (144.228.10.38) [AS 1239] 136 msec 136
msec 136 msec
12 sl-bb11-chi-0-2.sprintlink.net (144.232.0.173) [AS 1239] 132 msec 132
msec 132 msec
13 sl-gw14-chi-8-0-0.sprintlink.net (144.232.0.194) [AS 1239] 140 msec 136
msec 136 msec
14 sl-napnet-2--T3.sprintlink.net (144.228.159.18) [AS 1239] 176 msec 176
msec 180 msec
15 aads.fnsi.net (198.32.130.64) [AS 6196] 388 msec 300 msec 336 msec
16 ENTERZONE.Columbus.fnsi.net (209.115.127.22) [AS 6259] 288 msec 184
msec 192 msec
17 www.enterzone.net (209.41.244.11) [AS 6259] 188 msec 188 msec

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 184/190/196 ms

AMS-IX Looking Glass Results
  1 Amsterdam1.cistron.net (195.64.68.4) 4 msec 4 msec 4 msec
  2 jedi-s0-5.amsterdam.wirehub.net (195.86.26.2) 4 msec 4 msec 8 msec
  3 ajani-s0.1.washington.wirehub.net (194.165.94.149) 108 msec 92 msec 92
msec
  4 mae-east.fnsi.net (192.41.177.11) 108 msec 100 msec 100 msec
  5 core1-hssi7.Columbus.fnsi.net (206.183.239.89) 136 msec 124 msec 136 msec
  6 ENTERZONE.Columbus.fnsi.net (209.115.127.22) 124 msec 120 msec 124 msec
  7 www.enterzone.net (209.41.244.11) 160 msec 124 msec 156 msec

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 124/140/152 ms

DTI Japan:
DTI NSPIXP-2 Router (http://neptune.dti.ad.jp/routechk-ixp2.html)
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echoes to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 220/224/228 ms

IMNET Japan:
IMNET/NSPIXP2 Looking Glass (http://www.inoc.imnet.ad.jp/lg/ix-lg.html)
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echoes to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 180/194/204 ms

LINX London:
LINX Looking Glass (http://www.linx.net/cgi-bin/lg.pl)
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 209.41.244.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 120/120/124 ms

I guess again, for some reason, things are fast and clean from my network.
They always have been since our move to Fiber Network Solutions, Inc
(FNSI.NET) for our bandwidth. Then again, I guess not all Tier 1's are alike.

In all honesty, the only time I have experienced packet loss to any
destination, it has been isolated and attributed to
(heat/cold/incompetence/emergency, unscheduled maintenance - PICK ONE,
seems like the MAE operators do) at one of the MAEs or some idiot with too
much backhoe and too little clue.

You're saying that you never experience packet loss to _any_ destination
unless it's some kind of emergency situation? I find that hard to
beleive. What if the destnation's network is chronically congested? Or
if the exchange point between your upstream provider and the destination's
provider is congested? Or some other fill_in_the_blank ongoing
non-emergency problem at the destiantion?

I have no doubt that you maintain an excellent network, and that you have
an excellent working relationship with your upstream provider, but how are
they going to control what is occuring within {big Tier 1 provider}.

If you are accepting packet loss as "normal and unavoidable", your provider
is making excuses vs isolating and eliminating problems in their network or
in their interconnect at the MAEs.

Obviously no-one is interested in putting up with packet loss or latency
if it can be avoided, but do you really beleive that _any_ nationwide
'backbone' provider is going to be able to isolate and eliminate every
problem with their public NAP interconnections? Somehow it seems much
more likely to me that at least one of the NAP interconnect destinations
is going to sit on their hands about it, for whatever reason.

However, my main point was that _any_ solution that works for the
customers, whether it be via the provider John has mentioned, or be via
private exchanges, or bouncing packets off of Mars, is the best solution,
and that there is no 'one-size-fits-all' solution for this ongoing
dilemma.

SGA