Earthlink or BBN Tech Support? Forget it... Latency in Oakland/San Jose...

Ok, several days later and many multi-hour waits to get through to
someone who knows something at Earthlink has finally resulted in this
posting since I can't get help any other way. Maybe someone can shed
some light on if anyone is working on this particular problem.

Oakland, CA POP for mindspring/earthlink appears to have significant
latency and packet loss problems for the last week or so independent of
time of day. This appears to happen either through a session via
bbnplanet links or even through their own peering sessions there.

This might very well be nothing more than a BBNPlanet meltdown in San
Jose, however the first two included traceroutes show something other
than BBN... Any clues or anyone from either of these places care to
comment?

Examples include:

  1 <10 ms <10 ms <10 ms netopia.cstone.com [192.168.124.1]
  2 20 ms 40 ms 30 ms user-38lc1g1.dialup.mindspring.com
[209.86.6.1]
  3 90 ms 360 ms 30 ms cisco-g4-0-0.oak.mindspring.net
[209.86.81.129]
  4 460 ms * 441 ms
serial2-0-9.hsipaccess1.SanFrancisco1.Level3.net [166.90.48.37]
...

  1 <10 ms <10 ms <10 ms netopia.cstone.com [192.168.124.1]
  2 151 ms 60 ms 20 ms user-38lc1g1.dialup.mindspring.com
[209.86.6.1]
  3 30 ms 30 ms 30 ms cisco-g4-0-0.oak.mindspring.net
[209.86.81.129]
  4 470 ms 481 ms 471 ms sl-gw9-sea-1-0-0.sprintlink.net
[144.232.218.5]
  5 470 ms 461 ms 461 ms sl-bb10-sea-1-3.sprintlink.net
[144.232.6.101]
...

  1 <10 ms 10 ms <10 ms netopia.cstone.com [192.168.124.1]
  2 30 ms 30 ms 30 ms user-38lc1g1.dialup.mindspring.com
[209.86.6.1]
  3 30 ms 30 ms 30 ms cisco-g4-0-0.oak.mindspring.net
[209.86.81.129]
  4 30 ms 40 ms 30 ms s3-1-0.oakland-cr2.bbnplanet.net
[4.0.68.117]
  5 30 ms 40 ms 30 ms f0-0.oakland-br1.bbnplanet.net
[4.0.16.1]
  6 30 ms 40 ms 30 ms p5-2.snjpca1-cr1.bbnplanet.net
[4.0.1.194]
  7 30 ms 30 ms 40 ms p1-0.snjpca1-br1.bbnplanet.net
[4.24.9.133]
  8 451 ms 441 ms 450 ms p2-0.paix-bi1.bbnplanet.net [4.24.7.62]
  9 431 ms 440 ms 441 ms p7-0.paix-bi2.bbnplanet.net [4.0.3.142]
...

  8 * 30 ms 30 ms p9-0.snjpca1-br2.bbnplanet.net
[4.24.9.130]
  9 20 ms 30 ms 40 ms p1-0.sjccolo-dbe2.bbnplanet.net
[4.24.6.249]
10 30 ms 40 ms 30 ms p6-0.sanjose1-nbr2.bbnplanet.net
[4.0.6.233]
11 30 ms * 30 ms p12-0-0.sanjose1-cr2.bbnplanet.net
[4.0.3.198]
12 431 ms 440 ms 461 ms s2-1-0.xsanjose15-abovenet.bbnplanet.net
[4.0.85.5]
13 431 ms * 441 ms core6-sjc6-oc3.sjc1.above.net
[207.126.96.105]

As a quick hunch:

  Smacks of a problem on the reverse route being advertised to
Genuity's PAIX routers. I can't reproduce this from an OC-3 peering
session with them at PAIX.

  Of course, the best person to confirm this would be a Genuity
employee -- They have always been very responsive and helpfu. Have
you tried calling their NOC yet?

  --msa