Comcast doesn't appear to have any usable NOC contacts via whois, and
this issue is apparently very widespread. Comcast obviously has
multiple saturated paths out in this area, so if you're seeing issues
getting to your customers on Comcast...well, it's probably Comcast.
Sort of an ongoing/me too on last months thread about same...
Outbound traffic to Comcast via basically any of our (AS13331)
upstreams except Spectrum (AS11404) is experiencing very high packet
loss once inside Comcast's network, we suspect a router or link very
near to us in Seattle is failing. I've marked out the destination
customer IPs but they're available to Comcast engineers if they get in
touch with me directly or as noc@metapeer.com -- TIA -- and again,
sorry to the list.
Below are three failing/high loss traces, and then same three traces
from our other DC which is routing out via spectrum. The loss is
present on Cogent and L3 outgoing paths at the least, maybe others.
As of now I've routed around the problem point for via the good path.
fission:~# mtr -s 900 --report --report-cycles 30 1.2.3.4
fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev
ve75-gw1-xmr.metapeer.com 0.0% 0.4 1.4 0.3 11.8 2.9
te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.9 0.6 0.5 0.9 0.1
te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 6.7% 43.7 42.3 38.8 43.7 1.4
be-13-cr01.seattle.wa.ibone.comcast.net 3.3% 47.7 44.6 41.3 47.7 1.8
pos-0-7-0-0-cr01.denver.co.ibone.comcast.net 3.3% 69.4 68.7 65.0 71.3 1.5
he-5-15-0-0-cr01.350ecermak.il.ibone.comcast 6.7% 92.6 97.3 91.4 102.0 3.8
he-3-15-0-0-ar01.woburn.ma.boston.comcast.ne 10.0% 120.3 119.8 116.3 122.0 1.8
pos-0-1-0-0-ar01.needham.ma.boston.comcast.n 0.0% 122.0 120.5 117.1 123.9 1.8
po-80-ur01.deering.nh.boston.comcast.net 0.0% 121.4 121.3 118.3 133.8 2.8
po-21-ur01.concord.nh.boston.comcast.net 6.7% 119.6 130.1 119.3 272.8 31.8
te-1-0-0-ten01.concord.nh.boston.comcast.net 6.7% 118.7 120.0 117.6 121.4 1.3
??? 100.0 0.0 0.0 0.0 0.0 0.0
You have new mail in /var/spool/mail/root
fission:~# mtr -s 900 --report --report-cycles 30 5.6.7.8
fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev
ve75-gw1-xmr.metapeer.com 0.0% 0.3 0.6 0.3 5.2 0.9
te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.8 0.6 0.5 0.8 0.1
te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 0.0% 41.4 42.6 38.5 43.8 1.3
be-13-cr01.seattle.wa.ibone.comcast.net 0.0% 45.6 44.1 40.4 46.7 1.8
pos-0-8-0-0-cr01.denver.co.ibone.comcast.net 6.7% 71.2 69.0 65.0 72.1 1.9
he-5-12-0-0-cr01.350ecermak.il.ibone.comcast 3.3% 98.1 96.3 86.4 101.2 3.9
he-0-15-0-0-ar01.pontiac.mi.michigan.comcast 10.0% 93.5 97.3 93.5 100.6 2.3
xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.c 0.0% 97.2 99.1 92.9 141.8 10.4
te-17-10-cdn05.rochestrhlls.mi.michigan.comc 3.3% 118.9 113.2 104.6 118.9 3.5
??? 100.0 0.0 0.0 0.0 0.0 0.0
fission:~# mtr -s 900 --report --report-cycles 30 a.b.c.d
fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev
ve75-gw1-xmr.metapeer.com 0.0% 0.3 2.1 0.3 47.1 8.6
te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.7 0.6 0.5 0.8 0.1
te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 6.7% 40.6 42.4 39.7 45.2 1.3
be-15-cr01.seattle.wa.ibone.comcast.net 0.0% 47.3 48.2 41.3 54.8 4.3
68.86.92.34 3.3% 43.7 45.5 39.5 101.7 11.6
be-18-ur06.bellevue.wa.seattle.comcast.net 0.0% 43.8 43.0 40.2 44.3 1.4
te-3-0-0-ten15.bellevue.wa.seattle.comcast.n 6.7% 42.7 44.1 42.1 45.7 1.1
??? 100.0 0.0 0.0 0.0 0.0 0.0
mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 1.2.3.4
HOST: phobos Loss% Snt
Last Avg Best Wrst StDev
1. 207.229.74.1 0.0% 30
0.4 5.1 0.3 132.4 24.1
2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30
1.5 14.0 1.5 206.4 39.8
3. 23.30.206.33 0.0% 30
1.8 2.0 1.7 3.5 0.3
4. be-17-cr01.seattle.wa.ibone.comcast.net 0.0% 30
12.6 5.3 2.1 12.6 2.7
5. pos-0-4-0-0-cr01.denver.co.ibone.comcast.net 0.0% 30
28.5 29.2 27.2 31.4 1.3
6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 30
49.9 52.2 49.9 56.7 2.8
7. he-3-5-0-0-ar01.woburn.ma.boston.comcast.net 0.0% 30
78.5 79.5 78.3 81.0 1.0
8. pos-1-12-0-0-ar01.needham.ma.boston.comcast.net 0.0% 30
80.2 80.1 78.2 81.7 1.1
9. po-80-ur01.deering.nh.boston.comcast.net 0.0% 30
80.0 80.1 79.9 80.4 0.1
10. po-21-ur01.concord.nh.boston.comcast.net 0.0% 30
80.7 85.1 80.7 199.3 21.6
11. te-1-0-0-ten01.concord.nh.boston.comcast.net 0.0% 30
80.5 80.5 80.5 80.7 0.0
12. ??? 100.0 30
0.0 0.0 0.0 0.0 0.0
mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 5.6.7.8
HOST: phobos Loss%
Snt Last Avg Best Wrst StDev
1. 207.229.74.1 0.0%
30 0.3 3.6 0.3 96.3 17.5
2. agg1-sea-t7-8.bb.spectrumnet.us 0.0%
30 10.2 10.4 1.5 58.5 15.1
3. 23.30.206.33 0.0%
30 2.4 2.0 1.8 2.4 0.1
4. be-14-cr01.seattle.wa.ibone.comcast.net 0.0%
30 3.6 4.4 1.7 10.6 2.4
5. pos-0-3-0-0-cr01.denver.co.ibone.comcast.net 0.0%
30 31.1 29.5 27.2 31.6 1.3
6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0%
30 52.3 55.4 50.3 61.5 3.6
7. he-0-14-0-0-ar01.pontiac.mi.michigan.comcast.net 0.0%
30 58.9 58.0 56.1 60.1 1.2
8. xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.comcast.net 0.0%
30 71.2 58.9 56.6 83.6 6.4
9. te-17-10-cdn05.rochestrhlls.mi.michigan.comcast.net 0.0%
30 71.2 72.6 67.6 78.0 3.2
10. ??? 100.0
30 0.0 0.0 0.0 0.0 0.0
mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 a.b.c.d
HOST: phobos Loss% Snt
Last Avg Best Wrst StDev
1. 207.229.74.1 0.0% 30 0.7
5.1 0.3 143.4 26.1
2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 1.5
3.1 1.5 30.1 5.6
3. 23.30.206.33 0.0% 30 1.9
2.0 1.9 2.2 0.1
4. be-12-cr01.seattle.wa.ibone.comcast.net 0.0% 30 6.9
5.1 1.7 12.3 3.1
5. 68.86.94.70 0.0% 30 1.8
2.4 1.8 17.3 2.8
6. be-18-ur06.bellevue.wa.seattle.comcast.net 0.0% 30 3.5
3.4 3.3 3.5 0.1
7. te-3-0-0-ten15.bellevue.wa.seattle.comcast.net 0.0% 30 3.3
3.2 3.2 3.4 0.0
8. ??? 100.0 30 0.0
0.0 0.0 0.0 0.0
mloftis@phobos:~$