Does anyone have any details about the Cogent outage that started this morning (9am GMT-400) and is still continuing ? If its a fibre cut between Montville (NJ?) and Cleveland OH (http://status.cogentco.com/) why is it so bad in Chicago and Albany locations ? Is there really that little excess capacity ?
My connection out of Toronto is pretty bad via Albany
3 g8-22.mpd01.yyz02.atlas.cogentco.com (38.104.158.77) 7.470 ms 6.754 ms 6.481 ms
4 v3493.mpd01.yyz01.atlas.cogentco.com (154.54.5.85) 6.981 ms 6.730 ms 6.984 ms
5 g2-0-0-3490.core01.yyz01.atlas.cogentco.com (154.54.5.73) 6.482 ms 7.175 ms 5.974 ms
6 p4-0.core01.alb02.atlas.cogentco.com (66.28.4.217) 105.954 ms 112.055 ms 111.426 ms
7 p6-0.core01.bos01.atlas.cogentco.com (154.54.7.42) 115.413 ms 117.090 ms 113.816 ms
and Bell's through Chicago is even worse
6 64.230.229.5 (64.230.229.5) 12.572 ms 36.983 ms 200.187 ms
7 64.230.242.97 (64.230.242.97) 4.685 ms 5.439 ms 3.645 ms
8 64.230.147.14 (64.230.147.14) 14.351 ms 15.344 ms 14.387 ms
9 206.108.103.142 (206.108.103.142) 14.374 ms 14.280 ms 14.255 ms
10 p13-0.core01.ord01.atlas.cogentco.com (154.54.11.29) 156.616 ms * 142.150 ms
11 te3-1.mpd01.ord01.atlas.cogentco.com (154.54.1.206) 135.199 ms 138.900 ms *
12 t2-4.mpd01.mci01.atlas.cogentco.com (154.54.2.233) 152.292 ms 149.956 ms 148.095 ms
13 t4-2.mpd01.iah01.atlas.cogentco.com (154.54.5.221) 149.047 ms 150.556 ms 151.232 ms
---Mike
Not seeing any Cogent problems in Tyson's Corner, Virginia
Regards
Marshall
Yes, their status page is not accurate. We're seeing traffic hitting the bitbucket at various locations on their network including Dallas (IAH) and Ashburn (IAD). It's be nice if they pulled their routes for this stuff.
For example:
traceroute to grouse.dabbledb.com (64.15.129.72), 64 hops max, 40 byte packets
1 38.99.21.1 (38.99.21.1) 2.012 ms 1.122 ms 0.468 ms
2 g0-10.na21.b003104-1.sfo01.atlas.cogentco.com (38.104.128.129) 1.229 ms 2.223 ms 0.975 ms
3 g1-7.111.core01.sfo01.atlas.cogentco.com (38.112.39.45) 1.758 ms 1.153 ms 2.523 ms
4 p4-0.core01.sjc01.atlas.cogentco.com (66.28.4.94) 2.010 ms 2.290 ms 3.886 ms
5 p14-0.core01.iah01.atlas.cogentco.com (66.28.4.237) 47.753 ms 46.791 ms 47.996 ms
6 * * *
7 * * *
8 * * *
9 *^C
-david
Mike Tancsa wrote:
We're going crazy up here, I'm trying to nail down where exactly the problem is - We don't use Cogent anywhere, but we're having terrible problems with Bell and many sites in Europe...
Mike Tancsa wrote:
We're going crazy up here, I'm trying to nail down where exactly the problem is - We don't use Cogent anywhere, but we're having terrible problems with Bell and many sites in Europe...
Bell uses Cogent in a large way. The second traceroute was from an IP in their AS (577) out. I am prepending out Cogent, but Bell does everything it can not to use Teleglobe so I am having problems influencing their routes to come back that way. They also have a very odd path out of Chicago. This is from a site in Toronto (source IP in AS577) back to me peering with Cogent's router in Toronto... Toronto, Chicago, Kansas, Texas, Washington, Boston, Albany, Toronto. Thats quite the milk run. Usually its Toronto, Chicago, Toronto.
% traceroute -f 6 -q1 199.212.134.3
traceroute to 199.212.134.3 (199.212.134.3), 64 hops max, 40 byte packets
6 64.230.229.5 (64.230.229.5) 4.846 ms
7 64.230.242.97 (64.230.242.97) 4.030 ms
8 64.230.147.14 (64.230.147.14) 14.213 ms
9 206.108.103.142 (206.108.103.142) 14.216 ms
10 p13-0.core01.ord01.atlas.cogentco.com (154.54.11.29) 101.348 ms
11 te3-1.mpd01.ord01.atlas.cogentco.com (154.54.1.206) 102.507 ms
12 t2-4.mpd01.mci01.atlas.cogentco.com (154.54.2.233) 108.993 ms
13 t4-2.mpd01.iah01.atlas.cogentco.com (154.54.5.221) 108.496 ms
14 t2-2.mpd01.dca01.atlas.cogentco.com (154.54.2.145) 110.221 ms
15 t8-2.mpd01.bos01.atlas.cogentco.com (154.54.1.105) 129.529 ms
16 g2-0-0.core01.bos01.atlas.cogentco.com (154.54.2.213) 108.507 ms
17 p13-0.core01.alb02.atlas.cogentco.com (154.54.7.41) 116.526 ms
18 p14-0.core01.yyz01.atlas.cogentco.com (66.28.4.218) 118.463 ms
19 v3491.mpd01.yyz01.atlas.cogentco.com (154.54.5.78) 217.912 ms
20 v3492.mpd01.yyz02.atlas.cogentco.com (154.54.5.82) 225.491 ms
21 sentex.demarc.cogentco.com (38.104.158.78) 217.134 ms
22 i3-vl-814 (67.43.129.242) 65.576 ms
23 shell1 (199.212.134.3) 66.221 ms
---Mike
Bear in mind that doing "everything they can not to use Teleglobe" probably involves local preference. Local preference comes before AS path length in the BGP selection order, so nothing you can do with prepending is going to help. You'll need to either keep them from seeing the undesirable path at all (drop the announcement, ask your upstreams to limit its propagation, etc.) or convince Bell not to use it. Depending on the setup, you may be able to limit route propagation with communities, or it may require some phone calls.
-Steve
This appears to be affecting Telia as well. Here was their last update:
"Concerning the cable break near Cleveland we have been informed that the cables have been intentionally sabotaged. The provider informed that they need to change the whole damaged fibre part and that is 3600 feet. Fibre has been ordered and ETA is 1900 UTC. Once the fibre arrives they need to blow it into the 3600 feet long duct before the splicing can start."
-Eric
As opposed to ‘unintentionally sabotaged’? I think there is some redundancy there …
Sorry for the cheap shot, it was just too tempting.
Roderick S. Beck
Director of EMEA Sales
Hibernia Atlantic
1, Passage du Chantier, 75012 Paris
http://www.hiberniaatlantic.com
Wireless: 1-212-444-8829.
Landline: 33-1-4346-3209
AOL Messenger: GlobalBandwidth
rod.beck@hiberniaatlantic.com
rodbeck@erols.com
``Unthinking respect for authority is the greatest enemy of truth.’’ Albert Einstein.
Eric Spaeth wrote:
This appears to be affecting Telia as well. Here was their last update:
"Concerning the cable break near Cleveland we have been informed that the cables have been intentionally sabotaged. The provider informed that they need to change the whole damaged fibre part and that is 3600 feet. Fibre has been ordered and ETA is 1900 UTC. Once the fibre arrives they need to blow it into the 3600 feet long duct before the splicing can start."
Sounds like a DHS/FBI investigation will be starting soon.
Eesh.. if we start having to secure 500,000 route miles of fiber routes against sabotage, um... well, I guess I'll have to become a fiber installation contractor. 
DJ
Eric Spaeth wrote:
This appears to be affecting Telia as well. Here was their last update:
"Concerning the cable break near Cleveland we have been informed that the cables have been intentionally sabotaged. The provider informed that they need to change the whole damaged fibre part and that is 3600 feet. Fibre has been ordered and ETA is 1900 UTC. Once the fibre arrives they need to blow it into the 3600 feet long duct before the splicing can start."
-Eric
Um, anyone have any information on how, why and who might be involved with this sabotage? I'm not suggesting an increase in the threat level, but it might be nice to know if the "Free Silicon" movement has decided to make this a day of protest or something....
Anyone spots anything, please post!
That and carriers will have to stop value-engineering route diversity out of their networks.
jms
I see Cogent has updated their page - so we think this cut is fixed now?
Eric Spaeth wrote:
Bell uses Cogent in a large way. The second traceroute was from an IP in their AS (577) out. I am prepending out Cogent, but Bell does everything it can not to use Teleglobe so I am having problems influencing their routes to come back that way. They also have a very odd path out of Chicago. This is
Bear in mind that doing "everything they can not to use Teleglobe" probably involves local preference. Local preference comes before AS path length in the BGP selection order, so nothing you can do with prepending is going to help.
Yes, I realize that. I think its because they (Bell) pay Teleglobe for transit, so they dont want to use it where possible. Back when I signed up with Teleglobe, I was hoping there were some community tricks I could use to influence bell's local pref, but because they buy transit from Teleglobe, this was not implemented. I think the only thing I could do would be to withdraw the prefixes from Cogent or resort to deaggregation so that they would follow a more specific prefix.
---Mike