AT&T Hiccup

Alright, since things are mighty quiet on a Friday (either that or
everyone on NANOG is now blacklisted somehow and I'm not getting mail),
I'd figure to go the PITA route (as usual) and I'd ask a simple question
worthy of the mandatory responses of: "on topic questions are not the
topic of this list!", "this is not the outage list", "how is this
related to networking operations"

Did anyone see AT&T flake off the face of the map within the last 30-40
minutes? I know I did! (Seriously I did I did!)

Client: "We has no Internet"
Me: thinking to myself... "they need to call their provider... Due
diligence tracepath/ping/isitdownforeveryoneorjustme time... Global -->
AT&T looks sketchy"

# tracepath 12.200.xx.xx

3: so-1-3-1.585.ar1.JFK1.gblx.net (208.48.236.105) asymm 4 3.416ms
4: po3-40G.ar2.ATL2.gblx.net (67.16.131.198) asymm 7 141.179ms
5: no reply
6: no reply
NO REPLY 24 more x
31: no reply
     Too many hops: pmtu 1500
     Resume: pmtu 1500

AnotherClient (seconds later): "We has no VoIP"
Me: thinking to myself... "Damnit GBLX not again"

Another_Nother_Client (like even seconds after the first: "We has no p2p!"
Me: "time to tweet this question... No... post it to facebook... No -
make sure I update LinkedIn... wait?! NANOG"

Common denominator... All were on AT&T. So, was it only me or did one of
AT&T's cores take a long (3-5 minute) hiccup.

Post WTFH:

# tracepath 12.200.xx.xx

3: so-1-3-1.585.ar1.JFK1.gblx.net (208.48.236.105) asymm 4 3.774ms
4: te4-4-10G.ar5.NYC1.gblx.net (67.16.134.158) asymm 7 189.496ms
5: 192.205.37.137 (192.205.37.137) asymm 7 5.979ms
6: cr1.n54ny.ip.att.net (12.122.81.58) asymm 9 20.088ms
7: cr84.n54ny.ip.att.net (12.122.115.94) asymm 9 19.585ms
8: gar1.chsct.ip.att.net (12.122.105.117) asymm 7 10.234ms
9: 12.91.181.170 (12.91.18x.xxx) asymm 8 24.811ms
10: 12.200.42.67 (12.200.xx.xx) asymm 9
29.104ms reached
     Resume: pmtu 1500 hops 10 back 9

Just wanted to know if someone else saw it. All clients were spread
through the 12.x.x.x block not solely on 12.200.x.x

No issues with AT&T.

Two DS3's in West MI: Qwest towards NYC and AT&T
towards Chicago.

Now I did have a Qwest issue and it apepars that others towards NYC may
have had similar based on route ages of about ~1:45hrs ago.