Facebook Issues/Outage in Southeast?

Yes tis down. Watch productivity go up

Productivity grinds to a hault as everyone goes onto twitter to talk about facebook being down....

I can ping them but not access them over Cogent here in Tysons Corner, Virginia - looks like a server issue to me.

server:temp tme$ wget www.facebook.com
--15:52:17-- http://www.facebook.com/
           => `index.html'
Resolving www.facebook.com... 216.66.8.178, 216.66.8.225
Connecting to www.facebook.com|216.66.8.178|:80... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
15:53:03 ERROR 500: Internal Server Error.

server:temp tme$ ping -c 5 216.66.8.178
PING 216.66.8.178 (216.66.8.178): 56 data bytes
64 bytes from 216.66.8.178: icmp_seq=0 ttl=55 time=15.865 ms
64 bytes from 216.66.8.178: icmp_seq=1 ttl=55 time=15.693 ms
64 bytes from 216.66.8.178: icmp_seq=2 ttl=55 time=15.851 ms
64 bytes from 216.66.8.178: icmp_seq=3 ttl=55 time=15.953 ms
64 bytes from 216.66.8.178: icmp_seq=4 ttl=55 time=15.823 ms

--- 216.66.8.178 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 15.693/15.837/15.953/0.084 ms
server:temp tme$

Regards
Marshall

It may be some chunk of akamai being down/unreachable, no noticeable
issues from TWC in SATX.

J

I can ping them but not access them over Cogent here in Tysons Corner, Virginia - looks like a server issue to me.

Want to see something funnier:

Exactly the same as what your seeing for facebook. Working icmp, broken http.
I wonder if 10^234 people are all trying to find out if facebook is
down by going to this site, and crashing it.. :slight_smile:

IPv6 seems to be working fine for me www.v6.facebook.com :slight_smile:

Cameron

Change that, it is down from here too now ... packets don't get out of
tbone.rr.com from here.

J

Looks that it came back up now ...

root@tango:/var/cache/bind# ping -c 10 www.facebook.com
PING a1254.g.akamai.net (209.18.43.169) 56(84) bytes of data.
64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169):
icmp_seq=1 ttl=57 time=15.8 ms
64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169):
icmp_seq=2 ttl=57 time=16.9 ms
64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169):
icmp_seq=3 ttl=57 time=17.2 ms
64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169):
icmp_seq=4 ttl=57 time=16.6 ms
64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169):
icmp_seq=5 ttl=57 time=17.2 ms
64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169):
icmp_seq=6 ttl=57 time=16.2 ms
64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169):
icmp_seq=7 ttl=57 time=17.0 ms
64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169):
icmp_seq=8 ttl=57 time=17.2 ms
64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169):
icmp_seq=9 ttl=57 time=16.7 ms
64 bytes from 209-18-43-169.dfw10.tbone.rr.com (209.18.43.169):
icmp_seq=10 ttl=57 time=16.7 ms

J

are they down coz of DDoS?

Want to see something funnier:
http://downrightnow.com/

Exactly the same as what your seeing for facebook. Working icmp, broken http.

downforeveryoneorjustme.com is/was returning intermittent 500 errors, too. fun day.

..d (twiddling his thumbs waiting to test newly built servers that require facebook)

Yep, works great. You guys should really upgrade your networks to
something that works. :wink:

~Seth

We've always been able to ping facebook, it seems the people with the most trouble are hitting akamai's caching. The 209.165.150.24 is local to us, as we host akamai for facebook in Alaska.

ping www.facebook.com

Pinging a1254.g.akamai.net [209.165.150.24] with 32 bytes of data:

Reply from 209.165.150.24: bytes=32 time=41ms TTL=57
Reply from 209.165.150.24: bytes=32 time=4ms TTL=57
Reply from 209.165.150.24: bytes=32 time=4ms TTL=57
Reply from 209.165.150.24: bytes=32 time=8ms TTL=57

Ping statistics for 209.165.150.24:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 4ms, Maximum = 41ms, Average = 14ms

its up and running now. try 69.63.189.16

I heard it's an issue with users going through Quest.

Completely down again (UK).

+1 north eastern north america

yes, and Qwest is no longer experiencing issues according to IHR.

-b

http://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919

Apparently, our surmise about Akamai notwithstanding, the problem was actually
internal to their app-specific caching facilities, which went into Sorcerer's
Apprentice mode, and they had to kill them all and let ghod sort them out.

More if I get it; hope that posting's public.

Cheers,
-- jra

I'm hoping (desperately) that someone other than me sees the full irony
in this statement?

I also have visions of hundreds of techs worldwide thinking their snort
boxes have hung.

Gord

Agreed; my reading of this suggests database caching issues (i.e. all the frontend/middleware clients hitting the main sql cluster at once instead of the memcached farm they normally use), not HTTP/CDN caching issues.

-C