Seeing lots of reports of people unable to get to many Google services.
Seems to be affecting Comcast users disproportionately. It's fine for me,
but a lot of my staff are basically out of luck...but according to the
Google Apps Status page, everything is fine.
It's anecdotal, but it would seem like there's an issue based on these
reports.
Oh, and this:
http://www.cnn.com/2013/07/10/tech/web/google-down/index.html
Anyone know what's up? Fiber cut? DC outages?
-- blair
I did have connectivity issues for mobile devices this AM between 8:00am-10:00am EST. I am in NE Ohio.
Seems to be resolved now.
We saw the same thing, but seems to be cleared up now. All our providers
that routed to Google addresses in ATL had the issue. We have one provider
that lands on Google addresses in DFW, and it was working.
...And now I see that it isn't completely resolved. Some Google apps are
still inaccessible via the Atlanta routes.
Does anyone have traceroutes showing where the issues are?
-Grant
Traceroutes worked fine for me during the outage. Seems to have been
something at L4-L7.
We (were) peered with Google in Atlanta.
We were unable to bring up web pages for google.com or gmail.com.
Traceroute worked, though.
I shut off the peering & my outbound route switched to Cogent, which
works now.
I'll try peering again tonight. Maybe they'll have fixed it by then.
I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not
from Rogers/AS812 in Toronto. I've done a few other test traceroutes
through Rogers to verify that they didn't filter UDP/ICMP. At this point
nothing would surprise me from Rogers.
AS701
I just realized that it's not Google IP space (74.125.0.0/16), Rogers is
hijacking the DNS and resolving www.google.com to space within
64.71.240.0/20 which is Rogers IP space! (note the name server set as
8.8.8.8)
davinci#traceroute www.google.com
Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort.
Tracing the route to www.google.com (66.185.85.29)
1 x.x.x.x [AS 812] 4 msec 4 msec 4 msec
2 so-4-0-2.gw02.ym.phub.net.cable.rogers.com (66.185.82.129) [AS 812] 8
msec 8 msec 4 msec
3 69.63.252.222 [AS 812] 4 msec 0 msec 8 msec
4 69.63.250.162 [AS 812] 4 msec 4 msec 4 msec
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
TOR2-CORE-R1#show ip bgp 66.185.85.29
BGP routing table entry for 66.185.80.0/20, version 13095115
Paths: (1 available, best #1, table default)
Advertised to update-groups:
14
701 6461 812
205.205.23.121 from 205.205.23.121 (137.39.8.42)
Origin IGP, localpref 100, valid, external, best
TOR2-CORE-R1#
Thanks,
- Mike
tcptraceroutes working fine too?
I just realized that it's not Google IP space (74.125.0.0/16), Rogers is
hijacking the DNS and resolving www.google.com to space within
64.71.240.0/20 which is Rogers IP space! (note the name server set as
8.8.8.8)
so:
1) rogers is hijacking traffic to 8.8.8.8
2) the copy of 8.8.8.8 in rogers-land is replying with incorrect
information for google properties (at least).
err... any idea if it's lying about other things too? 
I just realized that it's not Google IP space (74.125.0.0/16), Rogers is
hijacking the DNS and resolving www.google.com to space within
64.71.240.0/20 which is Rogers IP space! (note the name server set as
8.8.8.8)so:
1) rogers is hijacking traffic to 8.8.8.8
2) the copy of 8.8.8.8 in rogers-land is replying with incorrect
information for google properties (at least).err... any idea if it's lying about other things too?
oops, so a polite caller noted that google often ships people at a
local 'google global cache' node if they have one... it seems like
that might be what's going on here with 8.8.8.8 sending you to 64.71
and/or 66.185 addresses.
so maybe rogers isn't doing something untoward after all!
-chris
(thanks for the headslap polite caller)
Hey Chris, long time!
From what I can tell, it's only Google Services (that I've found so far;
other things appear to resolve correctly). I'm wondering if they're
bouncing Google based traffic through some type of caching / accelerator?
Or maybe it's an NSA/DPI box 
I've tested Maps, Gmail, Translate as well as www.google.com,
www.google.caand they're all hijacked replies ---->>
Translating "www.google.com"...domain server (8.8.8.8) [OK]
(www.google.com)
Type escape sequence to abort.
Tracing the route to www.google.com (66.185.84.44)
(www.google.ca)
Type escape sequence to abort.
Tracing the route to www.google.ca (66.185.95.44)
(gmail)
Type escape sequence to abort.
Tracing the route to www3.l.google.com (66.185.85.39)
(maps)
Type escape sequence to abort.
Tracing the route to maps.l.google.com (64.71.249.114)
(translate)
Type escape sequence to abort.
Tracing the route to www3.l.google.com (66.185.84.30)
Cheers,
- Mike
That makes sense, I figured it was some type of a CDN caching environment.
I've found that youtube.com is in the same boat:
Translating "www.youtube.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort.
Tracing the route to youtube-ui.l.google.com (24.156.153.40)
[synkro:ROOT](~): jwhois 24.156.153.40 | grep -i orgname
OrgName: Rogers Cable Communications Inc.
[synkro:ROOT](~):
Thanks!
- MJ
That makes sense, I figured it was some type of a CDN caching environment.
I forget that we do this at times...
I've found that youtube.com is in the same boat:
Translating "www.youtube.com"...domain server (8.8.8.8) [OK]
yup, so this makes sense as well... good.
Hey Chris, long time!
From what I can tell, it's only Google Services (that I've found so far;
other things appear to resolve correctly). I'm wondering if they're
bouncing Google based traffic through some type of caching / accelerator?
Or maybe it's an NSA/DPI box
in .CA? don't you mean the meat-helmet-wearing CSE folks?
I've tested Maps, Gmail, Translate as well as www.google.com, www.google.ca
and they're all hijacked replies ---->>
yup... well, not hijacked in a bad sense... it's actually supposed to
be making things better.
Translating "www.google.com"...domain server (8.8.8.8) [OK]
Type escape sequence to abort.
Tracing the route to www.google.com (66.185.84.44)(www.google.ca)
Type escape sequence to abort.
Tracing the route to www.google.ca (66.185.95.44)(gmail)
Type escape sequence to abort.
Tracing the route to www3.l.google.com (66.185.85.39)(maps)
Type escape sequence to abort.
Tracing the route to maps.l.google.com (64.71.249.114)(translate)
Type escape sequence to abort.
Tracing the route to www3.l.google.com (66.185.84.30)
those all seem properly done, yes.
-chris