Akamai Edgekey issues ?

Hi There,

is anybody having any problems with sites that go through Akamai's CDN ?

I'm having problems to connect to many served by them, just two examples.

www.ti.com
www.newark.com

Cheers
Jorge

Well it seems that the NANOG list is back alive !!! I sent that message two
days ago (also reported via other channels that the list was not working)

It is not a persistent problem but intermittent and it seems to affect only
TWC customers that chose to use other public NS like Google's 8.8.8.8 and
trying to reach sites served by Akamai's CDN. One popular one
www.apple.comamong others.

Now if you switch back to TWC suggested and controlled NSs 209.18.47.61 and
209.18.47.62, everything seems to "work"

It is also a well known (imho) *bad* practice of TWC of tampering with DNS
query replies, in theory you should be able to opt-out of it going to some
obscure web page (http://www.dnsrsearch.com/prefs.php) which does not seem
to always work.

I spent almost two hours on the phone with TWC, unfortunately their basic
levels of support have no clue at all, I went up to Level 3 but found out
that I just escalated horizontally from India to Phillipines to Texas but
at the same clue level and not able to reach any engineering minds with a
clue.

The Level 3 rep didn't know what Akamai is, and at all levels they got lost
when I refused to go through the test your modem routine, and was based her
assumption that something was not probably working because she was getting
too many calls about the same issue. Quite effective monitoring system TWC
!!

I can't really identify if the problem is that TWC is tampering with akadns
query replies and then breaking the CDN or just plain bad routing/peering
or the NSA is running out of cycles/memory on the tap I've in my line :slight_smile:

Cheers
Jorge

Here is another bit of data... www.apple.com not reachable from a machine
using Google's NS, reachable from an iPad using TWC NS

IP addresses returned by each are different ... could be load balancing, or
creative (broken) traffic engineering

But for moments packets get lost at ae0.pr1.dfw10.tbone.rr.com this also
could be TWC routing/peering issues but I don't know how many more levels
of CS I have to go to reach somebody with a clue, after testing my modem
zillion times ...

pete@tango:~$ dig www.apple.com

; <<>> DiG 9.8.1-P1 <<>> www.apple.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46202
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.apple.com. IN A

;; ANSWER SECTION:
www.apple.com. 723 IN CNAME www.isg-apple.com.akadns.net
.
www.isg-apple.com.akadns.net. 49 IN CNAME www.apple.com.edgekey.net.
www.apple.com.edgekey.net. 408 IN CNAME e3191.dscc.akamaiedge.net.
e3191.dscc.akamaiedge.net. 9 IN A 184.86.141.15

;; Query time: 42 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Sep 2 21:06:57 2013
;; MSG SIZE rcvd: 161

pete@tango:~$ dig @209.18.47.61 www.apple.com

; <<>> DiG 9.8.1-P1 <<>> @209.18.47.61 www.apple.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20041
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.apple.com. IN A

;; ANSWER SECTION:
www.apple.com. 1135 IN CNAME www.isg-apple.com.akadns.net
.
www.isg-apple.com.akadns.net. 50 IN CNAME www.apple.com.edgekey.net.
www.apple.com.edgekey.net. 442 IN CNAME e3191.dscc.akamaiedge.net.
e3191.dscc.akamaiedge.net. 5 IN A 23.42.157.15

;; Query time: 39 msec
;; SERVER: 209.18.47.61#53(209.18.47.61)
;; WHEN: Mon Sep 2 21:07:09 2013
;; MSG SIZE rcvd: 161

Here is another bit of data... www.apple.com not reachable from a machine
using Google's NS, reachable from an iPad using TWC NS

IP addresses returned by each are different ... could be load balancing, or
creative (broken) traffic engineering

Far more likely to be simply due to Akamai
localizing the IP addresses to be as "close"
to the resolving nameserver as possible;
so, when using Google DNS, you end up
at an Akamai node "close" to the Google
DNS server; when using the TWC nameservers,
you end up pointing to an Akamai node closer
to those TWC nameservers.

Not a case of "broken" traffic engineering at all.

But for moments packets get lost at ae0.pr1.dfw10.tbone.rr.com this also
could be TWC routing/peering issues but I don't know how many more levels
of CS I have to go to reach somebody with a clue, after testing my modem
zillion times ...

Why not just use the TWC nameservers,
if thiings work when you use them instead
of the Google nameservers?

Matt

One reason would be that TWC used to hijack failed DNS requests and show
advertisements (
http://netcodger.wordpress.com/2010/09/14/roadrunner-returns-to-dns-hijack-tactics/
).

Also, Google DNS and OpenDNS helped manually clean up bad records after the
NYTimes had their nameservers changed at the TLD registry (
http://blog.cloudflare.com/details-behind-todays-internet-hacks).

—Scott

Sure it is.

It's assuming that the geographic location of a customer resolver server
has anything whatever to do with the geographic location of the end node,
which it's not in fact a valid proxy for.

Cheers,
-- jra

It isn't? How wrong is this assumption? Be specific. How far off is it, for how many users?

Perhaps look at the other side. Assumptions must be made. What assumptions would be better in the real world? What percentage of users are "closer" to anycast nodes? What are the real-world performance differences using this method vs. other methods?

Saying "not in fact a valid proxy" without hard data is not useful. What data do you have to prove your thesis?

Akamai seems to perform well for the vast majority of users. Or so I believe, but I fully admit I am biased. :slight_smile:

That said, always happy to be educated. If you have data, let us know.

From: "Patrick W. Gilmore" <patrick@ianai.net>

>> Not a case of "broken" traffic engineering at all.
>
> Sure it is.
>
> It's assuming that the geographic location of a customer resolver
> server
> has anything whatever to do with the geographic location of the end
> node,
> which it's not in fact a valid proxy for.

It isn't? How wrong is this assumption? Be specific. How far off is
it, for how many users?

Well, the vast majority of wireless users, for one: Sprint can't decide
whether I'm in Miami, Orlando, or Lenexa, Kansas on any given day.

More properly: people whose websites geolocate and whom I access
over Sprint 3g/Wimax/LTE. They're probably geolocating by the actual
assigned IP address, but it's roughly the same thing.

Perhaps look at the other side. Assumptions must be made. What
assumptions would be better in the real world? What percentage of
users are "closer" to anycast nodes? What are the real-world
performance differences using this method vs. other methods?

I didn't say there was a *good* answer to this, and in fact, I don't
think there is.

Saying "not in fact a valid proxy" without hard data is not useful.
What data do you have to prove your thesis?

Akamai seems to perform well for the vast majority of users. Or so I
believe, but I fully admit I am biased. :slight_smile:

Heh. :slight_smile:

That said, always happy to be educated. If you have data, let us know.

Well played, Patrick. No, it's anecdotal. But the percentage of wireless
users who are of necessity often nowhere near their DNS resolvers certainly
contributes to the percentages.

There are people who are manually stuck on the wrong network's servers, or
those who are configured to 4.4.4.4/8.8.8.8/4.2.2.1 by IT people (or themselves)
or to OpenDNS or the like, but I'd be surprised if those were more than 5%
overall.

It's mostly wireless. And that's a lot.

Is it relevant to Akamai? Possibly not.

For Akamai, the proxy may be Good Enough. Just so we remember that not everyone
is Akamai.

Cheers,
-- jra

Why not just use the TWC nameservers,
if thiings work when you use them instead
of the Google nameservers?

One reason would be that TWC used to hijack failed DNS requests and show
advertisements (
RoadRunner Returns To DNS Hijack Tactics | Net Codger
).

Without condoning or decrying this practice, I believe TWC allows you to opt-out of that. (Whether they should require you to "opt-out", or do it at all, is intentionally not discussed.)

Also, Google DNS and OpenDNS helped manually clean up bad records after the
NYTimes had their nameservers changed at the TLD registry (
Details Behind Today's Internet Hacks).

What makes you think TWC did not do the same?

And it was a lot more than the New York Times that had issues, and there was a lot more than a single instance of this.

To be clear, Google is Johnny On The Spot when these things happen, and kudos to them for it. But so are lots of other providers (e.g. OpenDNS, who has been doing this a lot longer than Google), they just might not have "teh GOOG" name to get them in the press & blogs.

.-- My secret spy satellite informs me that at 2013-09-03 8:07 AM Jay
Ashworth wrote:

There are people who are manually stuck on the wrong network's servers, or
those who are configured to 4.4.4.4/8.8.8.8/4.2.2.1 by IT people (or themselves)
or to OpenDNS or the like, but I'd be surprised if those were more than 5%
overall.

This can be improved by implementing support for the edns-client-subnet
feature ( http://www.afasterinternet.com/ ).

Both OpenDNS and Google support this, as well as numerous CDN's. Would
be great to have Akamai supporting this as well :slight_smile:

Cheers,
Andree