Traffic engineering and peering for CDNs

Lately I have been putting in some effort to maximize our IX connections by trying to work with the top 5-ish list of ASNs that still send us traffic via a paid transit connection despite the fact that we are both present on the same IX(s). In one case I missed the fact that one ASN wasn't using the IXs route-servers, that's on me for not spotting that one.

Even with proper IX peering in place though it seems like some CDNs are better at using the IX connections than others. ASN 15169 for instance does an excellent job sending more than 99.99% of traffic via the IX connection; thank you. While others only seem to manage to send 60 - 80% of traffic via the IX. What I am not understanding about the respective CDN's network wherein they don't send traffic to me through a consistent path? Is the content coming from widely different places and rather than transport it across their own network from a remote site they would rather hot-potato it out a local transit connection? Are their transit costs so low that they don't care about using an IX connection over transit unlike a small operator like me? Is this just a non-obvious issue wherein they maybe just can't originate enough of the traffic near the IX and therefore don't make use of the IX connection, again a hot-potato phenomenon?

Secondly can someone explain to me why some CDNs want a gigabit or two of traffic to be exchanged between our respective networks before they would peer with me via a public IX? I totally get those kinds of thresholds before engaging in a private interconnect but I don't understand the reluctance with regard to a public IX, that they are already established at. Is it again just a simple case of bandwidth economics that operate at a different scale than I can comprehend?

I'm hoping the community can shed some light on this for me as I'm trying to avoid grilling the operators that are working with me as I don't expect those front line individuals to necessarily have a full view of the factors at play.

Thanks,
Graham Johnston
Network Planner
Westman Communications Group
204.717.2829
johnstong@westmancom.com<mailto:johnstong@westmancom.com>
P think green; don't print this email.

What I am not understanding about the respective CDN's network wherein they don't send traffic to me through a consistent path? Is the content coming from widely different places and rather than transport it across their own network from a remote site they would rather hot-potato it out a local transit connection?

Depends on the CDN, but its possible the traffic is coming from different locations and not all CDNs even have a network, so if you don't have peering with their location serving the traffic (and they don't have a network), the traffic will have to come to you via other paths.

Are their transit costs so low that they don't care about using an IX connection over transit unlike a small operator like me? Is this just a

Maybe. Or maybe the traffic to you is small enough (to them) that you're not on their radar as a desirable peer. Or maybe they just haven't gotten around to sending you a peering request yet.

Secondly can someone explain to me why some CDNs want a gigabit or two of traffic to be exchanged between our respective networks before they would peer with me via a public IX?

Which ones want that much? We like to see some traffic before moving from "IX route-server peering" to direct peering via the IX, just because there are so many possible peers and only so much router resources. It's really not worth the resources (router or management) to direct peer with a network with which there's virtually no traffic being exchanged, just because we're on the same IX(s). 1-2G to peer seems kind of high. Some might insist that you move peering to PNI if you're doing >1-2G across an IX.

Hello,

Lately I have been putting in some effort to maximize our IX connections by trying to work with the top 5-ish list of ASNs that still send us traffic via a paid transit connection despite the fact that we are both present on the same IX(s). In one case I missed the fact that one ASN wasn't using the IXs route-servers, that's on me for not spotting that one.

Even with proper IX peering in place though it seems like some CDNs are better at using the IX connections than others. ASN 15169 for instance does an excellent job sending more than 99.99% of traffic via the IX connection; thank you. While others only seem to manage to send 60 - 80% of traffic via the IX. What I am not understanding about the respective CDN's network wherein they don't send traffic to me through a consistent path? Is the content coming from widely different places and rather than transport it across their own network from a remote site they would rather hot-potato it out a local transit connection? Are their transit costs so low that they don't care about using an IX connection over transit unlike a small operator like me? Is this just a non-obvious issue wherein they maybe just can't originate enough of the traffic near the IX and therefore don't make use of the IX connection, again a hot-potato phenomenon?

Most CDN’s do not have a backbone. Transit costs are not free, but as most traffic is served by local nodes from cache, the costs of transport between locations in many cases is higher than just sending via transit. In some cases, the CDN may not have good mapping and may not be certain which node is best to serve your customers. In other cases, not all content exists on all nodes, and they may redirect to serve from the nodes which have the content. Finally, there may be an outage or capacity limits from the closest location, and another location may be serving to make up the shortfall.

Secondly can someone explain to me why some CDNs want a gigabit or two of traffic to be exchanged between our respective networks before they would peer with me via a public IX? I totally get those kinds of thresholds before engaging in a private interconnect but I don't understand the reluctance with regard to a public IX, that they are already established at. Is it again just a simple case of bandwidth economics that operate at a different scale than I can comprehend?

This sounds like a surprisingly high threshold, but to some extent it boils down like this — setting up sessions requires some time. In the ideal world, the peer is intelligent and has everything set up properly, but even in this case, it still requires some time for making sure things go up properly. Some (but not all) CDN’s have it automated to reduce this time. Some potential peering networks are poorly run, and will leak routes, not announce all of their routes, will not configure the sessions properly, etc — this adds up to significantly more time. Before the CDN starts setting up peering with another network, it is not necessarily obvious if the potential peer is run by competent people or not. Many CDN’s are members of the route servers. If you are exchanging a small amount of traffic, and both you and the CDN are on the Route Server for the IX, there maybe no reason to set up direct sessions which will require both more coordination time for configuration, and more router cpu time/ram on an ongoing basis. From the perspective of the CDN, most likely, 1Gbps or less is a perfectly reasonable amount of traffic to exchange to peers who are learned only via the route server, and not directly.

I'm hoping the community can shed some light on this for me as I'm trying to avoid grilling the operators that are working with me as I don't expect those front line individuals to necessarily have a full view of the factors at play.

Thanks,
Graham Johnston
Network Planner
Westman Communications Group
204.717.2829
johnstong@westmancom.com<mailto:johnstong@westmancom.com>
P think green; don't print this email.

Best Regards,
-Phil Rosenthal
ISPrime

Some rely on performance testing to the client's DNS resolver and if they're not using on-net ones, they'll be directed to use a different CDN node.

Hi Graham!

In addition to the other two comments, I´d like to add some topics:

Lately I have been putting in some effort to maximize our IX connections

by

trying to work with the top 5-ish list of ASNs that still send us traffic

via a paid

transit connection despite the fact that we are both present on the same

IX(s).

In one case I missed the fact that one ASN wasn't using the IXs

route-servers,

that's on me for not spotting that one.

This brings up some ideas ... see here:

1) Check if the CDN is on the routeserver
2) Check if the CDN has maybe tagged his prefixes with a "do-not-announce"
tag (verify at looking glass) (relevant of course only for outbound traffic)
3) Try to establish a direct peering session with the CDN over the IXP so
that you are known to the CDN
   ... some CDN´s could maybe also prefer or have higher priority on direct
sessions than via only routeserver...
   ... quite some networks give you a larger set of prefixes on a direct
session...
4) Talk with the CDN and check his geolocation tagging's for your prefixes
and maybe let correct them after you have found out what they are doing
5) Think on the fact, that CDN´s could take their routing decision on the
geo-location of the used dns resolver server and not on the users ip address
6) Check, that your network or your customers are not referring to public
dns resolvers
7) Think on the fact, that ipv6 could maybe be in place and active too ...
so don´t forget your ipv6 path and ipv6 dns resolvers...
8) Check your RADB/RIPE/AFRINIC/... routing db entries - if you did
something wrong, maybe you are filtered... (check:
http://irrexplorer.nlnog.net/ and looking glasses)
9) Check that you have your set of IP prefixes in good order and do not
implement strange magic with asymmetric more specifics ... reduce everything
to supernetworks on all edges!
10) Check if you are not handing over bgp tagging's which could cause some
prevention automatics and check if
   you do not send high metric values which are maybe causing negative
routing decisions on the other side
11) Think on the fact, that CDN´s have the geographic distance as one of
their parameters - if the CDN node is too
   far away from your network - maybe a closer located content box via
transit is rated higher than maybe the long
   distance via peering (if there is no node or no node with the right
content next to you) ... typically CDN are very close to larger IXP´s...
12) Think on the fact, that your ip-transit upstream could be a peering for
the CDN ... This neutralizes the peering or transit rating
   from the point of view of the CDN, but of course not for you ... so: talk
with the CDN ... therefore you must be known ...
   therefore, you should have done a peering session ...
13) Talk with your IXP ... maybe he can help ...

Anything missing here? Anything wrong with my ideas?

(disclaimer: yes, I do work for an IXP ... but this is my personal opinion)

happy peering...
Bernd

Bernd Spiess (Manager Peering Service / DE-CIX Management GmbH)

as far as im aware ... a friend of mine on INEX in Ireland said most cdns
use source ip of the DNS requests to determine which network to direct them
to ... so if you use you have your own resolver on an ip address in your
network range cdns can accurately determine what network the request is
comming from and determine what ip address / what network that the cdn has
nearest to your network...

ff you use 3rd party dns servers for your clients... you may not get an
optimal ip answer for your dns queries from the CDNS involved

I hope this helps

Tom Smyth

Some CDN's use DNS (in addition to latency, congestion levels, busy
state, e.t.c.).

Others use Anycast routing, which I tend to prefer. The problem is the
latter run a network while the former may typically not.

Mark.

On 6/7/16, 2:46 AM, "NANOG on behalf of Mark Tinka"