Hello,
I'm trying to find out the best way to consolidate connectivity on an island.
The current issues are :
- Low redundancy of old cables (2)
- Low system capacity of said cables (<=20Gbps)
- Total service loss when both cables are down because of congestion on satelite backups
- Sheer price of bandwidth
On the plus side, there are over 5 AS on the island, an IXP and small-ish collocation capacity (approx 10kW available, could be upgraded, second site available later this year).
We'd like to host cache servers and/or VMs on the IXP, with an option to anycast many services - without hijacking them, that goes without saying - such as quad-whatever DNS resolvers, NTP servers and whatever else could be useful for weather-induced disaster-recovery and/or offload cables.
Do you think most CDNs, stream services and CSPs could accommodate a scenario where we'd host their gear or provide VMs for them to announce on the local route-servers ? If not, what could be a reasonable technical arrangement ?
Thanks !
a message of 36 lines which said:
- Low redundancy of old cables (2)
- Total service loss when both cables are down because of congestion on
satelite backups
A problem which is not often mentioned is that most (all?) "local
caches" (CDN, DNS resolvers, etc) do not have an "offline mode" (or
"disconnected-from-master mode"). During an outage, they continue to
work for some time then break suddenly, in a not-friendly way, serving
various error messages instead of old data and/or useful
messages. (For instance, the DNS resolver may not be able to serve
stale answers.)
The time during which they can continue to work when they are
disconnected from their master is typically undocumented (except for
the DNS), and discovered only when there is a long outage.
Making the Internet work better with sometimes-broken connectivity is
still an area of research.
Many CDNs have hardware options for self hosted caches. I think there would likely be concerns about <20G of connectivity to those caches though. With an incorrect setup, you could end up maxing out those links just with cache fill traffic itself.
Hello Mehmet,
VMs are no go for big content companies except Microsoft. You can run Microsoft CDN on VM but rest of the content will need to be cached. You can actually install this yourself
I've already read most docs for caching servers provided by major actors. What I'm mostly concerned about is their ability to peer with multiple AS on the local IXP, as to not over-replicate them.
Should I establish a dedicated network peering on the IXP ? Or will they come with their own ASNs ? The peering case is quite not documented on publicly available specs, if even possible.
Depending on how much traffic do you have , you may be able to get facebook, youtube, netflix caches i think minimum bw requirement changes per region
Those I'm nearly sure I could get, if I can pool caches amongst ISPs. The current constraints are issues to any content provider, not just for local ISPs.
Best regards,
Hi Gael,
Friends from PCH (www.pch.net <http://www.pch.net>) operate backend services for DNS authoritative ccTLDs and the Quad9 DNS resolver. They would be very happy to help.
I'm sure they would, I'm a big fan of their work BTW. Though hosting them in a densely connected area isn't the same as it will in remote locations, I guess there could be some work to be done to get it running properly, as Stephane wrote.
How would you think we could work on that ? I mean, disconnected or extremely high latency scenarii should be on a research roadmap by SpaceX' standards, right ? 
Best regards,
Some will work directly on the IX via BGP. Others have to go behind a member of the IX.
Hi Tom,
Many CDNs have hardware options for self hosted caches. I think there would likely be concerns about <20G of connectivity to those caches though. With an incorrect setup, you could end up maxing out those links just with cache fill traffic itself.
In a case where these servers are on a dedicated network peering with the ISPs, I think it would be safe to throttle the sync feeds to not saturate actual uplinks.
At least, that we can do, but throttling uncached content to customers is forbidden (net neutrality).
Though Netflix is supposedly sending pre-loaded servers, and I think that - in this location - it's gonna mean a lot already. The quastion is : how would the servers peer with local ISPs.
Best regards,
Hosting authoritative and recursive dns servers at the IXP would drastically improve the experience of users most of the time.
Of course, Stephane considerations are correct and there’s no solution for when global connectivity is lost and responses will stop being sent.
two issues here: the smaller issue is that CDNs sometimes want their own routable IP address blocks, especially if they're connecting directly to the IXP, which usually means /24 in practice. It doesn't always happen, and sometimes the CDN is happy to use provider address space (i.e. IXP), or smaller address blocks. But it's something to note.
The bigger issue is: who pays the transit costs for the CDN's cache-fill requirements? CDNs typically won't pay for cache-fill for installations like this, and if one local ISP is pulling disproportionate quantities of data compared to other ISPs at the IXP, then this can cause problems unless there's an shared billing mechanism built in.
Nick
Hi Nick,
Thanks for your remarks. It's actually an ongoing discussion.
two issues here: the smaller issue is that CDNs sometimes want their own routable IP address blocks, especially if they're connecting directly to the IXP, which usually means /24 in practice. It doesn't always happen, and sometimes the CDN is happy to use provider address space (i.e. IXP), or smaller address blocks. But it's something to note.
I'd rather have CDN use some of their anycast /24 to peer with the IX, with a back-end connectivity for their control-plane and back-feeding.
The bigger issue is: who pays the transit costs for the CDN's cache-fill requirements? CDNs typically won't pay for cache-fill for installations like this, and if one local ISP is pulling disproportionate quantities of data compared to other ISPs at the IXP, then this can cause problems unless there's an shared billing mechanism built in.
We're willing to provide a dedicated LAN, with routed access, to fill caches and administer the machines. It would be fully dissociated from the IXP though, unless we could find a way to make it work and as to meet extra requirements upon redundancy.
Best regards,