Its already happening in Italy, and now that schools are shutting down here as well, its going to get interesting:
The ultimate traffic test is coming, looking forward to hearing about it on this thread.
Maybe its a good time to start a communication channel between content providers/gaming companies and ISPs/CDNs.
I think under circumstances like this, I could definitely see some of the online based games shutting services down.
Next you’ll have us actually reading books!
I think under circumstances like this, I could definitely see some of the online based games shutting services down.
How so?
Signed,
Someone who works for an online gaming company and has heard nothing of this.
Hi,
I don’t know where y’all live, but here in the SF Bay Area, pretty much all public and private schools have closed down. My school district (in Santa Clara County) will be closed until Spring Break.
The impact of all these bored school kids on the networks due to gaming might cause some issues. I know that if I’m working from home and my videoconferencing slows down because of someones gaming, I’m taking the necessary action (read, change some rules on my firewall).
Playing games doesn’t take much bandwidth. Downloading games does. So as long as everyone already has their games and there’s no updates, playing the game is typically under 100 kbps which is negligible compared to streaming video which takes 1 to 25 mbps.
My kid has enough homework to reduce her gaming to normal levels.
If your kid doesn’t, perhaps you’ll want to supplement it. 
Owen
You don’t have kids, do you…
They have the attention span of Koi these days. They’ll play most games for about 15 minutes or so before downloading the next one. (At least that’s been my observation of behavior among my GF’s daughter and her friends).
Owen
The impact of all these bored school kids on the networks due to gaming might cause some issues. I know that if I’m working from home and my videoconferencing slows down because of someones gaming, I’m taking the necessary action (read, change some rules on my firewall).
People are welcome to do whatever they want on their own networks. I just didn’t get the suggestion that online gaming services would shut down. Or were you saying, Mike, that online gaming would crowd out other services and so “shut down” those other services?
My experience at $job[$now] (IXP) and $job[-1] (ISP with residential users) show otherwise. ISP-side traffic comes inbound from ASNs hosting gaming platforms, and IXP-side, gaming platforms have no issues taking 100G ports and pushing lots of traffic on them. Ratio-wise, they seem very much "heavy outbound". When new games are released, we see extra traffic from CDNs. Even if a game does not generate much traffic, in a MMO context every user pushes one data stream but receives several ones. And there may be reasons (avoiding cheats) where traffic pushed from the gaming platform contains more then each user's actions.
IMO, it depends on how game handles inter-player communication. I do recall playing some serverless networked games some 15-20 years ago, with 3 players each on their own ADSL or cable, and the upstream (in the 512-800 Kbps range) never getting saturated.
Somewhat of a duplicate reply here to another thread…
We have noticed as the organization has been sending various teams to WFH, an increase in bandwidth to our various VPN services. It’s been creeping up daily.
we are in process of upgrading our bandwidth to these areas to support this.
Basically that. It’s probably more streaming services that could crowd out what would be considered “mission critical” infrastructure. Maybe the Netflixs and Hulus of the world will limit 4K streaming or something along those lines. Basically cap resolution to 720p for the time being.
Seems arbitrary. Lots of networks have lots of Netflix/etc capacity. Who determines what is “mission critical”? Our mission as an ISP is to deliver Internet to our customers. If they want to play online games or watch video, who am I to say that isn’t critical to THEIR mission?
The last thing we need are a bunch of kids in quarantine that have NOTHING to do because Mike Bolitho thinks their entertainment isn’t part of the “mission” of the Internet.
About the only thing that might be useful is something to smooth out the big jumps in utilization on game releases - but even that is something that can be managed by adding capacity.
To quote Jay Leno - Crunch All You Want, We’ll Make More.
Seems arbitrary. Lots of networks have lots of Netflix/etc capacity. Who determines what is “mission critical”? Our mission as an ISP is to deliver Internet to our customers. If they want to play online games or watch video, who am I to say that isn’t critical to THEIR mission?..
…The last thing we need are a bunch of kids in quarantine that have NOTHING to do because Mike Bolitho thinks their entertainment isn’t part of the “mission” of the Internet.
We already have that. It’s called Telecommunications Service Priority and this is the charge:
Telecommunications Service Priority (TSP) is a program that authorizes national security and emergency preparedness (NS/EP) organizations to receive priority treatment for vital voice and data circuits or other telecommunications services.
I work for a hospital, we ran into some issues last week due to congestion that was totally outside of our control that was off of our WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission critical at the time but it was still disruptive. As more and more people are driven home during this time, more and more people will be using bandwidth intensive streaming and online gaming products. If more and more TSP coded entities are running into issues, ISPs, IXPs, and CDNs will be forced to act.
For more information:
https://www.cisa.gov/telecommunications-service-priority-tsp
https://www.fcc.gov/general/telecommunications-service-priority
https://en.wikipedia.org/wiki/Telecommunications_Service_Priority
The Internet is not a telecommunications service, according to your FCC. If you want predictability, buy WAN circuits, not Internet circuits. If your provider is co-mingling Internet and WAN traffic (i.e. circuits with defined endpoints vs. public Internet or VPN), then you need to talk to them about their prioritization.
If you have mission critical applications, put them on mission critical infrastructure, not the public Internet.
Oh, that’s right - Internet circuits are cheaper than WAN circuits.
First of all, we use a mixture of layer 2/3 private lines and DIA circuits. You don’t know our infrastructure, stop being condescending. It goes against the spirit of this mailing list.
Second, yes, the Internet is protected. Both public and private lines. I know this because we have TSP coded circuits and I spent four years at a Tier I ISP servicing TSP coded circuits
Third, the trouble we had was a third party service having congestion issues. They are hosted by the same CDN as Call of Duty. The problem was both outside of our control and our third party service’s control. The chokepoint was between ISPs/IXPs and the CDN. I’ve seen this time and time again while working at the aforementioned ISP. Saturated links on ISP/IXP/CDN networks. This is where the TSP code comes in. In this day and age of cloud services, it is financially unfeasible for every company to have a private line to every single cloud provider. That’s preposterous to even suggest.
The Internet is not a telecommunications service, according to your FCC. If you want predictability, buy WAN circuits, not Internet circuits. If your provider is co-mingling Internet and WAN traffic (i.e. circuits with defined endpoints vs. public Internet or VPN), then you need to talk to them about their prioritization.
If you have mission critical applications, put them on mission critical infrastructure, not the public Internet.
Oh, that’s right - Internet circuits are cheaper than WAN circuits
Clayton Zekelman
Managed Network Systems Inc. (MNSi)
3363 Tecumseh Rd. E
Windsor, Ontario
N8W 1H4
tel. 519-985-8410
fax. 519-985-8409
This is a tiny sample of what's coming. We're all about to be tested
in a major way, and lots of latent problems are about to become real,
pressing problems. So:
1. Get some rest. Stock up (judiciously, don't hoard) on supplies
including medications, fluids, food, etc.
2. Find all the phone chargers, laptop chargers, USB sticks, cables,
everything. If you're not already obsessive about keeping things
charged, get that way.
3. Make sure your role addresses are up-to-date and working:
postmaster@
webmaster@
security@
abuse@
noc@
and whatever else is appropriate. Make sure that eyeballs are watching
everything that comes in there and anticipate that some people -- under
stress and anxious -- will send things to the wrong place.
Same for your phone contacts. And make sure frontline support personnel
have the ability and judgment to rapidly escalate, do not allow urgent
needs to get lost in some ticketing system.
4. Make sure your WHOIS contacts on networks and domains are up-to-date
and working. Same for your phone contacts.
5. Identify any spare resources that you can lend out. Identify any
resources that you can guess will be needed.
6. Everyone who can telecommute should be telecommuting right now.
If you need hands on-site, and of course lots of people will, keep
those people separated from others. Make sure hands-on people know
how to sanitize equipment, tools, etc.
7. Find time in the midst of this for self-care. You can't help
anybody if you're exhausted. Take a shower, watch dog videos, do
whatever you need to in order to stay functional.
Here's a resource page that I threw together with a little help
from some epidemiologists. It's short, plain HTML so it should
load very fast, and of course because it's short it's probably missing
things. Send suggestions to me off-list.
http://www.firemountain.net/covid19.html
---rsk
The access facility and the underlying long haul are telecommunications services. The application provided using that facility may or may not be. The congestion you were experiencing was not with the telecommunications facility itself, but with the application running on it, and was as you state, outside of your network - on a CDN hosted service. Your argument is with your third party hosted service. Their argument is with their CDN.
Internet exchange points are not regulated. Interconnections between ISPs and CDNs are private agreements, and are always at risk of congestion and commercial dispute between the parties. There is a long history of this.
If you have a direct layer 2 or 3 private line to your hosted service provider’s CDN, and it was not performing as per the SLA, then you need to take that up with them.
If the underlying telecommunications facility failed, and was classified as critical infrastructure, and not restored in a timely manner, then you need to take that up with the provider of that infrastructure.
I’m not trying to be difficult, but the fact remains that there is a distinction between telecommunications services, and Internet services. The fact that Internet services (and I’m not talking about any one particular DIA circuit, but the rather the global network of networks) work so well most of the time, such that people tend to start treating it as a substitute for telecommunications services is pretty impressive.
There are cost/benefit tradeoffs for using cloud hosted services and public Internet infrastructure. You save money by not operating the data centre yourself, but you pay for it in reliability. Your organization may have made that choice, but to say that because you chose to put critical applications on Internet infrastructure, other users of the Internet should take a back seat to your needs seems to be a bit of a stretch. Again, if your provider sold this to you as something that was NOT relying on public Internet, and was Layer 2/3 private managed services with dedicated bandwidth, then you need to have a conversation with them.
I don’t want to get in a fight, but absolutely:
Folks saw congestion from a massive free content drop this past week.
But as folks had called out, that was the CDN angle of distributing that content rather than the actual game play. There is a rather long discussion about that in the “akamai yesterday - what in the world was that” thread from Jan/Feb/March.
I don’t want to trivialize the challenges people may have from knock-on effects of upstream providers facing congestion or CDNs that have co-located content on nodes serving those bits with other bits, but did want to carve out regular online gameplay from content distribution. Whether people end up adjusting plans around large content distribution at this time, I guess remains to be seen.
Thanks Rich.
Good, clear, reasonable steps to have on hand.