COVID-19 vs. our Networks

With talk of there being an involuntary statewide (WA) and then national quarantines (house arrest) for multiple weeks, has anyone put thought into the impacts of this on your networks if/when this comes to fruition?

We're already pushing the limits with telecommuters / those that are WFH, but I can only imagine what things will look like with everyone stuck at home for any duration of time.

We’ve already had 1 building delay us access pushing us into an SLA breach due to COVID-19 fuckups. I mean “procedures".
-Ben.

No WFH policy here yet, but I've unpacked a couple of our ASAs from before our
most recent edge refresh to run AnyConnect for some of our remote sites. I'm
not expecting too much load being that we're not a particularly large company,
but it's better to have an extra couple RU filled up and happily whirring away
than to get a phone call at 5am when everyone in Eastern Time logs in and the
throughput drops like the Dow Jones.

I'm more worried about the lead times on new hardware skyrocketing than the
impact of having 8-10x the teleworkers. At least we can still fulfill orders
for software licenses...

Ben;

I am sure your SLA’s have force majeure clauses. I mean, they must, right?

We are all on a split VPN here and slowly moving to a no-VPN solution with some users already on that. They are finding things that aren’t quite covered right or properly, but that list is slowly shrinking.

I’m expecting that many places will be moving from a full VPN to a split solution. At my prior employer we did split VPN as well for v4 and full for V6 as not everyone had native v6, so split didn’t make sense there. (Those with native v6 did gripe, but it was better to have a consistent solution).

I’m expecting that despite the usual game and download/streaming events, the baseline usage during the daytime is going to tick up significantly eating into network margin. Hopefully everyone has your upgrades on order due to the aforementioned lead-time issues. Hopefully everything is back to normal in 4-6 weeks.

I’m looking forward to a few people learning how to WFH and expecting many people to realize how much they don’t get along as well when they’re in the house all day with the kids.

We have an internal thread going on the WFH tips:

#1 Take a break. That walk you would take to Starbucks or whatever, build something comparable into it at home.

There’s a few other pro-tips, but the take a break one is one I feel is important.

- Jared

I like the topic, but I think we should dispense with comments like ‘house arrest’.

Agreed.

The situation is already plenty serious as it is.

Let’s not add any more fuel to the fire.

…though, on a slightly related note, I’ve been seeing an increase in ads for “Packet Scrubbing Services” recently.

Has anyone told the sales folks that’s not how this spreads? :stuck_out_tongue_winking_eye:

nice.

So.. ~2 wks back I left a comment in a thread about: "using vpns is
dumb" (paraphrased) and that: "You should just move your auth from 'by
ip' to 'application' based'.

people scrambling for vpn capacity...see what i mean? :slight_smile:
if all of your apps are web-based and behind a load-balancer (or dns
load-balancing or...) less problems/different problems, eh? :slight_smile:

I still stand by the 'ugh, don't use vpns' though - if you can avoid
that I mean... which really, you can .. maybe not 'right now
because...' but :slight_smile:

I am on the university enterprise network side and on the state research and education network ISP-ish side. Our users are the ones that will no longer be using either network, and going to their home connections, so my focus has been dealing with "AHHH something is broken" and it being that the user never used wifi for work at home, or 5 or 6 users go "AHHHHH we just dropped our RDP sessions" during the last couple of days. After teaching users traceroute and how to google "what is my IPv4 address" they were on the same ISP, crossing a peering point that is historically congested, but is already getting worse the first day of the "trial" for important staff.

I am only going to be running traceroutes back and forth for like the next couple to few weeks or however long I am on house arrest. They closed our campus after spring break, which starts at 5pm tomorrow (as does my fun week of maintenances), currently for another week.

I have never been so fearful of an IX as I am today.

Brian Miller
Network Engineering and Architecture
Clemson University and the C-Light Network
AS2721, AS2722, AS12148

Just imagine all of those people streaming Netflix and playing COD all day instead of only a few hours at night.

Mike,

For those nets with a higher peak in the evenings, the graphs will flatten out. If you're struggling any given weekday evening, you'll be in trouble from the start. Major events and software releases are what will use up available buffers.

IMO the Disney+ surprise was a good thing. It forced networks to realize they needed more capacity. Disney+ streaming isn't what it was but if you've been adding capacity as a result of it, you're in better shape to weather the latest network surges.

-- Stephen

I don't know but we just issued travel restrictions to the United States
as it is now a Hot Spot for the unrestricted spread of the coronavirus
which causes COVID-19.

Hopefully they're more sensible restrictions than the US policy that prohibits
travel from most of Europe except the UK... but only for foreigners. If you're
a US citizen, you're still perfectly welcome to go to Italy and come home with
a few extra microbes to pass around a week after you return.

The word for anybody who designs a network firewall with that sort of logic is
"pwned". Just sayin'.

(Fortunately, I'm in a position to hide in my apartment and only emerge for
grocery shopping at 2AM until things wind down... Hope everybody else has a
good contingency plan)

I hear enough politics on social media and tv , please leave it off of this list.

I don't know but we just issued travel restrictions to the United
States as it is now a Hot Spot for the unrestricted spread of the
coronavirus which causes COVID-19.

Hopefully they're more sensible restrictions than the US policy that
prohibits travel from most of Europe except the UK... but only for
foreigners. If you're a US citizen, you're still perfectly welcome
to go to Italy and come home with a few extra microbes to pass around
a week after you return.

No idea what the policy for foreigners is, as that is a matter of
Federal jurisdiction. And our Prime Minister is currently in
"self-isolation" apparently.

The word for anybody who designs a network firewall with that sort of
logic is "pwned". Just sayin'.

These are Provincial policies. The Federal Government cannot prohibit
Canadian citizens from entering Canada but the Province is in charge of
matter of Health and Civil Rights, so as soon as they enter the Province
from outside Canada they are "requested" to self-isolate for 14-days.
This is for citizens. Don't know what the policy is for non-Canadians.

(Fortunately, I'm in a position to hide in my apartment and only

emerge

for grocery shopping at 2AM until things wind down... Hope everybody

else

has a good contingency plan)

Yeah, sounds like a plan.

First time posting, little anxious.

Currently under isolation here in Saskatchewan, it was a self isolation till met with a doctor who ordered it,
I doubt was at risk, was in Southern Italy as Northern Italy was breaking out.
Rather disappointed with the provinces "meh" maybe come in and get tested, "meh" maybe not.
As I've read there are no restrictions on incoming passengers, citizens or not. 
Quarantining is only done if you report you've been to the Hubei province.
Citizens and PRs have to be let in, period, part of the charter (section 6) 
although being let in whilst under quarantine and being let out into the public are two different things, both are legal.

Little surprised Canada doesn't have higher cases than we do.

Once concern I've been thinking about is hardware maintenance under lock downs and quarantines.
Do politicians allow people only out to repair? or will they allow organizations and
their employees to be out and do expansion to deal with the enviable surge in traffic?

What about in Italy where only pharmacies and groceries are open and entropy hits equipment?

Stay safe, isolate your subnets and yourself ;)

On this side of the world, no major policies yet to force staff to work
from home.

But we are currently facing plenty of power generation shortages in
South Africa specifically, and while offices have generators and such,
most homes do not. So the problem with working from home is some people
won't be able to until they either install alternative power sources or
national power generation normalizes.

Over the years, we've built a myriad of pfSense-based OpenVPN servers
across several countries and cities so staff can always connect to the
closest one when traveling or working from home. Traffic flow is
non-split, and because they've been upgraded almost every year, we don't
anticipate load issues if more people are out of the office.

We've reduced reliance on internal VoIP PABX systems for conference
calls since dumping WebEx and going Zoom nearly 2 years ago. Everyone
knows how to use it, and we use it rather frequently, so folk can still
have their meetings as usual.

Mark.

I generally work from home most days of the month anyway... get more
done this way :-). So the change for me isn't a huge one.

But it will be interesting to see how the rest of my colleagues evolve
with this.

Mark.

Oh they do, we just don’t like having to explain to our customers anything other than “we’ve fixed it before you called.” I hate downtime.

People will turn to you and every other ISP hoping you keep them online. So besides demand issues, keeping your network up will be important to a whole lot of people.

Rubens