NYTimes: Egypt Leaders Found ‘Off’ Switch for Internet

http://www.nytimes.com/2011/02/16/technology/16internet.html

There has been intense debate both inside and outside Egypt on whether the

http://www.nytimes.com/2011/02/16/technology/16internet.html

There has been intense debate both inside and outside Egypt on whether the

cutoff at 26 Ramses Street was accomplished by surgically tampering with the
software mechanism that defines how networks at the core of the Internet
communicate with one another, or by a blunt approach: simply cutting off the
power to the router computers that connect Egypt to the outside world.

I do remember some intense debate, here and elsewhere, but I somehow don't remember those as being the primary debate parameters.

Regards
Marshall

Interesting article though.

There are several good pieces to take-away from that article (not
really 'news', but still healthy with the occasional refresher):

"Individual Internet service providers were also called on the carpet
and ordered to shut down, as they are required to do by their
licensing agreements if the government so decrees."

"When he, too, noticed that domestic fiber-optic cables were open, he
had a moment of exhilaration, remembering that he could link up
servers directly and establish messaging using an older system called
Internet Relay Chat. But then it dawned on him that he had always
assumed he could download the necessary software via the Internet and
had saved no copy."

Operating local IRC networks is good, as is having local OS mirrors,
such as Debian/Ubuntu and let's not forget, having a resilient DNS
configuration (root zone copy hint 101: "dig @k.root-servers.net. .
axfr"). A securely distributed
"network-contingency-plan-autocrat-generic-all.deb" could be useful,
as well.

Cheers,
Martin

From: "Martin Millnert" <millnert@gmail.com>
To: "Marshall Eubanks" <tme@americafree.tv>
Cc: "North American Network Operators Group" <nanog@nanog.org>
Sent: Thursday, 17 February, 2011 8:28:22 AM
Subject: Re: NYTimes: Egypt Leaders Found ‘Off’ Switch for Internet
>
>

"

Operating local IRC networks is good, as is having local OS mirrors,
such as Debian/Ubuntu and let's not forget, having a resilient DNS
configuration (root zone copy hint 101: "dig @k.root-servers.net. .
axfr"). A securely distributed

Would it make sense for an ISP to "store" the root zone on their DNS servers instead of letting it be refreshed by the DNS cache? A cron job could refresh it from time to time. It would avoid entries from expiring and would always serve to clients entries with max ttl?

A root server would be better, but that could be an intermediary step?

Just speaking out loud here, so it may be total non-sense...

This is a subject of intense debate amongst the DNS literati:

CON:
1. Failure to pay attention to your setup could cause you to have a stale root zone.

PRO:
1. Faster local resolution for your users, especially for malformed queries.
2. No spurious traffic will be sent from your network to the roots
3. Greater resilience to any potential root server failure/DDoS

Personally I've been doing it for years, never had a problem. On larger sites where I have a lot of resolvers I make the hidden master a slave for the root zone, and also allow the local resolvers to slave it from the hidden master to be more net.friendly. For BIND, make sure you include "notify no;" in your zone{} statement.

hth,

Doug

I don't think that the Egyptian shutdown of domain names had much effect; that's why the bgp prefixes were withdrawn. What was effective was the withdrawal of BGP prefixes.

http://www.renesys.com/blog/2011/01/egypt-leaves-the-internet.shtml notes, for example, that routes *through* Egypt were operational, but routes through the same fiber and the same routers *to* Egypt were non-functional.

https://labs.ripe.net/Members/akvadrako/live_eqyptian_internet_incident_analysis pretty clearly states that "prefixes associated with Egyptian ISPs were withdrawn".

ditto.

i'm not aware of any actions by the .eg registry operator, though i'll ask, coincidental to the prefix withdrawal.

i suppose in the interests of completeness i should also ask about the (wicked recent) (مصر.) IDN ccTLD.

these are both wicked small zones, relative to the density of names registered (registries other than .eg) by egyptian residents, and the larger number of network using egyptians.

it is possible that as a preliminary step, the recursive resolver operators of each of the subsequently prefix withdrawing providers modified their cached data to provide policy-based resolution.

-e

It also seems like a question that could be decided empirically. Can
anyone on here comment on whether or not the BGP session ended
gracefully and the link lights remained lit?

--Richard

Never mind, Messrs. Cowie and Baker answered my question:
<http://mailman.nanog.org/pipermail/nanog/2011-February/033181.html>

Couldn't have paths through Egypt if layer 2 were cut off.

(Right?)

--Richard

Per the NYT article, the issue was the Egyptian "Intranet" -- people couldn't contact other sites within Egypt by host name, even though the routes were up, because they couldn't resolve .eg, .com, etc.

    --Steve Bellovin, http://www.cs.columbia.edu/~smb

i'll have to check if the .eg servers were ever taken off-line.

resolution of .com (beyond local caches) would have been pointless post-prefix withdrawal, but if the claim is that local ix routing remained possible, so non-cached .eg resolution was successful from outside of the eun to any other egyptian provider net, then if there is data to support the claim, it will be interesting. i'll ask.

-e

ah

This is interesting, in that according to http://www.root-servers.org Cairo has two root servers (F and J). The presence of a Verisign-operated J Root leads me to assume there are probably also local .com and .net servers. One of the three name servers for .EG looks like it could plausibly be in Cairo (IP address space registered to an Egyptian postal address, 100 ms response time from London). If DNS look-ups at that level didn't work, it seems likely that there was some disruption of internal connectivity as well.

Or, it may be that "the Internet" still mostly means foreign services. Being able to look up the addresses of Facebook's name servers isn't the same as being able to access Facebook. The Times article was a bit short of specifics on that, and I haven't seen other information on what it looked like internally.

There's something important to keep in mind in cases like this, though. Having redundancy and local copies of things is very good for protecting against accidental disruptions or disruptions of services in other jurisdictions. Protecting things that local guys with guns want to have go away is a somewhat different story. It seems likely that if "the Internet" had still been working after the things the government did to shut it down, the government would have done more. If somebody had managed to put all the pieces together and provide wide access to content the government wanted gone, they would probably have been told to stop. I'm a bit skeptical that having more local copies of things would have helped much.

-Steve

No the BGP and the physical links were down.

Mounir,

Yes all sessions were operational during that period.

I don't think that the Egyptian shutdown of domain names had much
effect

what shutdown of egyptian domain names?

randy, who has a server which serves them

The ASCII one .eg or the UTF8 one .xn--wgbh1c?

xn--wgbh1c. 172800 IN NS ns1.dotmasr.eg.
xn--wgbh1c. 172800 IN NS ns2.dotmasr.eg.
xn--wgbh1c. 172800 IN NS ns3.dotmasr.eg.

eg. 172800 IN NS ns5.univie.ac.at.
eg. 172800 IN NS rip.psg.com.
eg. 172800 IN NS frcu.eun.eg.

No the BGP and the physical links were down.

What about non-Internet layer 2 links? A number of companies have
private IP networks extending into Egypt providing MPLS or other VPN
services. In addition, there are often longlines into the Gulf states
to provide the Egyptian sites with redundancy. Were these
communications also cut?

One way to find out would be to talk to the networking folks at any
major international consumer brand that is in Egypt. I would expect
that nowadays if a Coke or a Pepsi is in a country, they will have
some kind of IP VPN crossing that country's borders.

--Michael Dillon

there's an interesting point to be made about the geographic
administrative and political distribution of secondaries being essential
to insuring their survivability.

Oddly your name is on bcp 16.