143.228.0.0/16 and house.gov

Mailman List Mirror (Read Only)
1

Hi folks, just musing...

From an ops perspective, wonder just how much traffic caused:

  "This morning, our engineers sounded the alarms ... and we have installed a digital version of a traffic cop. We enacted stopgaps that we planned for last night. We had hoped we didn't have to."
  --Jeff Ventura, communications director for the House's chief administrator. (from http://www.cnn.com/2008/POLITICS/09/30/congress.website/index.html)

Don't .govs have enough b/w or at least ability to add b/w in order to satisfy their 'public outreach/information' role? (not a rhetorical question...hehe)

It also seems to me that adding load balancing, firewall, throttling, etc methods for traffic shaping might actually make the problem worse by adding yet another layer(s) of hardware/software that may be prone to bottlenecking or overloading.

whaddayathink?

Ernie M. Rubi
Network Engineer
AMPATH/CIARA
Florida International Univ, Miami

2

Ernie Rubi wrote:

whaddayathink?

I think the house is run by the same folks that can't run a party-line vote.

I'm surprised they have electric power.

3

I'm surprised it isn't outsourced to some managed (hosting) provider, or a CDN.. Like Akamai or LLNW. It would surely be far more efficient for their purposes.

Also, if you've planned your network correctly QoS/Shaping will not negatively effect your network. You always engineer your outer edge to take a beating.

Sargun Dhillon
925.202.9485
deCarta
sdhillon@decarta.com
www.decarta.com

4

Ernie Rubi wrote:

From an ops perspective, wonder just how much traffic caused:

  "This morning, our engineers sounded the alarms ...

More of a case of a worldwide press conference broadcasted live by news
networms around the world when Nancy Pelosi stated that "as of now, the
recovery plan is available for everyone to read at the following web
address". The web site was immediatly overloaded and remained so for
hours. Normally, this web site would have received just a few visits
from the public at large every day. All of a sudden, millions of people
tried to get to it at the same time.

5

Some political action groups probably decided to step up the astroturfing.

You know, enter your email address here and we'll send out some
boilerplate nonsense to a bunch of congressmen and senators.

Block or firewall the worst of them, whether left or right leaning,
and I guess that should leave the servers clear for real users ...

--srs

6

Well, I do know that there's two ways you can contact your congressman -

* Feedback forms on individual websites
* Email

srs

7

Suresh Ramasubramanian wrote:

Give me a break. You're telling me the White House's mail servers are even
on the same network as their web servers? What, is this 1997?

Well, I do know that there's two ways you can contact your congressman -

* Feedback forms on individual websites
* Email

Chris:

The "House" is the House of Representatives; see also the "Senate", the
other branch of Congress.

The "White House" is the place the President of the Executive branch lives.
One of its wings has executive offices in it....

As it is obvious, the various servers are not on the same network, as they
are controlled by different branches (that don't trust each other).

Suresh:

AFAICT, the www.house.gov stuff is Akamaized.

At least in the days I was hanging around there, all the email was sent
via a centralized system for the House. Even the web forms were actually

Some years ago, I caused considerable consternation bypassing that system
with a VPN for my Member, as the system was controlled by Republicans, and
it was apparent that they were snooping on the Democrats.

I don't know how it's setup these days. Democrats have only been in
control since early 2007, and it has changed considerably. I'll note that
the house.gov SOA still lists mail.house.gov, but there's no A record.

The clerk and primary DNS systems seem to be 143.231.x.x/16, with diverse
paths. The secondary NS is on 143.228.x.x/16, so it seems to be
reasonably well done.

8

Not talking about network redundancy here .. but I have a feeling they
use what would effectively be a typical corporate MTA / groupware. Not
something that's ISP class and capable of truly heavy loads.

srs

9

What makes you thing that .gov's "have" anything at all? They have to
buy any bandwidth they have (other than strictly internal bandwidth)
from ISP's. If the IT budget doesn't allow for it, the IT department
can't buy it. If the projected need is much lower than this surge, then
they would not have budgeted for it. The USGOV, contrary to some
folks' belief, does not own the Internet.

Some ISP's are able to quickly add bandwidth if the line is set up for
it, but I think the IT department would have had to have an existing
active relationship with the ISP to be able to know whom to ask.

10

Are you saying that the house.gov site is not in a large data center
with direct fiber connectivity along with many of the other large
federal web sites (with alternative hot sites ready to go at a moment's
notice, of course)? As someone who has been to different government data
centers, I can tell you they have huge amounts of data connectivity
there in case of emergency.

For a large site like house.gov, bandwidth should never be an issue. In
this case I highly doubt it was the issue, but instead overloading of
the hardware in place.

Just my $.02...

Mick

11

Is this really technical discussion of operation of networks?

I connected the internal network of the US House of Representatives to the Internet when I worked there, and operated it through both Democratic and Republican control. I never saw any snooping by either party of the network traffic, and I had sniffers for diagnosing problems in several communication closets. I do recall unfounded accusations both ways, but it would be sad for the rumors to outlive the reality. The notorious case of intercepted cell-phone conversations had nothing to do with the data network.

Not only is the data center, but so are all the committee and member offices that want it connected.

Skilled professionals operate the House's network. There has been a collegial relationship among the operators of both the Senate and House networks, as well as the rest of the Legislative branch. There are good reasons, including Constitutional separation of powers, that the Legislative Branch is not managed by the Executive Branch. The independence of the two houses of Congress is more a matter of tradition, and the fact that a different party sometimes controls the other house.

Bandwidth has ALWAYS been an issue because Internet access is acquired through normal business processes, and the appetite for bandwidth both to Congressional staff, and (occasionally - when something important happens) to the public. Since the source of money for these operations is Federal taxes, many readers of this list might appreciate that we have not bought more than we could justify.

I will not say anything about how large or redundant the data center is for obvious reasons, beyond that I am no longer employed there and do not have the details.

I really think this thread has outlived its entertainment value.

John

12

John Schnizlein wrote:

I connected the internal network of the US House of Representatives to the Internet when I worked there, and operated it through both Democratic and Republican control.

Aha, I wondered who was to blame....

Of course, my Member was on the Internet before the House, as MERIT -- the
very same organization that ran/runs NANOG -- had its own POP (called an SCP
in those days) in DC. Only later did we use the House net.

She usually took her Mac laptop to Science and Education committee meetings.
Her staff was often asked how they got her to use her own laptop, when they
couldn't get their own members to read (or type) their own email.

This was all pre-2001, and Blackberry mania.

  I never saw any snooping by either party of the network traffic, and I had sniffers for diagnosing problems in several communication closets.

Yet, there was verified interception of both House and Senate email
communications. Nobody claimed it was "on the wire" network traffic, as
there were many weaknesses in the data network security design.

And the vicious fight about our setting up a VPN to bypass the centrally
controlled system -- as in "if you do this, we'll cut off your network
access entirely" -- led all concerned to guess that there was a political
reason, not a technical reason. So, I just used non-standard ports, and
some other firewalling, to prevent your staff from detecting it.

Also, there was the long fight about members running their own servers
(as in member.house.gov), instead of relying on the central servers for
connectivity (www.house.gov/member). Again, we really didn't trust the
Republicans not to examine internal data.

  I do recall unfounded accusations both ways, but it would be sad for the rumors to outlive the reality.

Like this verified and widely reported:

   "Democrats Suggest Inquiry Points to Wider Spying by G.O.P."
   http://query.nytimes.com/gst/fullpage.html?res=940DE4D7173AF933A25751C0A9629C8B63&sec=&spon=&pagewanted=print

The notorious case of intercepted cell-phone conversations had nothing to do with the data network.

True, but irrelevant.

I will not say anything about how large or redundant the data center is for obvious reasons, beyond that I am no longer employed there and do not have the details.

I've not even visited DC since 2002, and the old building with the page
dorm was torn down that summer.

But I can dig and traceroute. I'm pretty sure this isn't an ideal (or
standard conforming) setup. But it shouldn't have been swamped, as seems to
be akamaized.

13

This will be my last response on this despite whatever spin follows.

John Schnizlein wrote:

I connected the internal network of the US House of Representatives to the Internet when I worked there, and operated it through both Democratic and Republican control.

Aha, I wondered who was to blame....

Thank you for the compliment.

...

I never saw any snooping by either party of the network traffic, and I had sniffers for diagnosing problems in several communication closets.

Yet, there was verified interception of both House and Senate email
communications. Nobody claimed it was "on the wire" network traffic, as
there were many weaknesses in the data network security design.

If you know any, please send them to me privately. I can assure the community that our design and implementation got repeated review and testing from the best we could find at the time.

And the vicious fight about our setting up a VPN to bypass the centrally
controlled system -- as in "if you do this, we'll cut off your network
access entirely" -- led all concerned to guess that there was a political
reason, not a technical reason. So, I just used non-standard ports, and
some other firewalling, to prevent your staff from detecting it.

I hope no damage was produced by any inadvertent back doors opened by your VPN.

Since we were not blocking applications other than IRC, I don't know what you felt you needed to get around.

Also, there was the long fight about members running their own servers
(as in member.house.gov), instead of relying on the central servers for
connectivity (www.house.gov/member). Again, we really didn't trust the
Republicans not to examine internal data.

Although I do not recall the particular offices, I do recall that several committees and members had both email and web servers in their own offices with domains delegated to them on request. I have no idea what "long fight" you might have experienced.

I do recall unfounded accusations both ways, but it would be sad for the rumors to outlive the reality.

Like this verified and widely reported:

"Democrats Suggest Inquiry Points to Wider Spying by G.O.P."
http://query.nytimes.com/gst/fullpage.html?res=940DE4D7173AF933A25751C0A9629C8B63&sec=&spon=&pagewanted=print

As I recall this was simply a case of one staffer logging into a server in a different office. As you mentioned above, not "on the wire" and not a data network security issue. As sometimes still happens, the "computer network" actually referred to a file server. This article is about activities in the Senate, which operates independently of the House - was your experience actually with respect to the Senate?

John

14

William Allen Simpson wrote:

But I can dig and traceroute. I'm pretty sure this isn't an ideal (or
standard conforming) setup. But it shouldn't have been swamped, as seems to
be akamaized.

I don't have traceroutes kept, but during that night when Pelosi
announced the bill was available for all to download, I tried to get to
that page and it was extremely slow. Doing a traceroute didn't *seem* to
end at an akamai point. My memory could be in error.

Question:

Is it possible to setup an akamai feed in hours once you know your
website is to be swamped ?

Obviously, the system managers there might not have been warned in
advance that the politicians would place a huge load on their servers.
But once they realised it, is it conceivable that they quickly setup an
akamai feed ? Or is that something which takes weeks to setup ?

15

<snip>

Question:

Is it possible to setup an akamai feed in hours once you know your
website is to be swamped ?

Obviously, the system managers there might not have been warned in
advance that the politicians would place a huge load on their servers.
But once they realised it, is it conceivable that they quickly setup an
akamai feed ? Or is that something which takes weeks to setup ?

I'm not sure about Akamai, but I believe Amazon is about to roll out CDN
services as well (and I would assume they're as flexible as their other
"cloud" offerings). As always, hindsight is 20/20.

http://www.amazon.com/gp/html-forms-controller/aws-content-delivery-service

-brandon

16

Pretty much no matter who you use, this can easily be done in an hour
or so if people really want it to and the right techs are
available. If there's a pre-existing agreement, this can go to mere
minutes. The setup doesn't take long. it's usually the business stuff
that drags it out.