Hey there,
Hi Chris,
In our case, we have several hundred thousands of DSL customers today, and
the
million plus subscriber mark is on the engineering horizon. The problem
of
security threats & resulting incidents is going to get considerably worse
before it gets better. And that's for at least two reasons.. the ramp up
of
broadband and presumably the declining sophistication of the subscriber
population as a result of the greater market penetration.
Lack of security knowledge is also a huge problem in the collocation market.
The percentage of security-conscious colo customers is low, and uneducated
customers tend to be connected to Fast Ethernet or Gigabit Ethernet feeds.
One compromise in a well-connected colo can have the same effect as hundreds
of broadband compromises, although it's easier to track down and stop.
How do you effectively scale the massive support effort need for
collaborative
marketing of personal firewalls and the potential for false positives and
negatives? Any ideas on the legal exposure of security services?
My former employer ran several different managed firewall services, both for
broadband customers and colos, with some success ... all firewalls were
either built into CPE or designed as part of a "managed colo" environment
where large numbers of colo customers were front-ended by a few firewalls.
I don't see the broadband issue fixing itself without some built-in stateful
inspection firewall in the CPE itself -- if the customer has to pay for an
additional piece of hardware or software, it will instantly reduce
penetration. If you can do what you need from a firewalling standpoint on
the CPE, it makes life a lot easier. If you can provide a default firewall
installation on your choice of CPE, configuration scaling becomes much
easier.
As we were never taken to court on the legal issues of managing a firewall,
the only shot in the dark I would make there is that our standard contract
made no warranty for host security at all, and accepted no liability, even
though the 'wall was managed. It's impossible to manage network security,
but not host security, and make any guarantees.
Like, in the current case, several providers have resorted to blocking
port 80
to their non-DIA subscriber base. Is this really scalable? Obviously not
for
every threat. You can't effectively keep this up with the myriad of
threats.
Or can you?
<this should get the crowd rolling>
I'd think that a good default stance would be to block all incoming TCP
connections that aren't part of an established session, for all broadband
customers. Most of them would never notice, as email and http still work.
However, at the scale you're talking about, I don't see blocking anything on
the aggregation device itself ... it'd have to happen in the CPE, since
firewall rules are going to have to be customized for clients who do need to
run servers on their LAN.
So, I think it's clear that something needs to be done, but coming up with
a
definitive plan of attack is everything but trivial.
This obviously doesn't just apply to DSL, it applies to Cable and whatever
other broadband networks are out there or will evolve...
Instant results could be had, by:
- firewalling *all* broadband customers by default, with a strict ruleset
that denies incoming connections. Most of these customers don't know that
they're running open servers on what they consider desktop machines. NT and
Linux are easy to yell at, but anyone seen a default installation of Solaris
8 lately? OS vendors are not going to fix this any time soon, which leaves
greater responsibility on the shoulders of whoever connects the machines to
the 'net.
- requiring a firewall on all collocations, whether administered by you (for
money) or by the customer. Exceptions exist, i'm sure, but there aren't
that many of them as compared to the vast number of poorly secured
installations.
- if you're going to go this far, firewall all T1 customers too ... CPE
exists that will happily do this.
these three should take care of a lot of the "oops i left that port open"
security problems, and I'd be interested to know what percentage of major
infections could be cured by just putting up a firewall. Of course, it
requires a lot more staff to manage, so that cost is going to affect your
bottom line.
That doesn't, however, address something like code red exploiting a
legitimate web server. As I'm already in security nazi mode, how about:
- put it in your contract that you will run vulnerability scans against all
customer servers, with advance notice if necessary. Run them.
- put a required response time in your contract for a customer to fix any
vulnerabilities found. Shut them off if they don't patch the holes. (note
that if you are firewalling the customer, you don't have to unplug them,
just block access to the server until it's fixed)
- recognize your limits. try to stop the canned, automated exploits --
after the fact if necessary, but realize that a determined attacker would
require much greater effort to stop. That one may be best left to the
customer.
This is really just a small start, but if you're going to get this serious
about security, some amount of education is necessary ... even if it's just
so you can say "i told you so." Translate and bounce all CERT, security
focus, etc. exploit announcements to your customer base, making it clear
what is in danger and what must be done to fix it. Remind customers that
they may be shut down if they're compromised. Put links up to patches on
your website.
Run an abuse department that responds quickly to customers, and to other
providers, within limits. 24 x 7 is necessary, responding instantly to
black ice freaking out because someone ran nmap past it is not.
I've probably ranted enough already, but I don't think that the cost of
adequate security has *ever* been addressed at SPs. I may be wrong, but the
emphasis just doesn't seem to be there. In order to protect yourself from
your customer base, you have to incur significant costs ... how do you
compete with someone who doesn't care? How do you blackball those who don't
care, so you can compete with them? Is this something that requires general
agreement from the SP community, or is it possible for single companies to
make an effort without pricing themselves out of the market?
I've probably already stepped on enough toes this morning, so i'll drop it
... except to say it's nice to see someone trying to wrap their head around
these issues.
Cheers.
-travis
Cheers,
Chris--
Christian Kuhtz <ck@arch.bellsouth.net> -wk, <ck@gnu.org> -hm
Sr. Architect, Engineering & Architecture, BellSouth.net, Atlanta, GA,
U.S.