I am looking for comments and suggestions regarding the merits of
purpose-built, appliance style firewalls (like a netscreen or Cisco PIX)
vs. running ipfw on a commodity server running FreeBSD. I am interested
only in packet filtering and rate limiting performance - NOT in VPNs or
IPsec/crypto considerations.
Hello all.
--- Where we are ---
Currently, I've got a single site, colocated on the East Coast. Currently, I've got two NetApps at that site, one serving as a mailspool, the other serving as a location for web documents. This system works via NFS to a fair number of mail and web servers, and it's running happily.
--- Where I'm going ---
What I seek is some help on implementing a second site, and the link between the two. The sites will be more or less the same in terms of the equipment in them, or so I hope.
I want to be able to have the changes made at one site replicated to the other site transparently. That is, if I update a file at site A, I want to be able to see the changes at Site B in a reasonable period of time (i.e., short), and without having to manually move data around. I specifically want to do this for allow both sites to offer the same mailspool, so that customers can check their mail at either site.
I am in the planning phase of bringing up a second site, and at this
site, there will be more web servers, and more mail servers. There will
also be an additional netapp for each of mail and www.
Between the pair of mail netapps (and to a lesser degree, the www
netapps), I want them to replicate changes to the other one. That is,
if a file is removed on Mail.NetApp A, it should also disappear on
Mail.NetApp B. And if a file is created on netapp B, it should also
come into existance on netapp A. Bidirectional updates.
My current setup consists of Linux and FreeBSD systems, and F740 NetApps.
And yes, there is a lot of pressure to stay with the NetApps.
Any hints, or advice will be much appreciated.
Thanks in advance,
Gabriel
I have run a EFnet irc server with FreeBSD+ipfw on the irc server itself.
Very few rules (like TCP syn ratelimiting, ICMP rate limiting, allow irc
ports, allow ssh port, drop the rest) and that crummy old machine was able
to handle a full 100megabit of spoofed SYN flooding.
I am not 100% up to speed as to what people are using on EFnet/IRCnet
nowadays but I am under the impression that they're still using the above,
ie letting the host protect itself. Sometimes they put a capable router in
front of it and let it do some of the limiting.
Back then, it wasn't the host that was getting hit worst by the flooding,
it was when the spoofed TCP SYNs were replied to by the machine, the
upstream Catalyst 5500 with RSMs totally choked on trying to route lookup
10kpps of diverse destinations, of which some were not even in it's full
routing table. The above TCP rate limiting etc (make the machine not
respond to a lot of pps generated by unverified connections) did a lot of
good in leveraging the upstream route lookup problem.
After implementing the above I survived several large floods without much
trouble and things were great for 3 months. After that the kiddies figured
out that they could attack other hosts on the same network or adjacent
networks and cause the RSMs to fall over and die and thus achieving their
goals anyway.
I have no specific suggestions to you in your specific case unfortunately,
my experience with FreeBSD+ipfw is limited to the above, but I thought it
might give you some insight into some of the problems I faced anyway.
You probally want something like snapmirror, but when a file is
changed/deleted in the mailbox, you hack your pop/imap server to delete
the file on the 'master' fileserver.
this isn't really on-topic though, imho.
- jared
There is really no benefit to purchasing a vendor-built firewall
when the real problem is protecting the servers' tcp/ip stacks and
the applications above them, as well as all the infrastruture in
between (routers, switches, whatever).
Do yourself a favor and spend half as much as you would on firewalls
and invest in a packet capture infrastructure to identify exactly
what types of attacks you are getting.
I believe the beta version of ipfilter allows you to specify bpf
logic to block packets. So just configure up each *BSD host with
bpf-enabled ipf filters that block the traffic you earlier identified
with your packet capture infrastructure (and if you are using
libpcap based tools, you are probably already using bpf to match
on packets).
For legitimate attacks, I suggest buying more bandwidth and scaling
your infrastructure appropriately. It also helps to report your
findings to others, especially the network and security communities,
the places of attack origin (even when spread out), and the transit
networks involved in passing along the attacks (especially your
upstreams).
It's also considered nice to block outgoing packets which match
the attacks you've seen, even if you believe your infrastructure
to be impenetrable.
However, if done right or wrong, any vendor-based or commidity *BSD
solution can be less or more powerful than any other solution.
dre
On Thu, Jan 16, 2003 at 03:17:44PM -0800, Josh Brooks mooed:
Currently, I run a FreeBSD firewall running ipfw (500 mhz celeron, 256
mags ram). This machine does nothing - runs no services but ssh, and
simply sits at my network border doing packet filtering. I have a lot of
hosts (four /24s - about 500 active IPs) behind this firewall, andThe problem I am running into is simply that my firewall CPU chokes. It
is not because the traffic is high - the line does not become saturdated,
and sometimes total traffic can be less than 5 megabits/s - BUT the
packets/s count goes way up (sometimes by a factor of 50) and because all
a) Shorten your rules. 
b) Have you tried ipfw2, or upgraded to 5.0-DR3?
(ipfw2 has some known bugs in 4.7-release, but I think it's
happy in stable. test, though)
c) Have you tried using polling mode for your ethernet device drivers?
(options DEVICE_POLLLING, options HZ=1000)
Can improve forwarding performance under heavy load/small packets,
e.g. a DoS attack
So my questions are as follows:
1. Am I wasting my time trying to make my FreeBSD+ipfw firewall more
resilient and sophisticated ? Again, I have probably only scratched the
surface, but let's say I emerge from my office 12 months from now having
memorized the ipfw source code and having learned _everything_ there is to
learn about this problem - will I simply conclude that FreeBSD+ipfw is not
good enough and I just need to go get an appliance ?
Not for 12Kpps. For some really sick rate, you might have to
go with an (expensive!) appliance. But for what you're seeing, it should
be quite feasible to handle with a host.
Other questions to check on: What ethernet device are you using?
If it's not de or fxp, you're shooting yourself in the foot.
-Dave
I am looking for comments and suggestions regarding the merits of
purpose-built, appliance style firewalls (like a netscreen or Cisco PIX)
vs. running ipfw on a commodity server running FreeBSD. I am interested
only in packet filtering and rate limiting performance - NOT in VPNs or
IPsec/crypto considerations.
[snip]
The problem I am running into is simply that my firewall CPU chokes. It
is not because the traffic is high - the line does not become saturdated,
and sometimes total traffic can be less than 5 megabits/s - BUT the
packets/s count goes way up (sometimes by a factor of 50) and because all
of these packets have to go through my entire ruleset, the firewalls CPU
chokes. It does not crash, it simply stops forwarding any traffic,
effectively blackholing my entire network. As soon as the attack is
stopped, the firewall is fine.
You may want to look into OpenBSD's new packet filter, pf(4). It's a stateful
filter, which, according to pf.conf(8), is usually faster than a rule-based
filter:
Also, looking up states is usually faster than evaluating rules. If one
has 50 rules, all of them are evaluated sequentially in O(n). Even with
50000 states, only 16 comparisons are needed to match a state, since
states are stored in a binary search tree that allows searches in O(log2
n).
Also, pf in -current now has filtering, NAT, queueing (rule-based bandwidth
control - QoS, formerly a separate altq utility), table definitions,
normalization (scrubbing), routing, macros and options to the filtering
engine. Word is that some of the new queueing features are very spiffy, but
that stuff is fairly bleeding edge, so you may want to test that or wait a few
months for 3.3 to come out. (Sufficient testing while it's in -current will
ensure it is stable enough for 3.3)
I'm not a developer, just a satisfied user. I have not personally used the
new pf code on anything more heavily trafficked than a home network, but
several of the developers use it on fairly high-traffic areas with good
results. Worth a look, at least.
2. I happen to like a host-based firewall (a firewall running on a normal
user OS like FreeBSD) better than an appliance. You get to do anything
you need with it, you have a full compliment of unix tools like grep and
awk and tcpdump and expect, etc. - it seems like you have more control.
Assuming (for a moment) that performance were equal, does anyone else feel
this way ? Does anyone else prefer a normal system for a firewall over,
say, a PIX ?
I'm with you on that, mainly for (a) flexibility of configuration, (b)
ease/speed of upgrades/patches, and (c) price involved in purchase and
maintenance. Also as you mentioned, a firewall that starts out just filtering
can later be modified easily to capture packets for analysis later, run
active or passive intrusion detection, etc.
3. I am not that high profile ... but what do the high profile (shell
servers like foonet and EFnet irc server operators) people use ? Would
any of those people consider even for a moment using a FreeBSD+ipfw system
for their packet filtering and rate shaping ?
Avleen Vig may be able to give an answer from involvement with the SAFE
project, or at least some interesting statistics ...
I just want to know if I should give up now and shell out a few grand for
an appliance, or if it is reasonable for me to attempt to protect a
network of my size.
When the traffic/attacks pass a certain point, my personal feeling is that
it's time to distribute the load, rather than look for a more expensive
single point of failure. Of course, this is not currently backed up by much
personal operational experience, so take that with a grain of salt.
Thank you very much.
cheers,
> 2. I happen to like a host-based firewall (a firewall running on a normal
> user OS like FreeBSD) better than an appliance. You get to do anything
> you need with it, you have a full compliment of unix tools like grep and
> awk and tcpdump and expect, etc. - it seems like you have more control.
> Assuming (for a moment) that performance were equal, does anyone else feel
> this way ? Does anyone else prefer a normal system for a firewall over,
> say, a PIX ?
I'm with you on that, mainly for (a) flexibility of configuration, (b)
ease/speed of upgrades/patches, and (c) price involved in purchase and
maintenance. Also as you mentioned, a firewall that starts out just filtering
can later be modified easily to capture packets for analysis later, run
active or passive intrusion detection, etc.
I agree on pretty much all the points there
> 3. I am not that high profile ... but what do the high profile (shell
> servers like foonet and EFnet irc server operators) people use ? Would
> any of those people consider even for a moment using a FreeBSD+ipfw system
> for their packet filtering and rate shaping ?
Avleen Vig may be able to give an answer from involvement with the SAFE
project, or at least some interesting statistics ...
Thanks! (unfortauntely SAFE has hit a little snag right now and we're
looking for some kind body to host our scans for us.. if anyone knows of
someone willing to do this, please let me know. It's very low bandwidth /
very low complaint generating).
My opinion on this is that IPFW sucks for packet filtering. IPFW2 is much
better - you can crunch hundreds of rules into just a handful but creating
groups of IP addresses and network block.
But I agree with Scott that a stateful packet filter like pf on OpenBSD or
ipf on FreeBSD is much better at this task.
Rate limiting using IPFW during a DoS/DDoS attack is nice if you don't
want your router to get overwhelmed trying to route huge numbers of
packets.
I can let the following advice:
On a FreeBSD router, with both IPF and IPFW compiled into the kernel,
packets are passed around like this:
INTERNET -> IPF -> IPFW+DUMMYNET -> Kernel -> IPF -> IPFW+DUMMYNET -> LAN
LAN -> IPF -> IPFW+DUMMYNET -> Kernel -> IPF -> IPFW+DUMMYNET -> INTERNET
This has the strong advantage of letting you filter off large numbers of
packets before doing your rate limiting.
The above combination works very well in my experience, during heavy DoS
attacks.
DRDoS on the other hand are more tricky.
but again, rate limiting to the destination can help with this.
With a stateful packet filter like pf/ipf, you can block out all packets
where the connection hasn't been established, and only allow in SYN's.
Then rate limit your SYN's to a very small number based on your needs.
I'm in total agreement as to the untily and significant
headache-reduction that a *bsd os (with real interactive editor
makes -- Vi for IOS must be too challenging). However, I do see a sore
spot.
One area that I've not seen much attention paid to (yet?) is
failover. Don't assume that I'm advocating the use of a PIX
here, but has anyone yet successfully used ipf/pf to export and
then import the state tables on a backup host? In my experience, doing
that w/ PIXen has been quite simple.
Forget all the ARP/ifconfig/heartbeat fudgery that'd be required to
acheive failover on *bsd with ipf/pf -- just finding a simple way to
move said state table from host to host seems interesting and
challenging.
How do we adress availability concerns while using comodity hardware and
Os's? Are they valid concerns, even? <G>
--Tk
I'm in total agreement as to the untily and significant
headache-reduction that a *bsd os (with real interactive editor
makes -- Vi for IOS must be too challenging). However, I do see a sore
spot.
One area that I've not seen much attention paid to (yet?) is
failover. Don't assume that I'm advocating the use of a PIX
here, but has anyone yet successfully used ipf/pf to export and
then import the state tables on a backup host? In my experience, doing
that w/ PIXen has been quite simple.
It'd be an interesting challenge to get this working with ipf/pf.
Forget all the ARP/ifconfig/heartbeat fudgery that'd be required to
acheive failover on *bsd with ipf/pf -- just finding a simple way to
move said state table from host to host seems interesting and
challenging.
ipf now has 'ipfs' which can dump and restore the current states table
You may want to look into OpenBSD's new packet filter, pf(4). It's a
stateful filter, which, according to pf.conf(8), is usually faster than
a rule-based filter:
...
But I agree with Scott that a stateful packet filter like pf on OpenBSD or
ipf on FreeBSD is much better at this task.
Don't confuse "stateful" firewalls with "compiled" firewalls.
Stateful just means you're maintaining state of established flows, which
is behaviorly different from a non-stateful filter.
Compiled is when you pre-process a normal ruleset and produce a matching
engine which is better suited to doing complex lookups. Some
implementations of this include Cisco's "turbo acl", Bill Fumerola's C
primitive generation from ipfw rules, Juniper's internal handling of all
firewalling, etc. People are trying anything, from adding a few binary
trees in your lookup to making a true compiler which produces packet
matching code.
As I understand OpenBSD's pf (which may not be complete so feel free to
point out if I'm wrong), it isn't actually doing anything to compile
normal packet lookups, it just added a non-sequential lookup engine for
the truely "stateful" filtering that it does. While this is nice and all,
it doesn't replace the functionality of normal rule-based filtering, and
it isn't the same as a true compiled filter. The closest comparison you
could make for the normal readers of this list is that it is the same as
speeding up acl matches by enabling the flow route-cache on a Cisco.
[snip]
As I understand OpenBSD's pf (which may not be complete so feel free to
point out if I'm wrong), it isn't actually doing anything to compile
normal packet lookups, it just added a non-sequential lookup engine for
the truely "stateful" filtering that it does. While this is nice and all,
it doesn't replace the functionality of normal rule-based filtering, and
From pf.conf(5):
For each packet processed by the packet filter, the filter rules are
evaluated in sequential order, from first to last. The last matching
rule decides what action is taken.
Does this not constitute rule-based filtering? Or am I misunderstanding you?
Yes and no. That would prove my point, if not for the fact that they are
describing the logical processing of a filter ruleset (aka "ipf-style"),
not the implementation of the matching engine.
But still, the stateful filtering and any lookup model it uses does not
negate the need for standard rule-based filtering, and AFAIK pf still
does those comparisons sequentially like any other traditional filter.
[Mail-Followup-To points to the pf list]
Tony Kapela wrote/schrieb/scripsit:
Forget all the ARP/ifconfig/heartbeat fudgery that'd be required to
acheive failover on *bsd with ipf/pf -- just finding a simple way to
move said state table from host to host seems interesting and
challenging.
OpenBSD's pf is moving there. -current now has the pfsync pseudo-
interface that exposes changes to the state table as they happen.
A daemon to make use of that for said purpose is expected after the
3.3 release.
'Rumor' says, a non patent-emcumbered vrrp-like mechanism will be
available as well.
-Stefan
Date: Sat, 18 Jan 2003 08:41:15 -0800 (PST)
From: Avleen Vig
But I agree with Scott that a stateful packet filter like pf
on OpenBSD or ipf on FreeBSD is much better at this task.
man ipfw
/-state <ENTER>
Eddy