Firewalls in service provider environments

Mailman List Mirror (Read Only)
1

All,

Looking for some recommendations on firewall placement in service provider
environments. I'm of the school of thought that in my SP network I do as
little firewalling/packet filtering as possible. As in none, leave that to
my end users or offer a "managed" firewall solution where if a customer
signs up for the extra service I put him in a VRF or VLAN that is "behind"
a firewall and manage that solution for them. Otherwise I don't prefer to
have a firewall inline in my service provider network for all customer
traffic to go through. I can accomplish filtering of known bad ports on my
edge routers either facing my customers or upstream providers.

What is the group's thought on this?

-Matt

2

I had a vendor actually suggest that that ALL my customer traffic should traverse a firewall. I asked why and they said "Ahhh it the internet, must have firewall". I suppose this must have been a great firewall.

So yes I would agree with you, firewall nothing for your customers unless they are paying you for a specific service. Filtering known bad ports, well, what's a known bad port? Bad for one person may be quite important for another. Whilst filtering port 25 outbound may help prevent some bots from emanating spam, it certainly does a lot to annoy other people.

3

I tend to agree with this, and I think you'll find that most providers agree with that as well.

There are several reasons for this:
1. Firewalls present another point of failure, and SPs are generally loath to force customer traffic* through another choke point.
2. Many firewall appliances are stateful. Multihomed customers and stateful firewalls can be a major headache. Asymmetric routing through stateful firewalls is pretty much a non-starter.
3. You (the customer) know your applications and internal network better than the SP does. It makes sense for you to manage your firewalls/NAT/
internal LAN. If you can't or don't want to do this, hire a consultant to do the work for you.
4. Most SPs would not want the liability of managing firewall service.
5. Dropping packets at the SP edge could be done, but I think most SPs would only want to do so in extraordinary circumstances.

* - As you mentioned, unless the SP offers, and those customers specifically pay for a firewalled service.

jms

4

From: Matthew Reath [mailto:matt@mattreath.com]
Sent: 07 February 2012 21:34
To: nanog@nanog.org
Subject: Firewalls in service provider environments

All,

Looking for some recommendations on firewall placement in service
provider
environments. I'm of the school of thought that in my SP network I do
as
little firewalling/packet filtering as possible. As in none,

I had a vendor actually suggest that that ALL my customer traffic should
traverse a firewall. I asked why and they said "Ahhh it the internet, must
have firewall". I suppose this must have been a great firewall.

So yes I would agree with you, firewall nothing for your customers unless
they are paying you for a specific service. Filtering known bad ports,
well, what's a known bad port? Bad for one person may be quite important
for another. Whilst filtering port 25 outbound may help prevent some bots
from emanating spam, it certainly does a lot to annoy other people.

--
Leigh Porter

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

From a filtering perspective there are some know worm ports and such that

we usually have a template created for.

Here is the template we typically use (or a variant of it):

<-- snippet -->
access-list 102 deny ip 10.0.0.0 0.255.255.255 any
access-list 102 deny ip 172.16.0.0 0.15.255.255 any
access-list 102 deny ip 192.168.0.0 0.0.255.255 any
access-list 102 deny ip 0.0.0.0 0.255.255.255 any
access-list 102 deny ip 127.0.0.0 0.255.255.255 any
access-list 102 deny ip 224.0.0.0 15.255.255.255 any
access-list 102 deny ip host 255.255.255.255 any
access-list 102 deny tcp any any eq 135
access-list 102 deny udp any any eq 135
access-list 102 deny udp any any eq netbios-ns
access-list 102 deny tcp any any eq 139
access-list 102 deny udp any any eq netbios-ss
access-list 102 deny tcp any any eq 445
access-list 102 deny tcp any any eq 593
access-list 102 deny tcp any any eq 4444
access-list 102 deny tcp any any eq 9996
access-list 102 deny tcp any any eq 5554
access-list 102 deny tcp any any eq 8888
access-list 102 deny tcp any any eq 7778
access-list 102 deny tcp any any eq 8594
access-list 102 deny tcp any any eq 8563
access-list 102 deny tcp any any eq 1434
<-- end snippet -->

This blocks some common worm ports as well as traffic sourced outside of
our network from reserved address space.

-Matt

5

One of my customers has a list like that. They can't understand why
one in every hundred or so TCP connections on port 443 fails.

Hint: you forgot "access-list 102 permit tcp any any established"
after "access-list 102 deny ip host 255.255.255.255 any". The
destination port in one direction is the source port in the other and
many of those are dynamic source ports picked by Windows. Unless you
restrict that filter to just packets attempting to initiate a new
connection, you're shooting yourself in the foot.

Regards,
Bill Herrin

6

Hi Matthew,

It Depends.

High end business customers (of the BGP speaking variety) generally
appreciate having a remote triggered black hole facility. That's a
kind of firewall. http://tools.ietf.org/html/rfc5635

Business customers in general shouldn't be filtered unless they buy a
managed firewall service from you. Don't tamper with their DNS either!

When you get down to the residential and Internet Cafe type users,
there is some common filtering you should consider:

TCP SYN to port 25 outbound from your dynamic IP customers should
generally be disallowed except to your local mail servers. 99 times
out of 100, connections originating to this port from dynamic IP
customers will be Email Spam from an infected PC. This will hurt you.
It will hurt you with spam complaints. It will hurt you with adverse
action by RBL providers. It will hurt you with damage to your
reputation and brand.

http://www.spamhaus.org/faq/answers.lasso?section=isp%20spam%20issues#133

Blocking TCP and UDP 137, 138, 139 and 445 is not terribly unusual.
These are associated with Microsoft file sharing protocols. Off the
LAN and outside the enterprise anybody actually open to this traffic
is generally asking to be hacked. Then a spam bot is installed and you
have another problem customer who isn't paying you enough to deal with
that crap.

Regards,
Bill Herrin

7

Here is the template we typically use (or a variant of it):

<-- snippet -->
access-list 102 deny ip 10.0.0.0 0.255.255.255 any
access-list 102 deny ip 172.16.0.0 0.15.255.255 any
access-list 102 deny ip 192.168.0.0 0.0.255.255 any
access-list 102 deny ip 0.0.0.0 0.255.255.255 any
access-list 102 deny ip 127.0.0.0 0.255.255.255 any
access-list 102 deny ip 224.0.0.0 15.255.255.255 any
access-list 102 deny ip host 255.255.255.255 any

I typically also include traffic to/from:

TCP/UDP port 0
169.254.0.0/16
192.0.2.0/24
198.51.100.0/24
203.0.113.0/24

Been wondering if I should also block 198.18.0.0/15 as well.

8

Yeah agreed. The only place this gets applied is inbound on the interface
facing an upstream provider. ACLs ingress from end customers are much
different. In theory this could cause issues with externally initiated
traffic that use lets say 8888 as its random source port.

-Matt

9

Yes, you should.

10

If you apply the ACL you showed as an inbound ACL on your provider facing
interfaces, you will be breaking any connections that exit your network
with source ports from your list of bad ports. For example, you connect
out from x.x.x.x:8888 to y.y.y.y:80, then the response packets coming back
into your network will be from y.y.y.y:80 to x.x.x.x:8888 and will be
dropped by your ACL.

This seems to be a common mistake, and is often missed because it manifests
as one-in-thousands failures of TCP connections. People tend to just try a
second time and it works and never investigate why they had one random
failure.

11

From: Matthew Reath [mailto:matt@mattreath.com]
Sent: 07 February 2012 21:34
To: nanog@nanog.org
Subject: Firewalls in service provider environments

All,

Looking for some recommendations on firewall placement in service
provider
environments. I'm of the school of thought that in my SP network I do
as
little firewalling/packet filtering as possible. As in none,

I had a vendor actually suggest that that ALL my customer traffic should traverse a firewall. I asked why and they said "Ahhh it the internet, must have firewall". I suppose this must have been a great firewall.

'of china'! ha! hahaha.... anyway.

So yes I would agree with you, firewall nothing for your customers unless they are paying you for a specific service. Filtering known bad ports, well, what's a known bad port? Bad for one person may be quite important for another. Whilst filtering port 25 outbound may help prevent some bots from emanating spam, it certainly does a lot to annoy other people.

I think for a purely SP network, transit-provider core links sort of
thing, why filter anything at all? why filter anything that's not
destined to your own equipment? You can't possibly know what some
customer (or set of customers) are going to do with their traffic, so
you can't possibly filter it sanely/safely.

for a consumer ISP, provided your TOS says it's ok, why not filter
some common problems:
  tcp/25
  ... not much else really... and REALLY you just want to send tcp/25
-> 587 on your mail-relays (or redirect to internal use addresses on
the relays).

If customers (in either case) want to pay you for 'security services'
then rock some filters on their CPE, with the option to move that
upstream to your PE if you have to (too much crap on customer link).

-chris

12

traffic to go through. I can accomplish filtering of known bad ports on my
edge routers either facing my customers or upstream providers.

I wouldn't want my provider breaking my connectivity in the name of
security. My thought on this is: by all means filter, firewall, and
mangle packets addressed From/To service provider equipment (for
example, port 22 TCP packets sent to the address of the ISP router), in
order to protect ISP network, But, with some minor exceptions, in your
role as ISP, internal firewalls should be separate from customer networks,
and, never ever filter, firewall, or mangle packets _forwarded_ by service
provider equipment To/From customer networks.

If you manage the downstream network, the customer has requested special
filtering or restrictions, or a pattern of abuse has been detected from a
certain downstream that can be temporarily mitigated with a filter, that's
a different story.

It is acceptable to drop packets to enforce network policy, such as no-spam
rules that a customer or host has violated, for example: all IP packets
to a host. It is acceptable to drop erroneous packets, such as ones with
an incorrect checksum or source IP that the customer has not been
assigned, or has not informed the provider that they were assigned.

I would say it's in general unacceptable for an ISP to discriminate against
packets addressed to or sourced from specific port numbers.

That's actually breaking connectivity.

There's no such thing as a "bad port number"; there are port numbers that
certain applications have abused; the entire port range should be
available for any host as indicated by the various RFCs that define the
protocol.

If packets with any valid bit pattern in the source port or destination
port field are dropped, based on a valid bit pattern in that field, then
that is really a breakage of proper connectivity by the provider.

What is the group's thought on this?

13

While I 100% agree that sticking a stateful firewall into a SP
environment is several kinds of dumb, I wouldn't run it wide open and
unfiltered either.

There are several things that a SP should definitely be looking at,
that'd still describe as a firewall, and are not the "stateful
firewall / IDS / IPS magic black box" half the posters in this thread
are instinctively reacting to. For the record, yes, I agree those
are a bad idea.

But how about these - All these are going to be implemented to a
greater or a lesser degree, and in different places, depending on how
you define SP (selling only transit OC-48s? T1..T3 to end user
corporations? Datacenter hosting?)

1. S/RTBH

2. Netflow based devices (Arbor, Tivoli TNPFA flow analyzers, etc)

3. DDoS mitigation - possibly resold as an extra service [built
inhouse / provided by other vendors or your upstream tier 1]

4. Router ACLs to get rid of common worm traffic

5. Filtering both ways to prevent async routing to bypass your filters
(http://irbs.net/internet/nanog/0408/0405.html and in that thread,
http://irbs.net/internet/nanog/0408/0465.html for a fun example)

6. Putting different customers into different VLANs rather than
packing everybody into a single VLAN - that way they don't spoof
unused IPs on the same VLAN (that is, unused IPs anywhere in your IP
space .. and this is, like #5, a rather old attack that I haven't seen
in a while, it used to be very popular with spammers some years back,
and sticking your customers into separate VLANs anyway makes a lot of
sense from a management perspective, leave alone the security
implications)

--srs

14

suresh@frodo 17:46:08 :~$ nslookup 1.113.0.203.bogons.cymru.com
Server: 127.0.0.1
Address: 127.0.0.1#53

Non-authoritative answer:
Name: 1.113.0.203.bogons.cymru.com
Address: 127.0.0.2

Also available as a bgp feed, for years now. Saves you updating your
martian ACLs from time to time.

15

Amen. v4 and v6 lists are available via free BGP feed (via v4 and v6 peering) from Cymru. Dynamic simplicity within community's finest standards.

Works wonders for those who have s/RTBH deployed.

16

Good point. Adding in an established entry, although may open you up for
TCP/SYN sort of packets is a better trade off than affecting customer
traffic.

-Matt

17

'established' is explicitly NOT 'syn' ...
maybe you meant 'ack flood' ? (or rst flood? or .... but certainly not
syn flood)

18

If I had an 'established' entry on an inbound ACL to filter traffic coming
be allowed to pass the ACL because of this?

But I see your point a connection initiation from external sources with
just the SYN flag set would not be allowed. However if a session is
initiated internally the returning SYN ACK from the external server would
be allowed as would ACK and data packets with ACK set.

19

I've always thought that reflexive access lists were quite elegant,
and a much better method than established, albeit for edge networks.

Do they not work in the SP space?

20

I'm an end user but I refer to these from time to time:

http://www.ietf.org/rfc/rfc3013.txt
http://www.ietf.org/rfc/rfc3871.txt

I suppose the salient question is what kind of customers are we talking about.

Best wishes.