http://www.cert.org/advisories/CA-2002-03.html
-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben --
-- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
ASN.1 is pretty cool, but I've been wondering are there that
many ISPs which allow external SNMP access to their equipment?
SNMP is a UDP management protocol, and even under the best of
conditions, accepting packets from out of the blue isn't a good
idea.
I guess we will find out in the next few hours (while many
ISP network engineers are stuck in airports travelling back
home).
Spoofed packets?
It's not feasible to filter antispoof at OC-12 or OC-48 line rate on
all customer facing interfaces.
ericb
The *real* problem is that many *host* systems (Solaris, SGI, AIX,
etc) have SNMP enabled by default. And remember that hosts outnumber
routers by a considerable margin.
The SANS Top20 listed SNMP as a "turn it off". It was in the Top10 list
before that.
Can I stop banging my head against the wall yet?
I can remember many cases when my HP Openview network discovery process
would attempt to map the entire Internet because it strayed into a
peers network. So it may fairly common.
At least one provider has told me they don't use in-band management for
their network infrastructure. They have a completely seperate frame
network connecting to POP management LANs which in turn is connected to
seperate management ports on the equipment. I don't know how common this
is among large providers.
I had a smaller network, so I filtered the IP block used for my management
LAN from all external sources (and "logged" the ACL's so I picked up the
stray packets from places I missed). A "real" packet should never
be sourced from outside my network topology, so even if you spoofed the
IP address the topology would block it. Of course, this depended on
topological integrity. I can understand if larger providers why large
can't do that, it doesn't scale.
But there are a lot of small and medium providers that can do it.
I agree, its a glass house issue. I was just wondering how bad of an
issue it really is.
The PROTOS group also tested HTTP, WAP, and LDAP in addition to SNMP.
They also have a paper on constructive disclosure wherein they outline
the need to a black-box test suite for use by customers and vendors to
ensure simple buffer overflows, format string errors, etc. are easier
to find.
So, they are essentially attempting to empower the consumer and the
vendor to prevent these types of programming errors by releasing a
test suite to test for these things.
See this paper for more information:
http://www.ee.oulu.fi/research/ouspg/protos/sota/FIRST2001-disclosures/paper.pdf
But it should be not only feasible, but standard practice.
-ron
ASN.1 is pretty cool, but I've been wondering are there that
many ISPs which allow external SNMP access to their equipment?
SNMP is a UDP management protocol, and even under the best of
conditions, accepting packets from out of the blue isn't a good
idea.
Spoofed packets?
It's not feasible to filter antispoof at OC-12 or OC-48 line rate on
all customer facing interfaces.
But it should be not only feasible, but standard practice.
It's impossible using most high bandwidth gear that's out there. At
these speeds, you can either route the bits, or look at them, but not
both. Juniper is the one vendor that's given us packet inspection
abilities that scale with bandwidth. We have non-Juniper routers.
Please, tell your vendors you want line-rate filtering up to layer 4.
We're tired of being told "But you're the only ones that ask for
this".
Without control plane seperation (and it's not possible with Cisco,
Juniper, or most other routers out there), management services are
listening on the public network, and that makes this very scary,
regardless of filtering policies, etc.
ericb
'Should be' is the key word here... in practical terms though this is not
feasible. There are revisions of oc-12 and oc-48 cards in platforms that
don't support filtering.
Long term all users of internet routing hardware (or routing hardware in
general) should push their vendors to implement line-rate filtering. There
really is no reason NOT to do it is there? Even better would be the
ability to look inside the entire packet, this way the next code-red can
be stopped at a higher level in the network where people that actually
care about the problem can take appropriate action.
-Chris
C'mon guys. Exchange point rate anti-spoof filtering is not necessary to solve this problem.
This is why there are switches (using vlans if you choose) and router interfaces. Unless you are taking an OC3's worth of management traffic, you create a net just for your management traffic, put in on an interface and hang your entire site's snmp gear off of that. If you want it to be private, GRE and 1918 addresses are your friends, and filter to allow only traffic from those nets. None of this is new or hard.
Also, most everyone now supports snmpv3 security, so you can do that as well. (I just do it the old way I know how, so I haven't played much with this.)
jerry
Nice theory, but in practice it's a little ickier than you make it sound.
Consider most people on this list deal with networks (not just single
sites) spanning multiple states or countries. Not everyone can afford to
build both a backbone and a separate management WAN. Putting management
in 1918 space is ok at one location, but gets tricky on a large network.
Do we then also buy/maintain VPN hardware to connect all the various 1918
management networks to the NOC?
This actually might be an interesting use for MPLS VPN for those networks
where all the core gear supports it, but a totally separate management WAN
is cost prohibitive.
C'mon guys. Exchange point rate anti-spoof filtering is not necessary to
solve this problem.
How do you filter your peers to prevent them from spoofing your
infrastructure space? Not everyone filters their
custoemrs because either a) they have a large and varying set of
routes (and ip sources) they may send at you b) they can't manage
it or c) their routers can't filter (fast enough).
This is why there are switches (using vlans if you choose) and router
interfaces. Unless you are taking an OC3's worth of management traffic, you
create a net just for your management traffic, put in on an interface and
hang your entire site's snmp gear off of that. If you want it to be
private, GRE and 1918 addresses are your friends, and filter to allow only
traffic from those nets. None of this is new or hard.
No it is not but the problem is when extracing snmp data
(for billing for example) one can not always use an oob network
to extract this data or a vpn solution due to port-cost, etc..
IMHO router vendors should be able to do the various types
of filtering at line-rate (strict rpf, loose rpf, "any rpf",
rate-limit icmp, filter based on exact config to prevent DoS or track
such items).
Some vendors did not consider this key functionality when
they designed their routers/linecards.
Also, most everyone now supports snmpv3 security, so you can do that as
well. (I just do it the old way I know how, so I haven't played much with
this.)
Sure this works assuming all your pollers can support snmpv3
without any complicated problems and have resources to allocate to the
various projects that collect this data. I'm sure there are a few
companies these days that are having a harder time getting the money
and resources to perform non-critical upgrades to these systems when
the current one works just fine.
- Jared
BOTH linerate filtering and packet inspection should be part of the minimal
requirements to sell routing hardware. Hmm...so in case any vendor out there
hasn't heard this directly from us, consider this a clarification of our
requirements. And UUnet's...and ?? any other providers want to make sure
that the vendor community gets the message here?
-ron
interfaces {
lo0 {
unit 0 {
family inet {
filter {
input RE;
}
}
}
}
}
firewall {
filter RE {
term BGP {
from {
protocol tcp;
destination-port bgp;
}
then accept;
}
term TCP-established {
from {
protocol tcp;
tcp-established;
}
then accept;
}
/* insert other term's allowing routing protocol traffic etc. */
term only-fxp0 {
from {
interface-group-except fxp0;
}
then discard;
}
/* allow ssh, snmp etc. traffin only on the mngt. lan */
term allow-from-fxp0 {
from {
interface-group fxp0;
}
then accept;
}
}
}
/Jesper
Without control plane seperation (and it's not possible with Cisco,
Juniper, or most other routers out there), management services are
listening on the public network, and that makes this very scary,
regardless of filtering policies, etc.
interfaces {
...
}
firewall {
...
}
OK, but that's filtering. The telnet/ssh/snmp daemon is still
listening on all interfaces. You can't get there, as long as your
filter stands, but those are some hard filters to write. They're
simple when they're simple, but they're very complex when they're not.
You're relying on your filters, rather than on proper configuration of
the daemon. On a UNIX system, you can bind a service to all
interfaces (e.g. *.161) or just to a specific interface
(10.1.2.3:161). This should be possible in general, on all routers.
We HAVE an OOB management network. This is where all our console
servers, switches (there is no Ethernet in the backbone, don't shove
VLANs at me), etc all live. This address space is not routed to you.
We like this. There's no cost issues, we've already paid for it, and
need it for our layer 1/2 network anyway.
But then you plug an IP port on the router (vs. a console port) into
the mgmt net, and you've bridged the public net and the mgmt net.
Virtual routers are capable of maintaining multiple routing tables,
but last I checked, Juniper was not. So how do you route this?
I send an SNMP query to the device. It comes in over the mgmt net
(because for me, in my datacenter, the loopback for that device (or
it's mgmt IP) is routed across the mgmt net). The device recieves,
digests, and decides to respond to this query. Where does it send it?
My datacenter is routed on the internet, so does it send it out the
public interface? Or do I route my datacenter over the mgmt net? You
can start filtering, but then those filters are suddenly very
important, crucial to the proper operation of the network. Better not
fat finger anything. Ever.
Or do I move all my backbone facing datacenters into a network that is
not routed on the Internet, but only on the mgmt net? That has it's
own host of problems.
And you still have to convince the router not to propagate routes that
it learns from the mgmt net into the public network. This can be done
with filters, but when you have a global mgmt network spread over many
netblocks, regions, etc, it's ugly.
The router needs to act as a router to the public network. But it
needs to act as a host (with only 1 interface) to the mgmt net. This
is not how current routers work.
Been there, done that, it's not that simple.
ericb
Eric Brandwine wrote:
Please, tell your vendors you want line-rate filtering up to layer 4.
And when you do so, be prepared to pay what it will cost to deliver
that.
OK, but that's filtering. The telnet/ssh/snmp daemon is still
listening on all interfaces. You can't get there, as long as your
filter stands, but those are some hard filters to write.
Creating a 'source interface' ACL for local services (vty's, snmp, sshd,
*cough* httpd), etc would suit the purpose nicely, and make the GRE
approach feasible w/o touching production paths. ...and an on-going wish
of mine for an 'evaluate <extended _or_ reflexive>' syntax would simplify
the maintance of ACL's in general. But of course, even under 12.2
snmp-server still insists on numbered acl's so maybe this is all overly
optimistic.
..kg..
Eric Brandwine wrote:
Please, tell your vendors you want line-rate filtering up to layer 4.
And when you do so, be prepared to pay what it will cost to deliver
that.
Absolutely! And it'll be well worth it too. The up-front cost is
higher, but economy of scale will bring it down. Having routers that
can protect themselves, protect devices behind them, track attacks,
and provide vastly improved visibility into your network will pay for
itself quickly. Imagine, a router that cannot be knocked over!
I have this argument with Chris any my management all the time. Up
through layer 4, headers are well defined, bit fields, 16/32 bit ints,
etc. Filtering at this point is just making a decision to do so, and
designing it into hardware. Juniper did it from the beginning
(mostly), and does it well. More recent Cisco GSR line cards (Engine
4, etc) come close. It's just another ASIC. Filtering past layer 5
is open to argument, that's a much harder job, but up till 4, it's
almost free.
When we installed CF chips in all our M40s, the amount of extra
information that we gained about our network was amazing. We
regularly see multi-gigabit attack flows in the network, and are now
able to mitigate/filter/track them. It's a good thing, and with
general purpose filtering capabilities, you're always finding new uses
for them.
ericb
jlewis@lewis.org wrote:
Do we then also buy/maintain VPN hardware to connect all the various 1918
management networks to the NOC?
Um, it isn't that hard or expensive. I just put an older box -- like a
133 or 200 MHz machine -- at each pop, running OpenBSD. Allows a
simple VPN throughout, and runs ntpd, too. And sometimes running a
remote copy of MRTG at a particular POP is nice for hunting down infected local DSL customers without tying up the backbone.
Look, it's a lot less costly than the routers, the DSLAMs, even the
managed switches. My main difficulty is they aren't rackable (just
old desktop machines), so they sit in the bottom of the rack. Someday,
someday.
It's time we all run with better security. (As we frantically put in more filtering in the middle of the night based on the
report -- no matter how proactive we try to be, the bar keeps moving and moving.)
>> Without control plane seperation (and it's not possible with Cisco,
>> Juniper, or most other routers out there), management services are
>> listening on the public network, and that makes this very scary,
>> regardless of filtering policies, etc.> interfaces {
...
> }
> firewall {
...
> }OK, but that's filtering. The telnet/ssh/snmp daemon is still
listening on all interfaces. You can't get there, as long as your
filter stands, but those are some hard filters to write. They're
simple when they're simple, but they're very complex when they're not.
Ah, I wrote our filters in an hour or so, from the principle default is
deny, only allow what is strictly needed for the routing protocols etc
to work.
You're relying on your filters, rather than on proper configuration
of the daemon. On a UNIX system, you can bind a service to
all interfaces (e.g. *.161) or just to a specific interface
(10.1.2.3:161). This should be possible in general, on all routers.
Agree, there is no reason why one cannot configure services to
only listen to a specified interface/address.
We HAVE an OOB management network. This is where all our console
servers, switches (there is no Ethernet in the backbone, don't shove
VLANs at me), etc all live. This address space is not routed to you.
We like this. There's no cost issues, we've already paid for it, and
need it for our layer 1/2 network anyway.But then you plug an IP port on the router (vs. a console port) into
the mgmt net, and you've bridged the public net and the mgmt net.
A Juniper will not forward packets between fxp0 and "real interfaces" or
the other way around.
But if a hacker does get control over the router, he/she can use it as a
jump station towards the mngt. network.
Virtual routers are capable of maintaining multiple routing tables,
but last I checked, Juniper was not. So how do you route this?
Use non routed addresses on the mngt. network, and enforce filters on
the routers making up the mngt network, so the only traffic allowed from
the routers are return traffic to requests from the mngt. systems to the
routers.
I send an SNMP query to the device. It comes in over the mgmt net
(because for me, in my datacenter, the loopback for that device (or
it's mgmt IP) is routed across the mgmt net). The device recieves,
digests, and decides to respond to this query. Where does it send
it? My datacenter is routed on the internet, so does it send it out
the public interface?
Just make sure the request is seen from the routers as comming from
private non routed addresses, either by assigning them to the mngt.
servers, or by using NAT.
Or do I route my datacenter over the mgmt net? You can start
filtering, but then those filters are suddenly very important, crucial
to the proper operation of the network. Better not fat finger
anything. Ever.Or do I move all my backbone facing datacenters into a network that is
not routed on the Internet, but only on the mgmt net? That has it's
own host of problems.And you still have to convince the router not to propagate routes that
it learns from the mgmt net into the public network. This can be done
with filters, but when you have a global mgmt network spread over many
netblocks, regions, etc, it's ugly.
routing-options {
static {
route 172.16.0.0/16 {
next-hop 172.16.0.1;
no-readvertise;
}
}
}
Where 172.16.0.1 is a router on the mngt. network, and 172.16.0.0/16 is
the entire mngt. network.
The router needs to act as a router to the public network. But it
needs to act as a host (with only 1 interface) to the mgmt net. This
is not how current routers work.
Correct, that would be ideal, but there are ways to work around
it ...
Been there, done that, it's not that simple.
But not impossible either ...
/Jesper