It's better than a sharp stick in the eye, I'll tell ya,
lad.
Listen to me: It's called a "best current practice" for a
reason -- people should do it. Not sit and around and endlessly
discuss it (we've already done that a thousand times).
I wrote it, I stand beside it. I'm sick of hearing why people
haven't implemented it yet -- it's almost five years later
and there's simply no excuse. It's sickening.
- fergie
the problem is that isp security folk doing actual measurement
see very little spoofing. it's easy for the bad folk to get
real bots. and tcp bad things are more popular and desirable,
e.g. spam, ... and tcp does not work from spoofed addresses.
isp security folk have limited resources. so why should they
spend them where there is little perceived benefit when there
are gaping holes elsewhere that need those resources?
yes, it's a nice thing to do. but, in today's disasterous
economy, so are 42 other things that won't get done. so it's
growing slowly. when it solves critical problems, it'll grow
more quickly.
randy
it's cheaper to ignore bcp38 than to implement it.
operators are reactive to abuse, not proactive. though this is slowly
changing as abuse becomes a significant % of network traffic.
-Dan
Date: Sun, 10 Oct 2004 20:14:01 -0700
From: Randy Bush
when it solves critical problems, it'll grow more quickly.
Maybe.
* Use 25/TCP for SMTP and 587/TCP for submission
* Block outbound SMTP by default, but allow for the clueful
* Run SMTP authentication
* Let each authenticated user have whitelisted sender addresses
that they can use
* Limit whitelist size
* Add a delay and/or rate limit to whitelist additions.
Not perfect, and certainly subject to social engineering and
possible automated attack on the whitelist mechanism, but it
should decrease the number of cable/DSL pipes filled with forged
mail transmissions.
This isn't the first time I've suggested it, and I'm sure others
have, too. Not once have I received a response to the extent of
"I'd love to implement this if it existed". People are worried
about OPNs, not their own networks. IMNSHO, worrying about N-1
ASNs scales far more poorly than worrying about one ASN.
Of course, note the parallel to BCP38; I'm sure someone will be
quick to point out that, unless adopted universally, forged mail
will still exist. Enter SPF as a bandaid on the receiving side,
and rehash that discussion. Combine with BMF, DNSBLs, and one is
well on the way to much cleaner mail.
Has anyone on NANOG ever solved a jigsaw puzzle with 500+ pieces?
Or are "age 2 to 4" puzzles too difficult, as even they tend to
have around ten pieces per puzzle?
Eddy
Tell it to the vendors of the layer 3 customer attach devices. I was just
speaking about this with a major "up and coming layer 2/3 switch vendor"
the other day, who had no implementation or real plans to implement uRPF
in the immediate future. It apears that there are no enterprise customers
asking for this feature (not a shock), despite the fact that they are
probably a leading generator of hacked machines throwing bad packets down
reasonably big pipes.
uRPF support is hardly universal outside of Cisco and Juniper, and even
Juniper didn't add uRPF until semi-recently. I know Foundry still doesn't
have support for it (not really anyways, they added something they call
uRPF and have you activate through "ip verify unicast reverse-path" but it
isn't really uRPF, just a way to prevent outside networks from spoofing
local addresses, and you have to manually tag every interface as internal
or external or the addresses aren't protected), and I'm pretty sure
Extreme doesn't have it (or at least I've never seen anyone use it, but I
don't follow 0 day Extreme code). Short of the Cisco L3 switch solutions,
I'm not aware of any other vendor with a L3 switch who has uRPF
functionality.
I can't think of a single web hoster (or at least someone who wasn't a
real network operator who got into hosting somehow) who even knows what
uRPF is, let alone implements it for their customers. We're counting on
their IP providers to filter the bad packets from the hundreds of FE
connected servers on burstable GigE pipes out there, but some of them just
aren't. I can name several major well-known "cheap" networks who don't do
any uRPF filtering of their customers, and a couple more who don't do any
real prefix-lists on their BGP speaking customers, only prefix-limits.
There are even fewer vendors who are implementing automation for filtering
basic BGP speaking customers, such as Juniper's ability (sort-of) to
re-use a route filtering prefix-list for source address filtering in a
firewall. It's not quite perfect, since you can't do length modifiers
(upto /24, etc) in a prefix-list, and since you have to create a seperate
policy-statement (with all routes listed in route-filter statements),
prefix-list, AND firewall filter for each and every customer, but at least
it is a start. If you don't have this, or uRPF at all, you are left trying
to script ACLs based on interface configurations, static routes, and/or
registered prefixes, and the bottom line is that most networks aren't
going to do it if it takes that much work.
If and when all the vendors who are making boxes that are supposed to be
used for customer aggregation start implementing uRPF, especially
considering all the enterprises, universities, and international network
using these L3 switches as their sole routing equipment (think price),
maybe we'll start seeing some more progress. And of course, those
remaining service providers with the gear to implement it who havn't done
so need to be yelled at more. Unfortunately, no customer ever complains
(or takes their business elsewhere) when their provider doesn't do source
address filtering, only when they are getting a DoS attack and their
provider can't do anything about it.
I've removed the rest of your message, talking about which vendors do or don't have what capabilities. While I agree it'd be nice if more vendors offered automated tools for implementing ingress filtering, such tools are unnecessary in most corporate network cases, thus the lack of corporate customers asking for the feature. In reality every device offering access control lists capable of filtering on source IP address can and does have sufficient capability to implement BCP38.
While I appreciate the desire to have a single switch solution, like was possible with BCP34, it's a bit more complex to do in this case. It is, however, disingenuous to say that devices don't support BCP38 because they don't have an automated widget to implement it. Keep in mind that uRPF is an implementation of BCP38 capability, and other implementations are entirely possible.
This was probably obvious to you, but others reading might find the clarification useful.
Yes if a box has source address packet filtering capabilities you can
filter packets by source address ("Duh"). This doesn't mean that it is
going to be sane or easy to implement the filtering by manually
maintaining an acl of every prefix/host on every interface where you could
have a customer or corporate box injecting spoofed packets into the
network. I believe there are plenty of corporate networks out there that
are far too complex to maintain with manual ACLs, I believe the reason
that no one cares is simply because... no one cares. 
If you expect people to be able to maintain these filters on any scale,
they need tools. Certainly uRPF is a good tool to do this, and certainly
someone could implement some others that are different, but the complete
lack of any tool, especially on the boxes where you should be doing this
filtering, counts as a failure in my book.
>
> I've removed the rest of your message, talking about which vendors do or
> don't have what capabilities. While I agree it'd be nice if more vendors
> offered automated tools for implementing ingress filtering, such tools are
> unnecessary in most corporate network cases, thus the lack of corporate
> customers asking for the feature. In reality every device offering access
> control lists capable of filtering on source IP address can and does have
> sufficient capability to implement BCP38.
>
> While I appreciate the desire to have a single switch solution, like was
> possible with BCP34, it's a bit more complex to do in this case. It is,
> however, disingenuous to say that devices don't support BCP38 because they
> don't have an automated widget to implement it. Keep in mind that uRPF is
> an implementation of BCP38 capability, and other implementations are
> entirely possible.
>
> This was probably obvious to you, but others reading might find the
> clarification useful.Yes if a box has source address packet filtering capabilities you can
filter packets by source address ("Duh"). This doesn't mean that it is
going to be sane or easy to implement the filtering by manually
maintaining an acl of every prefix/host on every interface where you could
have a customer or corporate box injecting spoofed packets into the
network. I believe there are plenty of corporate networks out there that
are far too complex to maintain with manual ACLs, I believe the reason
that no one cares is simply because... no one cares.
One of your arguments presented was that corporate customers weren't asking for unicast RPF, and I responded that corporate customers are not in need of automated mechanisms to implement BCP38, since in most cases their networks are EDGE networks, and it's quite simple to filter your egress points to ensure you don't send out any spoofed packets.
You laid out a complaint against the equipment makers claiming they weren't implementing automated mechanisms BECAUSE the corporate customers were not asking for such tools, and I simply pointed out they shouldn't be expected to do so. If network operators need features, they need to ask for them when talking with potential vendors.
Network operators need to ensure downstreams don't advertise AS's they're not supposed to. Last I looked, that required some custom work (whether done by scripting or whatever, it's done off the router and pushed in). At the same time you're building those lists, you could be building ACLs. Some border routers will do this just fine, others won't. Next time you're qualifying routers for possible use, maybe the ability to handle wire speed acls might be worth testing?
If you expect people to be able to maintain these filters on any scale,
they need tools.
Why do those tools need to be built into the router? Are your tools for maintaining AS path filteirng built into the routers? Are the tools to archive and compare router configurations most of you use built into the routers?
Certainly uRPF is a good tool to do this, and certainly
someone could implement some others that are different, but the complete
lack of any tool, especially on the boxes where you should be doing this
filtering, counts as a failure in my book.
I disagree that the tools must be an integral part of the router software. Perhaps it's time to think outside the (router) box?
Daniel Senie wrote:
One of your arguments presented was that corporate customers weren't asking for unicast RPF, and I responded that corporate customers are not in need of automated mechanisms to implement BCP38, since in most cases their networks are EDGE networks, and it's quite simple to filter your egress points to ensure you don't send out any spoofed packets.
There is, of course, the issue of multihomed networks, or networks that have satellite connectivity etc emitting spoofed source packets.
Yes I know that multihoming customers must make sure packets going out to the internet over a link match the route advertised out that link .. but stupid multihoming implementations do tend to ensure that lots of people will yell loudly, and yell loudly enough for several tickets to be escalated well beyond tier 1 NOC support desks, for ISPs to kind of think twice before they put uRPF filters in ..
srs
a common occurance we've seen is a customer of a customer NOT announcing
, nor planning on announcing, their routes to their upstream#1 which they
use ONLY for outbound traffic (cheap transit for instance, and perhaps
only for some portions of their total sources) though they announce to
upstreams#2-N the proper sources to gather the return traffic. These
things make uRPF 'difficult'.
As to the entire conversation I think more folks will implement BCP38, or
parts atleast, as they replace gear which is not capable of dealing with
parts of the implementation of BCP38. Hopefully new gear is being
tested/certified/bought that can do wirespeed filtering on all interfaces.
-Chris
* christopher.morrow@mci.com (Christopher L. Morrow) [Tue 12 Oct 2004, 05:18 CEST]:
a common occurance we've seen is a customer of a customer NOT
announcing , nor planning on announcing, their routes to their
upstream#1 which they use ONLY for outbound traffic (cheap transit for
instance, and perhaps only for some portions of their total sources)
though they announce to upstreams#2-N the proper sources to gather the
return traffic. These things make uRPF 'difficult'.
You could use uRPF-loose there, or the customer could do:
!
route-map outbound-only permit 10
match prefix-list myprefixes
set community no-export
!
And bash the people who, in this age, don't have "neighbor x.y.z.a
send-community" on all their BGP sessions.
-- Niels (who recently had a CCIE claim that he was "not aware
of a single ISP accepting communities from its peers"
- well, my experience begs to differ, with his
employer a rare and lonely exception to the rule)
this does not address the problem, the customer's customer isn't
announcing routes for this traffic so there is nothing to no-export 
Example:
the 'chris.net' network is a customer of MCI, his customer "bakker.net".
'bakker.net' decides 'chris.net' has priced transit cheaply this
year/month/day and choses not to accept traffic from 'chris.net' but send
all outbound traffic through 'chris.net'. 'chris.net' never seens routes
for the sources sending this traffic, yet passes it along to the upstream,
which also has no routes for 'bakker.net' via 'chris.net'.
Regardless, the point here is: "Things seem like they may be getting
better, as 'security' requirements are now firmly being included into new
equipment purchases."
There is, of course, the issue of multihomed networks, or networks that
have satellite connectivity etc emitting spoofed source packets.
y'know, SAC004 (http://www.icann.org/committees/security/sac004.txt) is
only four pages long, and one of those is references. you should read it
before you call multihomed networks an "issue" wrt BCP38 deployment. in
fact, you should read it, and BCP38, and BCP84, before participating in
this discussion at all, either here, or at the bar-bofs next week.
Paul Vixie wrote:
There is, of course, the issue of multihomed networks, or networks that have satellite connectivity etc emitting spoofed source packets.
y'know, SAC004 (http://www.icann.org/committees/security/sac004.txt) is
only four pages long, and one of those is references. you should read it
before you call multihomed networks an "issue" wrt BCP38 deployment. in
fact, you should read it, and BCP38, and BCP84, before participating in
this discussion at all, either here, or at the bar-bofs next week.
Multihomed networks are not an issue. Stupid implementations of these? Yes sure.
srs
...
Example:the 'chris.net' network is a customer of MCI, his customer "bakker.net".
'bakker.net' decides 'chris.net' has priced transit cheaply this
year/month/day and choses not to accept traffic from 'chris.net' but send
all outbound traffic through 'chris.net'. 'chris.net' never seens routes
for the sources sending this traffic, yet passes it along to the upstream,
which also has no routes for 'bakker.net' via 'chris.net'.
uPRF is only one of several ways to implement BCP38. you could do it with
contracts and reverse-SLA's and thus no technology (on your side) at all:
demand that a customer pay 10X his bill, or $1.00 per packet, whichever is
lower, if they emit packets with source addresses no explicitly named in
the contract. why pay for expensive upgrades on your end of the link, when
all you really care about is that BCP38's rules are observed?
Regardless, the point here is: "Things seem like they may be getting
better, as 'security' requirements are now firmly being included into new
equipment purchases."
that is of course good news. but it demonstrates a pitfall in CFO-think,
which is the belief that participating in assymetric cost:benefit efforts
(where uunet bears the cost of an upgrade in order for all the non-uunet
parts of the internet to get the benefit of less spoofed traffic, and the
abuse incident costs don't drop nearly enough to pay for the upgrades) is
essentially a selfless act.
we all want cleaner ddos flows. when we get ddos'd, we want to be able to
look at the source addresses, look 'em up in whois, and call the launch-isp,
and get things stopped. we want to be able to turn on flow shaping and
know that an attacker can't cause us to use an arbitrarily large number of
buckets. we *all* want these things. even the bad guys, who are often the
victim of ddos attacks by other bad guys, want these things.
how are we going to get there? the first thing is, some nets who want the
internet to work this way have to implement BCP38 in their own corner of the
internet. then they have to start de-peering with nets who don't do it, and
offer a better rate to customers who do it than to those who don't. then
they have to de-peer with anyone who doesn't require their peers and customers
to do it. then they have to refuse as customers anyone who won't do it.
it's all very simple, and it's inevitable. you and your CFO's have a couple
of choices to make. first thing is, do you want the insurance companies,
government regulatory agencies, and ISO9000 people to be making these rules
or do you want to make them at the technical and business level? (this is
called "industry self regulation" and it's an either-or proposition.) second
choice to make is, do you want to do this early enough that you can fold it
into your normal purchase/retire cycle, or do you want it rammed down your
throat later in an irritating, short-timeframe, expensive, embarrassing way?
fergie complained about the lack of chutzpah among a community who used to
be distinguishable from other technical communities by that very chutzpah.
i agreed but pointed out that the same engineers are here, but with vastly
fewer of their buddies to help out, and vastly more supervision from the CFO
than used to be the case. HOWEVER, there is still an opportunity to show
some leadership, and GET THIS DONE without waiting for fireman's fund and
a bunch of ISO9000 wonks to have to recognize it in a corporate risk profile.
uPRF is only one of several ways to implement BCP38. you could do it with
contracts and reverse-SLA's and thus no technology (on your side) at all:
demand that a customer pay 10X his bill, or $1.00 per packet, whichever is
lower, if they emit packets with source addresses no explicitly named in
the contract. why pay for expensive upgrades on your end of the link, when
all you really care about is that BCP38's rules are observed?
No reasonably sized provider is going to do that. There is too much
competition, most of which is based on price. Until the companies creating
the price pressure die (as in die completely, not re-emerge under a new,
slightly different name), there is going to be no financial insentive for
anyone to spend money improving their network. Let me underline, I am not
talking about smaller ISPs, smaller networks or smaller service providers.
that is of course good news. but it demonstrates a pitfall in CFO-think,
which is the belief that participating in assymetric cost:benefit efforts
(where uunet bears the cost of an upgrade in order for all the non-uunet
parts of the internet to get the benefit of less spoofed traffic, and the
abuse incident costs don't drop nearly enough to pay for the upgrades) is
essentially a selfless act.
Rubbish. CFO speak is what keeps the companies alive. Engineer-speak
typically lands the company in chapter 11. Companies in Chapter 11 have too
many operational decisions dictated by the courts, and those that think CFO
speak would greally hate to hear courts on the topic.
we all want cleaner ddos flows. when we get ddos'd, we want to be able to
look at the source addresses, look 'em up in whois, and call the launch-isp,
and get things stopped. we want to be able to turn on flow shaping and
know that an attacker can't cause us to use an arbitrarily large number of
buckets. we *all* want these things. even the bad guys, who are often the
victim of ddos attacks by other bad guys, want these things.
It is possible that _nanog_ subscribers want this. I am not quite clear how
one can make that generalization about those behind kor.net, those in .ru,
.ua etc. Finally,
how are we going to get there? the first thing is, some nets who want the
internet to work this way have to implement BCP38 in their own corner of the
internet. then they have to start de-peering with nets who don't do it, and
offer a better rate to customers who do it than to those who don't. then
they have to de-peer with anyone who doesn't require their peers and customers
to do it. then they have to refuse as customers anyone who won't do it.
Last time I checked it was 2004, not 1998. The companies are financed by
revenues that they generate, not IPOs or VCs based on a promise of enormous
payoff sometime down the road. Cash is the king.
it's all very simple, and it's inevitable. you and your CFO's have a couple
of choices to make. first thing is, do you want the insurance companies,
government regulatory agencies, and ISO9000 people to be making these rules
or do you want to make them at the technical and business level?
Yes, I do. This will level playing field and hopefully force a few of the
big networks out of business completely, decreasing price pressure on this
service. A drop in the price pressure will create an opportunity for those
companies to spend the money (should they want to or be forced to) to be
better internet citizens.
This is just the cold blooded economic reality. The same reality which
dictates that only smaller companies can enfore strict anti-spam policies,
and prevent their customers from behaving badly.
Alex
Excerpt from the text quoted above:
2.3. For a DDoS attack to succeed more than once, the launch points must
remain anonymous. Therefore, forged IP source addresses are used. From
the victim's point of view, a DDoS attack seems to come from everywhere
at once, even from many IP addresses that are unallocated or otherwise
invalid.
How many people have seen "forged" spoofed IP addresses being used
for DOS attacks lately?
Bora
You want cold-blooded economic reality? I'm nominally a consultant,
programmer, writer, and sysadmin - in that order. Over the past two
years, I've spent over half of my time dealing with spam that I would
never see if the rest of you had your acts together.
You're costing me money and detracting from my ability to earn my keep.
And yet, over those past two years, I've cut my spam load inbound by
98-99% with very few FPs (on the order of one report per week, which is
far less time than I spend proactively blocking unwanted mail from
badly policed networks). I taught myself the dark art of writing sendmail
rulesets, built a database of ~7K rDNS naming conventions, ~18K poorly
configured mail sources that inundate my servers with bounces from mail
I didn't send, ~1.7K legitimate mail hosts that have relayed spam, 419
advance fee fraud scams, phishing frauds, or otherwise so utterly failed
to police your own network egress points that we've blocked mail from
you altogether. Where are the rest of you and why aren't you doing this?
If I, and my little 7-man company, can afford to have me solve the
problem on our end, why the heck can't you do the same?
If I can do it, as the lone admin in a 7-man company, I should think
that anyone on this list could do it.
But I don't see it. All I see is arguing over whether or not to implement
this or that BCP, or this or that antispam measure, or this or that
anti-backscatter measure, or this or that antivirus measure.
Get on with it. Email is dying, if it isn't already dead. The killer app
isn't the Web, it's email. If email dies, who will want Internet
services? You'll all be out pounding pavement and wishing you'd had the
guts to do what you know is right and necessary.
It's not possible for anyone here to say they didn't see it coming. The
Cassandras have been screaming loudly, and often clearly, for years.
Profoundly disappointed in the utter moral bankruptcy and cowardice of
the NANOG constituency,
Steve
<raises hand>
Not saying that I have not see non-forged DoS attacks too, or even which is more common, just saying they exist, are happening today, and cause non-trivial problems for some providers.
sampling), it appears non-spoofed sources are a bigger problem. But ignoring spoofed sources would be a mistake, IMHO.
it does still happen... I've not run the numbers for our reactions to say
'50% spoofed/50% non-spoofed' but it certainly seems like 'more' are
non-spoofed lately. This could be a simple swing of the pendulum, or other
'better' things like more people egress filtering.
-Chris
