No, the SP can't be the 'Internet
firewall' for customers,
They can if the SP supplies and manages the CPE device. Nowadays, a lot
of functionality could potentially be provided in a CPE device. Hardware
cost and hardware capabilities are no longer barriers to doing this.
There is still software cost, of course, but that can get amortized over
many years and many, many subscribers.
but protecting one's
customers from being spammed/packeted from purported source
addresses
which are by definition nonsensical (as well as protecting everyone
else's customers from same) doesn't seem much to ask.
What's needed here are better/easier/less brittle mechanisms for
same. Until such time as they're invented and deployed, let's not
make the perfect the enemy of the merely good, yes?
Less brittle mechanisms don't need to be invented. There is nothing hard
about putting an operational process in place to manage bogon filtering.
There is also nothing hard about writing some software to update bogon
filters. People do this kind of thing every day in network operations
without much fanfare precisely because it is nothing special. It's just
good operational practice in a company that does not expect its vendors
to spoonfeed them everything.
There's kind of a disease in the telecomms business. Many years ago
someone got the idea that developing telecomms devices and systems was
one business, and operating those devices and systems was another. Over
time this crystalized into the idea that an operator bought platforms
from vendors and then operated those platforms according to the vendor's
instructions. This may be good for the capital investment industry
(banks etc.) but it fails when there is a need for creativity in the
telecomms operator. Sometimes, network operators have to take the bull
by the horns and develop their own systems to do a job that vendors
simply don't understand.
--Michael Dillon
Concur - but it seems that many seem to be looking for someone else to do this for them (or, perhaps, the lack of someone to do it for them as an excuse to do nothing at all).
How much of a problem is traffic from unallocated addresses? Backbone operators probably have NetFlow data which they could mine to find out.
On the other hand, how much of a problem is obsolete bogon filters causing
everytime IANA delegates another block to an RIR?
Or by the way, how much spoofed traffic uses allocated addresses?
I think Sean raises a good point. I guess the larger picture is what are we
trying to protect and what are trying to protect that from.
Donelan
Bingo.
The problem isn't with "security people", it's with "security people
who use checklists -- and old ones at that -- in preference to
thinking".
Droids exist in every field....
--Steve Bellovin, http://www.cs.columbia.edu/~smb
How do you know, if you're the one being attacked and you have no idea if the originating network or their immediate upstream implemented BCP38? Shall we just discard ingress filtering? If few attacks are using it today, should we declare it no longer relevant? At the same time we should ask if we should be x-raying shoes at the airport, since there's only been one guy who tried to blow up his shoes. The larger security question is, "do you stop looking for old threats simply because they're not the most common threats?" How many CodeRed packets flow over the Internet on a typical day? I assure you it's not zero.
The initial drafts of the document that became BCP38 were written 10 1/2 years ago, triggered by a serious problem of spoof-based attacks that were causing serious problems including serious interruption of services. The problem had a solution, but one that required cooperation among networks. The operation of the entire Internet required cooperation among networks. I don't know to what degree any sense of cooperation is left these days. Probably won't matter when Google or ATT take over the whole thing. In the mean time, the presence of an ACL line or two at the border of each edge network is NOT a significant burden. Yes, Cisco and others have implemented uRPF that can do the same thing with a bit less typing in some cases. I really don't care which mechanism is used. I do care when my network is hammered with packets. When I send reports to other networks and they can't be sure the packets came from their networks, that's not helpful.
So there, that's my rant about why we might all want to try and keep the 'net a cooperative place, and a bit about how ingress filtering continues to play a part in that cooperation.
This is pretty far from the topic of the bogon list issue with 96/8 though.
No one has done the digging required to answer any of these questions, unfortunately.
Speaking of bogons and more practical daily operation issues, perhaps
you guys can help reaching the fine folks at AS7643 or maybe their
upstream provider can be kind enough to filter out the following:
BGP routing table entry for 123.0.0.0/8, version 14613827
Paths: (1 available, best #1, not advertised outside local AS)
Not advertised to any peer
3561 3491 7643 7643 7643 7643
Thanks.
Show me the data.
How many CodeRed packets originate from unallocated addresses?
Is the proposal actually effective at detecting or protecting against the threat? Or is it just a wasted effort for show?
http://www.tsa.gov/press/happenings/kip_hawley_x-ray_remarks.shtm
Instead of dropping packets with unallocated sources addresses, perhaps backbones should shutdown interfaces they receive packets from unallocated address space. Would this be more effective at both stopping the sources of unallocated addresses; as well as sources that spoof other addresses because the best way to prevent your interface from being shutdown by backbone operators is to be certain you only transmit packets with your source addresses.
http://www.completewhois.com/hijacked/files/203.27.251.0.txt
http://www.completewhois.com/hijacked/index.htm
This can proof the opposite.
Malware comes from redirected allocated blocks, not from bogons.
Kind regards
Peter and Karin
Sean Donelan wrote:
uRPF or plain source-based filtering for the IP blocks allocated to the customer is enough. Shutting it down doesn't make any commercial sense, customers wont buy your service if their port is going to be shut down due to a single packet. They'll (likely) understand if you won't forward a packet from them which has a source address not not belonging to them, though.
completewhois.com steht zum Verkauf - Sedo GmbH
completewhois.com steht zum Verkauf - Sedo GmbH
This can proof the opposite.
Malware comes from redirected allocated blocks, not from bogons.
I don't think this is proof. The haphazard way that BCP38 and ingress
prefix filtering of Bogon/DUSA make 'spoofing' from these Bogon/DUSA
blocks unprofitable to a miscreant and forces them to work too hard.
What this data does demonstrate is that hijacking of valid prefixes has
not been mitigated. And, there is most likely an economic motivation to
use the hijacked prefixes. In other words, the miscreants can use the
technique over and over - not get caught - not work too hard - and make
money (the first three and most important principles of miscreants).
This data points to another problem - where SPs are not putting ingress
prefix filters on their BGP speaking customers. That is another area
where you have a lot of operational entropy issues. OPEX is increased on
the building of the prefix provision tool, the maintenance of the
policy, synchronization of that policy with the peer ingress filters,
and customers calls required when ever the customer gets prefix updates.
Hence many (most) operators rather not do the prefix filters on their
customers (usually 2 to 4 lines of policy on a J & C router). For many,
the OPEX is just too high.
No one has done the digging required to answer any of these
questions, unfortunately.
Can you get a valid answer to this based on the existence of BCP38?
What I mean is, if your upstream is filtering bogons, you can't get a
good read on the amount of "bad" traffic sourcing from "illegal"
addresses. However, I'm sure it's there. If we stop filtering
so-called "bad" addresses, I'm sure that the attacks from those
addresses will increase when it's realized that the filters are gone.
I agree with others in that you can't stop looking for old attacks
just because they don't happen much anymore. But we can improve the
ways we look. uRPF is definitely a dynamic option, but as I
understood it, there were issues with using it on multi-homed networks
with asynchronous routing. Granted, it has been some time since I've
looked at uRPF.
I think something like the Cymru bogon route server is great, but I'm
not a very trusting person when it comes to something like that. I
don't like giving up that level of control. Of course, at some point,
I suppose have to trust something...
I definitely believe in filtering both bogons and RFC 1918 space, it's
just a management issue that has to be dealt with.
When customers misconfigure their router, e.g. wrong BGP neighbor or ASN, wrong interface IP address, exceed max prefix limit, etc; don't they lose Internet connectivity until they fix it?
A properly configure router should never forward even a single bad packet. If it does, isn't it likely to have configuration problems so
why continue to keep misconfigured routers connected?
Customers are unlikely to fix problems which don't cause them to lose
service.
Even though the BOFH in me agrees with you, I also know that every cent on my paycheck comes from the customers, so I prefer not to treat them like crap. If I can protect the internet from my customers by doing uRPF or source IP based filtering, I achieve the same thing as you but with less customer impact.
Also, all the examples you give implies a BGP transit customer. I am imagining all kinds of customers, from colo customers where I am their default gateway, to residential customers where it's the same way. Disabling their port and punting them to customer support is NOT a cost efficient way of dealing with the problems, at least not in the market I am in.
Also, all the examples you give implies a BGP transit customer. I am imagining all kinds of customers, from colo customers where I am their default gateway, to residential customers where it's the same way.
I tried to give examples upstream of a router, not a bridged/direct
connection which may have all sorts of unroutable junk which a router should not (and mostly doesn't) forward. Although spoofing MAC addresses
is probably suspicious behaivor in most bridged networks too.
Disabling their port and punting them to customer support is NOT a cost efficient way of dealing with the problems, at least not in the market I am in.
Isn't this true of everything (bad source addresses, worms, abuse, etc). Does hiding/ignoring the problem just makes it worse because there is no incentive to fix the problem while it is still a small problem? If it isn't important enough to bother the customer, why bother to fix it?
How you stop forwarding bad stuff is a local decision. As long as you stop it, no one will turn off your interface. If your network is forwarding so many packets with false source addresses that it would be a major customer support cost issue to fix, your network probably has other configuration problems. You are probably just deferring those customer
service costs until an unpredictable time in the future when those misconfigurations disrupt other parts of your network.
Let's take a concrete example:
Customer gets hacked, one of their boxen starts spewing traffic with spoofed addresses. The way I understand your solution is to automatically shut their port and disrupt all their traffic, and have them call customer support to get any further.
Do you really think this is a good solution?
I don't see any customer with a choice continuing having a relationship with me if I treat them like that. It will cost me and them too much.
So instead I just drop their spoofed traffic and if they call and say that their line is slow, I'll just say it's full and they can themselves track down the offending machine and shut it off to solve the problem.
This doesn't sound very scalable. You're almost certainly overcommitted on
the upstream side and likely looking at congestion if many customers are
spewing.
What do you tell the customer who calls and complains that *he* isn't a major
traffic source, but he's seeing dropped packets and delays on your upstream
link? Do you tell him its full and they can track down which other customer
is the offender?
So instead I just drop their spoofed traffic and if they call and say that
their line is slow, I'll just say it's full and they can themselves track
down the offending machine and shut it off to solve the problem.
This doesn't sound very scalable. You're almost certainly overcommitted on
the upstream side and likely looking at congestion if many customers are
spewing.
If they're spewing spoofed traffic I'm dropping it, so that's not a problem.
What do you tell the customer who calls and complains that *he* isn't a major
traffic source, but he's seeing dropped packets and delays on your upstream
link? Do you tell him its full and they can track down which other customer
is the offender?
Do you usually design networks that can't handle customers using what they have paid for? I don't. (for any reasonable amount of statistical oversubscripion of course)
Mikael Abrahamsson wrote:
Isn't this true of everything (bad source addresses, worms, abuse,
etc). Does hiding/ignoring the problem just makes it worse because
there is no incentive to fix the problem while it is still a small
problem? If it isn't important enough to bother the customer, why
bother to fix it?
Let's take a concrete example:
Customer gets hacked, one of their boxen starts spewing traffic with
spoofed addresses. The way I understand your solution is to
automatically shut their port and disrupt all their traffic, and have
them call customer support to get any further.
Do you really think this is a good solution?
I don't see any customer with a choice continuing having a
relationship with me if I treat them like that. It will cost me and
them too much.
So instead I just drop their spoofed traffic and if they call and say
that their line is slow, I'll just say it's full and they can
themselves track down the offending machine and shut it off to solve
the problem.
Neither one is really all that good but both have merit - some
compromises are in order. We shut them off only if it's causing
serious problems.
If we can mitigate the problem without shutting them off completely we
will. The usual example is customers spewing spam on port 25. We
block port 25 at the customers CPE and notify them as to why and how to
work around the block (use webmail or submission) while they fix the
problem. It's amazing how many customers are just plain OK with that
and never do get around to fixing the machine - but at least they know
that we blocked something for a reason.
Anything you do silently tends to cause customers to decide 'you suck'
and go elsewhere. Line is slow 'cause there machine is beating it to
death? Just get a new provider. When the new one also sucks they
either shrug and decide that's the way it is or finally fix the
problem. Either way the customer is lost to you 'cause they won't come
back even after they figure out it was their problem in the first place.
Shutting them off causes churn, leaving problems silently in place also
causes churn. The middle road mitigates damage and still manages to
keep the customers happy (well.. that might be stretching it a
bit...happier?).