We've been handling a multi-vector DDoS - 40-byte spoofed SYN-flooding towards www.cisco.com (198.133.219.25/32) as well as an HTTP-AUTH resource-exhaustion attack, and working these issues with our upstreams. Our apologies for any inconveniences, and our thanks to those who've assisted in tracing and blocking the spoofed traffic.
We're continuing the work the issue, and would be grateful if operators would check for 40-byte spoofed TCP headed towards 198.133.219.25/32 and trace/block it as warranted. Your patience and understanding are greatly appreciated.
My mailbox has filled quite a bit (to the tune of a dozen-plus mails)
with comments along the lines of "don't quote me, NANOG is too important
for my work, I don't want to get on Sue Harris' bad side" since my last
so-called "off-topic" NANOG post (which all but *one* person, other than
Sue Harris, found to be "within range and reason").
The spammers,
the DDoS'ers,
the proxy scanners and rapists,
the SMTP auth crackers.
the trojan spreaders,
the DNSBL-DOS'ers,
the hardcore computer criminals
are the evil army of one?
The following well-remembered lines come to mind here, and excuse me if
you hear a slight hysterical laughter from my direction:
"First They Came for the Jews
First they came for the Jews
and I did not speak out
because I was not a Jew.
Then they came for the Communists
and I did not speak out
because I was not a Communist.
Then they came for the trade unionists
and I did not speak out
because I was not a trade unionist.
Then they came for me
and there was no one left
to speak out for me."
The following well-remembered lines come to mind here, and excuse me if
you hear a slight hysterical laughter from my direction:
I don't know what your post has to do with the original topic, but if
you don't like the way NONOG is moderated, please feel free to start
your own Network Operators mailing list.
As far as comparing NANOG moderation to Nazi Germany that is
disgusting and beneath contempt.
I'm only guessing here, but I think what he may have meant was:
First They Came for the IRC bots
and I did not speak out
because I did not run a bot.
Then They Came for the IRC servers
and I did not speak out
because I did not run an IRC server.
...skip a few years...
Then They Came for the DNSBLs
and I did not speak out
because I did not run a DNSBL.
Now that they've come for cisco, maybe law enforcement, network operators,
and router vendors will all get their $h!t together and do something to
put a stop to these DDoS attacks that have been going on in various forms
for several years.
A handful of people (an assumption on my part) have the power /
distributed bandwidth to bring just about any internet site/network to its
knees using the distributed.net meets DoS tools they've created and
distributed to thousands, perhaps millions of internet connected windows
boxes.
Anyone who doesn't think that's an operational issue, just wait until it
bites you on the ass.
> > The following well-remembered lines come to mind here, and excuse me if
> > you hear a slight hysterical laughter from my direction:
>
> I don't know what your post has to do with the original topic, but if
> you don't like the way NONOG is moderated, please feel free to start
> your own Network Operators mailing list.
I'm only guessing here, but I think what he may have meant was:
First They Came for the IRC bots
and I did not speak out
because I did not run a bot.
Then They Came for the IRC servers
and I did not speak out
because I did not run an IRC server.
...skip a few years...
Then They Came for the DNSBLs
and I did not speak out
because I did not run a DNSBL.
Now that they've come for cisco, maybe law enforcement, network operators,
and router vendors will all get their $h!t together and do something to
put a stop to these DDoS attacks that have been going on in various forms
for several years.
A handful of people (an assumption on my part) have the power /
distributed bandwidth to bring just about any internet site/network to its
knees using the distributed.net meets DoS tools they've created and
distributed to thousands, perhaps millions of internet connected windows
boxes.
Anyone who doesn't think that's an operational issue, just wait until it
bites you on the ass.
Now we have clear evidence that there are no less than three who
understand the threat.
Zombie networks of 10K or 20K machines all controlled by *one* black
hat are not uncommon now, and I've seen a citation for a single net of 140K.
Let's assume the interesting hosts are on cablemodem, that they have 2Mbit/sec
connectivity, and that one black hat has 10K (if you prefer, he's got 20K but
the rest are on slow links). Now tell me - how many of you have enough
*excess* bandwidth that you can afford not to worry about suddenly being handed
a 200Gbit/sec inbound stream? And if you don't have enough spare capacity,
are you set up to deal with 10K machines attacking, quite possibly with spoofed
addresses because your peers don't ingress filter?
Remember guys - Yahoo got whacked by MafiaBoy using only several hundred
machines. You could be the recipient of a flood 200 times bigger.
And if you're not ready, it won't be an operational issue - it will be a NON-operational
issue, because that's what your network will be....
Read it again. He has a point (not yours).
Perhaps this should be an agenda topic for the upcoming get-
together: A common strategy for dealing with Internet crime. Much of
it does appear to have common roots. (And I'm not even a conspiracy
buff.)
Hm. Oddly enough there's a blurb on <overclockers.com> that
follows this somewhat: <http://www.overclockers.com/articles843/>.
> First They Came for the IRC bots
> and I did not speak out
> because I did not run a bot.
> Then They Came for the IRC servers
> and I did not speak out
> because I did not run an IRC server.
> ...skip a few years...
> Then They Came for the DNSBLs
> and I did not speak out
> because I did not run a DNSBL.
> Anyone who doesn't think that's an operational issue, just wait until it
> bites you on the ass.
Let's add a very important line:
"Then They Came for the OC-3 or smaller connections
and I did not speak out
because I run fat OC-12 - OC-48 pipes"
It's my guess that the "top providers" that ignore cries for help because
they can sink the traffic (and bill for it) without breaking a sweat will
one day find themselves without a plan and without a clue when the Kiddiez
come pounding down the door with something that can saturate their pipes
and bring a major customer down. I hope we don't have to wait until that
time comes around to figure out how to cooperate.
Now we have clear evidence that there are no less than three who
understand the threat.
Heh. Why things like this don't scare everyone on this list sh*tless is
beyond me.
If anyone ever sees garbage coming out of 8059, hit the abuse@ address or
peek at Jared's list for phone info.
If you mean the threat from those who will attack and disable sites
because they don't like what people at those sites say or do, then I
assure you there are many who do understand that threat; some of whom
can see little difference in terms of effect between DDoS attacks run
by individuals, and the null-routing by a backbone network of IPs (or
ranges of IPs) for which they make BGP announcements.
Both are actions designed to interfere with individual freedoms;
both are serious operational issues, and need to be discussed here.
Or was it a different kind of threat that you were referring to, which
might have discouraged some who understand the real threat from talking
about it?
> Anyone who doesn't think that's an operational issue, just wait until it
> bites you on the ass.
Now we have clear evidence that there are no less than three who
understand the threat.
My first thought was that the DDoS was a means of obscuring access to
patches for other vulnerabilities that might be simulatenously exploited.
I'm assuming, though not certain, that Cisco would have alternative
distribution/communication/update channels in such an event, but is that
indeed the case?
We've been handling a multi-vector DDoS - 40-byte spoofed
SYN-flooding towards www.cisco.com
Now that they've come for cisco, maybe law enforcement,
network operators, and router vendors will all get their
$h!t together and do something to put a stop to these DDoS
attacks that have been going on in various forms for several
years.
Maybe this will have the positive effect of motivating Cisco to do more
to encourage best practices such as edge anti-spoof filtering. To begin
with, Barry Green's presentations on these issues are hidden away on
his/Cisco's FTP server (ftp://ftp-eng.cisco.com/cons/) -- maybe it would
be beneficial to put them (along with write-ups) in an easily-accessible
and often-visited area of the main site where people will see them.
These issues aren't just for ISPs: if edge networks would filter their
borders, ISPs wouldn't have to do it for them. (Or in most cases, fail
to do it for them.)
> I'm assuming, though not certain, that Cisco would have alternative
> distribution/communication/update channels in such an event, but is that
> indeed the case?
My access to ftp.cisco.com is working fine whilst the website remains down..
Hi Steve,
No I do realize that what I suggested in my email was just a scenario
removed from the case at hand. What I was suggesting though is that 1) if
the portals of distribution (http and ftp) were DDoS'd, say as a precursor
to exploitation of some other vulnerability. I was not trying to suggest
that all means of communication were blocked and that this particular
instance was one of an opportunistic DDoS. Sorry if I was unclear.
As a jew, I must admit that I also understood the point, and didn't
think of Nazi Germany, although you'd think it would evoke an immediate
emotional reaction (which it admitedly did), but that reaction did not
cloud my judgement.
I think it's safe to assume that most people on this list have a reason
for being on it. Although I am not trying to say that sometimes we get to
see posts that are ... well, that shouldn't be sent before thinking, it
would be wise to read an e-mail twice, even three times, before assuming
mal-intent from its originator.
Maybe this will have the positive effect of motivating Cisco to do more
to encourage best practices such as edge anti-spoof filtering. To begin
with, Barry Green's presentations on these issues are hidden away on
his/Cisco's FTP server (ftp://ftp-eng.cisco.com/cons/) -- maybe it would
be beneficial to put them (along with write-ups) in an easily-accessible
and often-visited area of the main site where people will see them.
As the bandwidth ramps up on the access side, this problem is only going to become more and more prevalent (as if it's not already enough of a problem). While I don't think filtering is the silver bullet, it can certainly help at times. I think some of the larger watch sites (eg SANS, etc.) out there have the right idea - even though reactive in nature, almost real-time dissemination of attack vectors and trending of methods goes a long way towards slowing down some of these attacks. Unfortunately, these single target attacks, such as attacks on Cisco, Ebay, Yahoo, etc. cannot be entirely thwarted if the attacker(s) is/are determined enough. We could go down the client side discussion (you know, the one about certain software vendors, etc.) but that topic has already been covered in great length.
You are making assumptions.. Cisco havent said if the source was spoofed or not,
as a recent nanog thread indicated a lot of attacks do not use spoofed addresses
any more simply because the controllers have access to enough legitimate windows
boxes to not care about discovery of source.
Even with all your BCPs in place if someone can get control of enough machines
across enough networks collectively they can produce enough traffic to overwhelm
absolutely any single system on the Internet.
I am increasingly sharing the opinion that many of these high profile attacks
are carried out by a small group.. spammers or whoever they are, the only way to
tackle them is directly by hunting them down and prosecuting them. Assuming that
there is a cash motivation somewhere (eg spam) this also means that there is a
very high probability the attackers reside in a country where prosecution would
be possible eg US/Europe
You are making assumptions.. Cisco havent said if the source was spoofed or not, as a recent nanog thread indicated a lot of attacks do not use spoofed addresses any more simply because the controllers have access to enough legitimate windows boxes to not care about discovery of source.
I did say "for starters". I put it to you that there is still a non trivial amount of attacking going on that does use spoofed traffic.
Yes, there are lots of IRC controlled zombies, and yes, there are pissed off teenage skript kiddies who shut down the port of houston's servers trying to bomb someone they had a pissing match with on IRC (don't have more details than what I read on Dave Farber's IP list today).
I am increasingly sharing the opinion that many of these high profile attacks are carried out by a small group.. spammers or whoever they are, the only way to tackle them is directly by hunting them down and prosecuting them. Assuming that there is a cash motivation somewhere (eg spam) this also means that there is a very high probability the attackers reside in a country where prosecution would be possible eg US/Europe
Easier said than done. First - prove that the guy did it (or hired a kiddie in china or eastern europe or wherever to do it) Next, prove to the Feds that damage > [what, USD 25K?] was caused. And that is for starters.
You are making assumptions.. Cisco havent said if the source was spoofed or not,
as a recent nanog thread indicated a lot of attacks do not use spoofed addresses
any more simply because the controllers have access to enough legitimate windows
boxes to not care about discovery of source.
Interesting. I read (and just now reread) Mr. dobbins posting and made
the same assumptions, based on the part where he said:
We've been handling a multi-vector DDoS - 40-byte spoofed SYN-