Has anyone had any experience with this device? Turntide.com. Looks like a
traffic-shaping device designed specifically for cutting down spammers
throughput to your inbound SMTP servers. My main concern is, how does it
make the distinction between legitimate mass-mailings (e.g.: mailing lists
such as this one), and spam? Interesting approach to killing spam though I
must say.
-Andy
Andy Johnson [05/04/04 10:49 -0400]:
Has anyone had any experience with this device? Turntide.com. Looks like a
traffic-shaping device designed specifically for cutting down spammers
throughput to your inbound SMTP servers. My main concern is, how does it
make the distinction between legitimate mass-mailings (e.g.: mailing lists
such as this one), and spam? Interesting approach to killing spam though I
must say.
Sounds like is basically the same principle as that of an IDS.
srs
Sounds like YABA (Yet Another Band Aid) solution for spam. If rate-limiting the spam packets does an effective job at killing spam. It will only make the spammers switch to a distrubuted attack method using trojaned virus hosts sending 1 mail message at a time. They are already doing this in some cases. SPAM is a living breathing entity that can learn and adapt. The smarter the network gets at killing it off, the smarter it gets in attacking. The evolution of spam/viruses is astounding and getting quicker all the time. The turntide box may be a good solution but it is expensive, I'll wait for the SNORT add-on that does the same thing 
-Matt
Sounds like YABA (Yet Another Band Aid) solution for spam. If
rate-limiting the spam packets does an effective job at killing spam.
It will only make the spammers switch to a distrubuted attack method
using trojaned virus hosts sending 1 mail message at a time. They are
already doing this in some cases. SPAM is a living breathing entity
that can learn and adapt. The smarter the network gets at killing it
off, the smarter it gets in attacking. The evolution of spam/viruses
is astounding and getting quicker all the time. The turntide box may
be a good solution but it is expensive, I'll wait for the SNORT
add-on that does the same thing-Matt
That's exactly what I was thinking when reading their documentation. A large
portion of the spam we receive on our servers comes from hijacked broadband
customers. I don't see how effective it would be rate limiting these guys,
because the source IP is almost always unique. Still, if it could cut spam
traffic by even half of what they advertise (92% on a system similar to
ours), that would help tremendously.
-Andy
You might want to consider an inexpensive mail relay running OpenBSD's spamd (not to be confused with SpamAssassin's spamd) in conjunction with PF. Spamd is really nice for hurting spammers and/or relays where it can... in their spool. Granted, it's based on address lists like spamhaus or spews, but it's better than content filtering in one very important way... bandwidth savings. With content filtering, the payload is already at your doorstep.
http://www.openbsd.org/cgi-bin/man.cgi?query=spamd
http://www.benzedrine.cx/relaydb.html
... will only make the spammers switch to a distrubuted attack method
using trojaned virus hosts sending 1 mail message at a time.
"switch to"? i don't know where you're getting your spam from, but the
spammers "switched to" that methodology a long long time ago.
The evolution of spam/viruses is astounding and getting quicker all the
time.
which is why many of us have fought so hard against bad solutions. the
baynesian filtering crowd is the worst. it's like treating every illness
with antibiotics... what you end up with is a lot MORE illness in the
medium to longer term, due to antibiotic-immune mutations.
... [OpenBSD] Spamd is really nice for hurting spammers and/or relays
where it can... in their spool.
spammers don't have spools. they don't care if you get their stuff or not,
and they don't retry. that's why greylisting has been so effective -- to
combat it the spammers would have to add the one thing they cannot afford:
"state." see http://www.rhyolite.com/dcc/ for how to get started.
why is 'state' so hard to afford? they already have a list of email
addresses to spam, and they already have compromised boxes -- those are
the big costs for spammers. another byte of state per email address is
cheap (or if you are clever, a single bit stored in the email address
itself, which doesnt cost you anything).
i see greylisting being effective only as long as it doesnt get widely
deployed. as soon as greylisting starts having any impact on spammers,
they'll start spooling -- and it is very cheap to do so. after all, just
about everything on compromised boxes costs them nothing. and compromised
are the source of 99.9999999% of all spam.
-Dan
this is actually not so much about spam as it is about security models.
> that's why greylisting has been so effective -- to combat it the
> spammers would have to add the one thing they cannot afford: "state."
> see http://www.rhyolite.com/dcc/ for how to get started.why is 'state' so hard to afford? they already have a list of email
addresses to spam, and they already have compromised boxes -- those are
the big costs for spammers. another byte of state per email address is
cheap (or if you are clever, a single bit stored in the email address
itself, which doesnt cost you anything).
that presumes a definite system wherein a spammer knows who he has sent
what to. as if they felt it was nec'y to send only one copy of a spam
to each person, or indeed, as if they had any records of what addresses
bounce, what addresses (or servers) lead to quicksand, or whatever. it
is difficult for the average professional engineer to comprehend, down
in their bones, how little attention a spammer can afford to pay to any
one server or address.
i see greylisting being effective only as long as it doesnt get widely
deployed. as soon as greylisting starts having any impact on spammers,
they'll start spooling -- and it is very cheap to do so. after all, just
about everything on compromised boxes costs them nothing. and compromised
are the source of 99.9999999% of all spam.
i half agree. any technique that pinches a spammer's success rate (which
means, the rate at which they hit blind trap addresses monitored by their
customers) will be cause for attention. this is information warfare, and
there's an effort budget on both sides, asymmetric though it damnably is.
however, "they'll start spooling" is simplistic. the compromised middle-
boxes don't have state -- nothing gets written to disk. these are not
mail relays, but rather, deliberately open proxies. if state were kept
it would be (a) evidence to be used against the spammer, and (b) cause for
the box-owner to notice the activity and perhaps scrape their malware.
the endgame for greylisting is the same as for every moderately successful
"antispam" technique. there will be a three-way schism. (1) some spammers
won't notice or won't care that their success rate drops, and they'll
eventually upgrade their spamware when somebody else improves it. (2) some
spammers will explore ways to keep state and do the retries necessary to
get past the greylist filters. (3) some spammers will just send everything
to everybody every 30 minutes, no matter whether the last response code was
4xx or 5xx. all three will make themselves easier to triangulate upon, and
the conviction rate will edge upward slightly.
(the things spammers do to avoid brightmail and DCC smell really strong --
there's no mistaking that kind of zwil for honest e-mail, even robotically.)
which is why many of us have fought so hard against bad solutions. the
baynesian filtering crowd is the worst. it's like treating every illness
with antibiotics... what you end up with is a lot MORE illness in the
medium to longer term, due to antibiotic-immune mutations.
Any content analysis that can be done to classify content can
be defeated by building a content generator that builds content
to meet the criteria of the analysis tool.
To succeed against the spammers we need to IGNORE the content
and target the behaviors. Why does your mail server accept
incoming email from unknown and unauthenticated sources?
Why does your mail server allow your customers to relay
more than a few messages a day without special permission?
If we change the email system architecture so that every
node in the architecture only accepts mail from known
sources and enforces a pre-arranged behavior pattern,
then we can clean up this mess. It makes sense to accept
large quantities of inbound email from aol.com, i.e. the
large flows are acceptable behavior in that instance. But
it does not make sense to accept large quantities of inbound
email from a broadband customer. It also does not make sense
for a site with a large inflow of mail to accept mail from
all and sundry. It does make sense for them to prearrange
mail exchange with their 100 or 200 or 1000 largest mail
exchange partners and force everyone else to deal with
one of those partners. This is called adding hierarchy to
build a scalable solution. It's called spreading the load.
It's called sharing the pain. And it is not something that
can be avoided as we now see. Everyone is sharing the pain
alright but it is the spammers who are choosing how much
and when it hurts. I guess it's a case of pay me now,
or pay me later...
--Michael Dillon
If the behaviours were easy to identify, there would be no spam.
My mail server accepts incoming e-mail from unknown and unauthenticated sources (a) because there is no widely-deployed mechanism to recognise or authenticate sources such that good ones can be distinguished from bad ones and (b) because the same sources are frequently responsible for sending spam and non-spam.
How do you distinguish between a home user sending twenty legitimate, real messages per day, and a home user whose PC has been 0wned, and which is sending twenty illegitimate messages per day?
The behaviours will adapt to defeat any attempt at classification. The content is the only thing which reliably identifies messages as spam, and the only way to classify the content with high confidence is to have the recipient read it and decide whether she is glad she received it.
I have now exceeded my self-imposed mailing list threshold of 0 messages about spam per month.
Joe
Back of the envelope handwaving calculation (we're not worrying about
exact numbers, merely having somewhere near the right number of zeros):
Media reported that Hotmail was rejecting 2 billion pieces of mail a day (and
that's not including AOL, Yahoo, and every single smaller ISP - our site alone
is seeing several million a day). Let's say it adds up to 10 billion across the
board.
Let's assume that 75% of spam is sent via hijacked zombie machines. This
would mean that to get 7.5 billion spams/day at 20 msgs/day/zombie,
you'd need several hundred million compromised machines. And even though
the average machine is woefully insecure, there's not THAT many zombies.
On the other hand, 20K msgs/day/zombie is only about 1 ever 4 seconds,
not enough to make the average cablemodem user notice - and reduces the
number of zombies down to several million - a much more plausible number.
If you rate-limit 2 million compromised machines to 20 msgs/day each,
there's only 400 million spams. Total.
If you rate-limit 2 million compromised machines to 20 msgs/day each,
there's only 400 million spams. Total.
IF you can rate-limit them across the whole Internet, If you limit 2 million machines to 20 msgs/day per mail server you are back up to your 10 Billion msgs/day mark. This is where DCC or other distributed checksum systems come into play.
-Matt
My point was that there's no real *need* to distinguish between a legitimate
user sending 20 emails and an 0wned box sending 20 emails, as the distinction
is "legitimate 20 emails" versus "0wned 20K emails".
If I were to only give my users 20 outbound connections/day, there wouldn't be
a per-mail-server issue. Whether I can make such a policy stick is another
question entirely.
> IF you can rate-limit them across the whole Internet, If you limit 2
> million machines to 20 msgs/day per mail server you are back up to your
> 10 Billion msgs/day mark. This is where DCC or other distributed
> checksum systems come into play.My point was that there's no real *need* to distinguish between a legitimate
user sending 20 emails and an 0wned box sending 20 emails, as the distinction
is "legitimate 20 emails" versus "0wned 20K emails".If I were to only give my users 20 outbound connections/day, there wouldn't be
a per-mail-server issue. Whether I can make such a policy stick is another
question entirely.
I sent more than 20 mails in the last hour. Given that I have a local mta
each of those results in a seperate connection attempts to the machine I
use as smart-host. I'm sure I could batch them all up and send them at
once thereby returning to my uucp days, but bleh, that really breaks up
the pattern of back-and-forth communication that we've gotten used to.
There's a bunch of forces pushing in various directions that make email
less usable for me and I assume everyone else... The big one is spam,
restricitive mta behavior is another, and there are others. When my mta
becomes more selective about what senders I choose to accept mail from or
in this case when or how often, then eventually I lose mail from people I
would otherwise have communicated with. That's frustrating becuase it's as
disruptive, if not more so than having a mail box full of crap.
Eventually I suspect I'll be forced to abandon the rfc2821 email system as
a communications tool entirely, and brick myself up in the cellar, but I
actuallly liked it as a tool when it worked.
joelja
OK. Make it 100, or make it "20 by default, user can ask for 100". Or
anything else like that. The *POINT* was that too often, a compromised
end-user machine can send *THOUSANDS* of messages. Not tens. Not
hundreds. Thousands.
Remember - if you're catching 1M spams/day, that means that the spammer
has to have a machines*rate product over 1M spams/day. If there's 10 billion
spams/day total, the total machines*rate product has to be over 10G. And
if there's only several million source machines, that means the rate has to be
in the thousands.
this implies network operators will suddenly find a clue, something which
will never happen. ever.
(well, they sometimes suddenly find clue when it is forced upon them, like
say with a subpoena or search warrant.)
-Dan
Dan Hollis wrote:
If you rate-limit 2 million compromised machines to 20 msgs/day each,
there's only 400 million spams. Total.
this implies network operators will suddenly find a clue, something which will never happen. ever.
Clue is generally available in exchange of money. However it requires a seed of foresight or clue to hire more in. Or a business neccessity and a strike of luck. Clue only pays in the long run, so todays quarter-capitalism does not promote clue.
Pete
Death of the Internet Predicted. Film at 11.
Note that *no* anti-spam solution will work unless the network operators
have enough clue to deploy it, be it rate limiting, extensions to SMTP,
a replacement for SMTP, or what have you.
The only reason network operators don't have clues is because there's been
little to no evolutionary pressure to acquire same. At some point, there *will*
have to be a weeding out....
So we need to make it expensive to avoid clue.
-Dan
Although I fail not to sound like a broken record here;
Say hello to the X.400 guys when you meet them...
Pete