Time to check the rate limits on your mail servers

CNET reports
http://news.com.com/Zombie+trick+expected+to+send+spam+sky-high/2100-7349_3-5560664.html?tag=cd.top
that botnets are now routing their mail traffic through the local
ISP's mail servers rather than trying their own port 25
connections.

Do you let your customers send an unlimited number of
emails per day? Per hour? Per minute? If so, then why?

--Michael Dillon

http://news.com.com/Zombie+trick+expected+to+send+spam+sky-high/2100-7349_3-5560664.html?tag=cd.top

that botnets are now routing their mail traffic through the local
ISP's mail servers rather than trying their own port 25
connections.

Now? We (and AOL, and some other large networks) have been seeing
this thing go on since over a year.

Do you let your customers send an unlimited number of
emails per day? Per hour? Per minute? If so, then why?

Doing that - especially now when this article has hit the popular
press and there's going to be lots more people doing the same thing -
is going to be equivalent of hanging out a "block my email" sign.

One additional thing that I think wasnt mentioned in the article -
Make sure your MXs (inbound servers) are separate from your outbound
machines, and that the MX servers dont relay email for your dynamic IP
netblock. Some other trojans do stuff like getting the ppp domain name
/ rDNS name of the assigned IP etc and then "nslookup -q=mx
domain.com", then set itself up so that all its payloads get delivered
out of the domain's MX servers

Hi!

http://news.com.com/Zombie+trick+expected+to+send+spam+sky-high/2100-7349_3-5560664.html?tag=cd.top

that botnets are now routing their mail traffic through the local
ISP's mail servers rather than trying their own port 25
connections.

Now? We (and AOL, and some other large networks) have been seeing
this thing go on since over a year.

Indeed, we also see this a long time now. Most of them specific spamruns towards the bigger players... (AOL alike).

Do you let your customers send an unlimited number of
emails per day? Per hour? Per minute? If so, then why?

One additional thing that I think wasnt mentioned in the article -
Make sure your MXs (inbound servers) are separate from your outbound
machines, and that the MX servers dont relay email for your dynamic IP
netblock. Some other trojans do stuff like getting the ppp domain name
/ rDNS name of the assigned IP etc and then "nslookup -q=mx
domain.com", then set itself up so that all its payloads get delivered
out of the domain's MX servers

So the next article would say 'lets now all seperate MX and SMTP servers' still a LOT of large players combining those two. Giving troyans doing the above scenario a open door.

Bye,
Raymond.

http://news.com.com/Zombie+trick+expected+to+send+spam+sky-high/2100-7349_3-5560664.html?tag=cd.top
> that botnets are now routing their mail traffic through the local
> ISP's mail servers rather than trying their own port 25
> connections.

Now? We (and AOL, and some other large networks) have been seeing
this thing go on since over a year.

> Do you let your customers send an unlimited number of
> emails per day? Per hour? Per minute? If so, then why?

Doing that - especially now when this article has hit the popular
press and there's going to be lots more people doing the same thing -
is going to be equivalent of hanging out a "block my email" sign.

I just implemented a patch to tcpserver which allows me to limit the
number of simultaneous SMTP connections from any one IP, but have not yet
looked into daily/hourly limits. I know Comcast has started limiting
residential customers to 50-100 emails per day, and that customers with
legitimate reasons for using more than that are starting to complain.

One additional thing that I think wasnt mentioned in the article -
Make sure your MXs (inbound servers) are separate from your outbound
machines, and that the MX servers dont relay email for your dynamic IP
netblock. Some other trojans do stuff like getting the ppp domain name
/ rDNS name of the assigned IP etc and then "nslookup -q=mx
domain.com", then set itself up so that all its payloads get delivered
out of the domain's MX servers

Easier said than done, especially if you're a small ISP that's been doing
POP before SMTP and changing this requires that every customer's settings
be changed.

Is there any info on how this zombie is spread? ie, email worms, direct
port attacks, etc. If the former, there's hope of nipping it in the bud
with anti-virus filtering.

James Smallacombe PlantageNet, Inc. CEO and Janitor
up@3.am http://3.am

There is one mistatement in this article, though: the author says:

        "This means the junk mail appears to come from the ISP [...]"

If it's coming from their servers (or their network), it IS coming
from the ISP, and they bear full responsibility for making it stop.

---Rsk

One additional thing that I think wasnt mentioned in the article -
Make sure your MXs (inbound servers) are separate from your outbound
machines, and that the MX servers dont relay email for your dynamic IP
netblock. Some other trojans do stuff like getting the ppp domain name
/ rDNS name of the assigned IP etc and then "nslookup -q=mx
domain.com", then set itself up so that all its payloads get delivered
out of the domain's MX servers

Easier said than done, especially if you're a small ISP that's been doing
POP before SMTP and changing this requires that every customer's settings
be changed.

IMHO, if you are a small ISP and limit the # of e-mails per user per day, even to something like 1K, you probably don't have to separate the MX & SMTP servers. But that's me, others might still think you were being "irresponsible".

Is there any info on how this zombie is spread? ie, email worms, direct
port attacks, etc. If the former, there's hope of nipping it in the bud
with anti-virus filtering.

All of the above.

> Do you let your customers send an unlimited number of
> emails per day? Per hour? Per minute? If so, then why?

Doing that - especially now when this article has hit the popular
press and there's going to be lots more people doing the same thing -
is going to be equivalent of hanging out a "block my email" sign.

I don't understand your comment. This is an
arms race. The spammers and botnet builders
are attempting to make their bots use the
exact same email transmission channels as
your customers' email clients. They are
getting better at doing this as time goes
on. I think we are at the point where the
technical expertise of the botnet builders
is greater than the technical expertise of
most people working in email operations.

We cannot win this battle by continuing to
attempt to trump their technical abilities.
However, if we shift the battleground to
a location where network operators have the
upper hand, we can do better.

And that's why I suggest that people should
start looking at email volume controls. The
vast majority of individual users only send
a small number of emails over a given time
period whether you measure that time period
in minutes, hours or days.

SPAM is a form of DDoS against the Internet's
email architecture. Rate limiting has proven to
be an effective way of mitigating DDoS because
it strikes at the very core of the DoS methodology.
Why not deploy this strategy against email?

Please note that I am not suggesting that
this is a way to "solve" the SPAM problem.
First of all, I do not agree that there is
a SPAM problem. The fundamental problem is that
the Internet email architecture is flawed. SPAM
is merely a symptom of those flaws. If we fix
the architecture, then nobody will care about
SPAM. As you can see, two separate problems
are becoming intertwingled here. In the past
we had viruses, DDoS, botnets, SPAM, phishing.
But now, all of these things are merging and
evolving together.

And secondly, I'm only pointing out that there
are reasons for people to start thinking about
rate limiting email on their networks. I'm
suggesting that people should be asking questions.
I don't think it is wise to run out and slap
rate limits on mail infrastructure without
thinking through the implications.

--Michael Dillon

Both on ASRG and here on NANOG, many of us said many times, and most of the times people called me crazy;

1. Block port 25 for dynamic ranges - that will kill the current strain of worms.
2. It won't solve spam, and neither will SPF or anything else of the sort, as when you have 100K zombies, you don't need to act a server, you can use the real credentials for the user, and even if limited to a 1000 messages, that times 100K drones is...

The issue is numbers, and how to reduce them, not stop the tide.

Currently there is a discussion of this on Spam-Research [1], quite interesting.

  Gadi.

1 - Spam-Research archives: https://linuxbox.org/cgi-bin/mailman/listinfo/spam

up@3.am wrote:

<snip>

Easier said than done, especially if you're a small ISP that's been doing
POP before SMTP and changing this requires that every customer's settings
be changed.

drac http://mail.cc.umanitoba.ca/drac/
supports seperate pop/smtp servers. Which is not neccessarily what is being recommended by having seperate in-mx-smtp and out-smtp.

That, on the other hand, gets you into trouble with rather stupid Spam
filters, that only accept mails from a server, if that server is also
MX for the senders domain.

Yes, this is stupid, but that does not change the fact, that these
setups are out there.

Nils

Hi!

One additional thing that I think wasnt mentioned in the article -
Make sure your MXs (inbound servers) are separate from your outbound
machines, and that the MX servers dont relay email for your dynamic IP
netblock. Some other trojans do stuff like getting the ppp domain name

That, on the other hand, gets you into trouble with rather stupid Spam
filters, that only accept mails from a server, if that server is also
MX for the senders domain.

Yes, this is stupid, but that does not change the fact, that these
setups are out there.

Start using authenticated SMTP for this.

Bye,
Raymond.

Hi!

CNET reports http://news.com.com/Zombie+trick+expected+to+send+spam+sky-high/2100-7349_3-5560664.html?tag=cd.top
that botnets are now routing their mail traffic through the local
ISP's mail servers rather than trying their own port 25
connections.

Both on ASRG and here on NANOG, many of us said many times, and most of the times people called me crazy;

1. Block port 25 for dynamic ranges - that will kill the current strain of worms.
2. It won't solve spam, and neither will SPF or anything else of the sort, as when you have 100K zombies, you don't need to act a server, you can use the real credentials for the user, and even if limited to a 1000 messages, that times 100K drones is...

Did you actially read the article? This was about drones sending out via its ISP mailserver. Blocking outbound 25 doesnt help a bit here. In general sure, good ide, and also start using submission for example. But in this contect its silly.

Bye,
Raymond.

Did you actially read the article? This was about drones sending out via its ISP mailserver. Blocking outbound 25 doesnt help a bit here. In general sure, good ide, and also start using submission for example. But in this contect its silly.

No, it is relevant or I wouldn't have mentioned it.

Allow me to elaborate; and forget about this article, why limited ourselves?

Once big ISP's started blocking port 25/outbound for dynamic ranges, and it finally begun hitting the news, we once again caused the spammers to under-go evolution.

In this particular case, they figured they'd have to find better ways to send spam out, because eventually, they will be out of working toys.

Using the user's own mail server, whether by.. erm.. just utilizing it if that is possible, sniffing the SMTP credentials or stealing them from a file/registry, maybe even using Outlook to send is all that's about to happen.

heck, I don't see how SMTP auth would help, either. They have local access to the machine.

Now, once 100K zombies can send *only* 1000 spam messages a day instead of 10K or even 500K, it makes a difference, but it is no solution.

I am happy to see people are starting to move this way, and I personally believe that although this is happening (just go and hear what Carl from AOL says on Spam-R that they have been seeing since 2003), this is all a POC. We have not yet begun seeing the action.

Should I once again be stoned, or will others see it my way now that the tide is starting to turn?

  Gadi.

Hi!

Now, once 100K zombies can send *only* 1000 spam messages a day instead of 10K or even 500K, it makes a difference, but it is no solution.

I am happy to see people are starting to move this way, and I personally believe that although this is happening (just go and hear what Carl from AOL says on Spam-R that they have been seeing since 2003), this is all a POC. We have not yet begun seeing the action.

This is no POC, we have seen this happen many many times. Perhaps some drone networks are a little 'behind' but in general, they are perfectly able to do this. Even with some static lists for some large ISPs mailservers they can perfectly initiate it large scale. And yes, it does limit, but with the number of bots we see controlled on the few botnets we monitored the impact will still be hudge.

Should I once again be stoned, or will others see it my way now that the tide is starting to turn?

Its not turning, its happening.

Bye,
Raymond.

This is no POC, we have seen this happen many many times. Perhaps some

Wrong, and I will tell you why in a second.

drone networks are a little 'behind' but in general, they are perfectly able to do this. Even with some static lists for some large ISPs mailservers they can perfectly initiate it large scale. And yes, it does limit, but with the number of bots we see controlled on the few botnets we monitored the impact will still be hudge.

You have been seeing them try it, yes. But why should they use it when they can send 10,000,000,000 spam messages out with no trouble? The answer is because they will soon have to.

As much as some are capable of it, most are not yet there. They will be soon.

This is the first evolutionary step I can see that we pushed the spammers into doing, according to our wishes.

It may be a bigger "attack" on your servers, but it's nothing in comparison to spam messages out there where every available host sends the spam out.

Why SPF won't work? Why it is all useless (SPF, etc.) is because there are 100K and more drone armies out there, but don't kid yourselves - you ain't seen nothing yet.

Should I once again be stoned, or will others see it my way now that the tide is starting to turn?

Its not turning, its happening.

You will know when it's happening. That will be when every spammer will be at the corner and will have to move to this way of working.

Just because you see a POC and some people are either more adavanced or bored to do it, and spam is a massive thing so you feel it, doesn't mean it's a trend.

  Gadi.

Now, once 100K zombies can send *only* 1000 spam messages a day instead
of 10K or even 500K, it makes a difference, but it is no solution.

I'd like to see rate limits set much
lower than that. Perhaps one message per day
to begin with. After the message is sent,
send the customer a reminder about the limit
and tell them how to get to a web page
to increase the limit. The web page would
only accept an incremental increase. For
instance, if your limit is one, you can
bump it up to five per day and that is all.
Then, if you exceed the new limit, you once
again have the opportunity to bump it up
by five more. Most people won't need more
than 10 or 15 per day limits.

People who need more can call their customer
representative and order the volume mail
add-on product. They will have to agree to
a contract that allows you, the operator,
to completely block their net access without
notice if it appears that a bot/virus may
have infected their systems.

I'm sure if you discuss this kind of stuff
with your product development and product
marketing people, they will come up with more
interesting variations.

One message per day is not too low. There are
people who never use email. They just browse
the web and use IM. Why should you, the operator,
allow those customers to inject huge numbers of
email systems into the Internet as botnet drones?
1000 a day is way too high, IMHO.

--Michael Dillon

Hello
I am a bit concerned that blocking any port at all preventing abuse of the affected service will make the abusers go through other services instead. Port 139/445 is already blocked by several isps due to excessive abuse or I believe they call it 'a security measurement'. Even port 23 has been blocked (inbound and outbound) by atleast 1 large isp I am aware of. When that mssql worm was lurking around isps were also blocking that port. I hope I'm not the only one seeing a pattern here. Really, blocking ports makes no sense to me in the long run. You are destroying the service, and even if you block all ports there are several ways to spam anyway. You would probably reply now saying that "yeah but you get rid of 99% of the spammers that way". That is only partly true. As time goes on all spammers will adopt to your isps new "security policy" and if you still don't see the pattern I am talking about now there is nothing more I can say. I don't have the solution to all of this, but I sure know how to see what is not the solution. Teach people how to write "Hello world" better perhaps.

Joergen Hovland
Joergen Hovland ENK

I know that I'm in the middle of trying to figure this out with the mail
server software that is used where I work but if limits are going to be put
into
place per email box of say 1,000 messages per day and a total daily sending
limit of say 200 megabytes, I feel there also needs to be methods in place
for the end-user (customer) to be able to view where they stand in
relationship to their "quota".

Yes this becomes more of something for the "help desk" side of a provider
but as operations, I have to support the "help desk" in being able to give
the user information when they call about the "limits"

David

Hello
I am a bit concerned that blocking any port at all preventing abuse of the affected service will make the abusers go through other services instead. Port 139/445 is already blocked by several isps due to excessive abuse or I believe they call it 'a security measurement'. Even port 23 has been blocked (inbound and outbound) by atleast 1 large isp I am aware of. When that mssql worm was lurking around isps were also blocking that port. I hope I'm not the only one seeing a pattern here. Really, blocking ports makes no sense to me in the long run. You are destroying the service, and even if you block all ports there are several ways to spam anyway. You would probably reply now saying that "yeah but you get rid of 99% of the spammers that way". That is only partly true. As time goes on all spammers will adopt to your isps new "security policy" and if you still don't see the pattern I am talking about now there is nothing more I can say. I don't have the solution to all of this, but I sure know how to see what is not the solution. Teach people how to write "Hello world" better perhaps.

I quite agree, blocking ports is not the best answer, as it is a self-inflicted-DDoS.

Still, please tell me, how is not blocking un-used or un-necessary ports a bad thing? It is a defensive measure much like you'd add barricades before an attack.

The Internet is a war zone, but I don't have to tell the NANOG community that.

Thing is, blocking port 25 won't cause spam to stop, there are no FUSSP solution. Yet, we all recognize that SMTP is far from perfect.

And indeed, as others here are more qualified than me, by far, to tell you, most development in anti-spam technology only helped short-term, and caused the bad guys to evolve. Well, why is blocking port 25 different? See for yourself.

They now evolved, and are using user-credentials and ISP-servers. This evolution means that their capabilities are severely decreased, at least potentially.

This is the best next thing after dark Irish stout and ketchup.

It means ISP's will have to re-think their strategies, just like AOL did. It also means it's once small step to victory for us. We are a long way from it, and please - not everybody blocks port 25 so current-day worms are more than efficient still.

It is nice to see fore thinking and long-term planning with the bad guys, where all we can do is disagree.

  Gadi.

Still, please tell me, how is not blocking un-used or un-necessary ports
a bad thing? It is a defensive measure much like you'd add barricades
before an attack.

Agreed. And depending on your service, there are different ports
worth blocking. For residential users, I can't see a reason to not
block something like Netbios. And blocking port 25 effectively
prevents zombies from spamming. Unfortunately, it also blocks
legitimate users from being able to use SMTP AUTH on a remote server..

They now evolved, and are using user-credentials and ISP-servers. This
evolution means that their capabilities are severely decreased, at least
potentially.

Has this been confirmed? Does this new worm, in fact, use SMTP AUTH
where necessary? Will it also check the port that the user's computer
is set to send mail on? So, for instance, if SMTP AUTH is required,
and the mail submission port is being used rather than standard port
25, will the worm detect all this?

The nice part about SMTP AUTH, though, is that there is at least a
direct link to the user sending the spam. This means, of course, that
ISP's will need to police their users a little better.. :slight_smile:

It means ISP's will have to re-think their strategies, just like AOL
did. It also means it's once small step to victory for us. We are a long
way from it, and please - not everybody blocks port 25 so current-day
worms are more than efficient still.

So I guess users will have to stop clicking that "Save Password"
button... That is, until the worm records the keystrokes when the
password is entered... *sigh*