Problems sending mail to yahoo?

Mailman List Mirror (Read Only)
1

Joe Greco wrote:

So it's a vast sea of security by obscurity and standards be damned.
It's a real and serious failure of the IETF et al.

...
Having nearly given up in disgust on trying to devise workable anti-spam
solutions that would reliably deliver requested/desired mail to my own
mailbox, I came to the realization that the real problem with the e-mail
system is so fundamental that there's no trivial way to "save" it.

Sounds like the party line inside Yahoo, but there are plenty of ISPs that
do a really good job of combating spam. They do it with standard tools
like RBLs, Spamassassin, OCR, ClamAV and without ineffective diversions
like SPF or DKIM.

Add a few local customizations (I know, this is the time consuming part),
IP-layer IDS, stir carefully and voila, spam to real mail ratios well below
1 to 100. All without big junk folders, with very rare false positives,
and little or no effort on the part of end-users.

The problem is that it is an art, not well documented (without reading
5 or 6 sendmail/postfix and anti-spam mailing lists for a several years),
is not taught in school (unlike systems and network administration), and
rarely gets measured with decent metrics.

Not that spam really has much to do with network operations, well, except
perhaps for those pesky Netcool/Openview/Nagios alerts...

Roger Marquis

2

Sounds like the party line inside Yahoo, but there are plenty of ISPs that
do a really good job of combating spam. They do it with standard tools
like RBLs, Spamassassin, OCR, ClamAV and without ineffective diversions
like SPF or DKIM.

Unless you have actually implemented filters on production mail
platforms with several million users.. please.

Not that spam really has much to do with network operations, well, except
perhaps for those pesky Netcool/Openview/Nagios alerts...

You havent been sitting in on most of the security related talks and
bofs at *nog, right? If you have, that'd be a surprisingly naïve
statement.

srs

3

Roger Marquis wrote:

Sounds like the party line inside Yahoo, but there are plenty of ISPs that
do a really good job of combating spam. They do it with standard tools
like RBLs, Spamassassin, OCR, ClamAV and without ineffective diversions
like SPF or DKIM.

Seen from inside, it is not spamfilters but it is the routing table.
I have seen spam dropping by 98% when zerorouting some networks.

Nobody complained about false positives :slight_smile:

But this is another story for the big ones. They might have customers.

The problem is that it is an art, not well documented (without reading
5 or 6 sendmail/postfix and anti-spam mailing lists for a several years),
is not taught in school (unlike systems and network administration), and
rarely gets measured with decent metrics.

That is true. Plus the rules are constantly changeing.

Not that spam really has much to do with network operations, well, except
perhaps for those pesky Netcool/Openview/Nagios alerts...

At the edge it does. It can bring your VoIP down and video on demand.

I know from campus networks who improved p2p service when zerorouting
networks known for sending spam.

Peter

4

I realize it's natural and predictable, when spam is mentioned, to
repeat the folklore...then the robots came and we were all driven
underground to survive...

However my point was something more in the realm of standards and
operations and what we can do rather than going back over what we
can't seem to do.

For example, and it's only an example don't quibble the example,
defining a list of return SMTP codes which are actually specific and
meaningful like (let's assume they should be 5xx, maybe 7xx would be a
better start? Policy failure codes)

    540 Sending site in internal blacklist contact: URL or MAILBOX
    541 Sending site is in external blacklist: URL
    542 FROM address blocked: MAILBOX
    543 RCPT address blocked: MAILBOX
    544 BODY contained blacklisted URL or MAILBOX: URL or MAILBOX
    545 BODY contained blacklisted string not a URL or MAILBOX
    546 SUBJECT contained blacklisted URL or MAILBOX: URL or MAILBOX
    547 SUBJECT contained blacklisted string not a URL or MAILBOX
    548 SPF Failure (note: could be subsetted further or detail code added)
    549 DKIM Failure (note: could be subsetted further or detail code added)

and so on, a taxonomy which could then at least be dealt with
intelligently by sending MTAs and supporting software rather than each
side cooking up their own stuff.

That's the first problem with this yahoo flap, right? You have to go
to the backed up mail queues and stare at them and try to pattern
match that a lot of these are from yahoo, and oh look they're
deferred?, wait, inside the queue files you can find this "421
Deferred due to user complaints see URL" which then leads you to a
form to fill out and you're still not sure what exactly you're
pursuing other than hoping you can make it go away either by your
action or theirs.

Gak, there isn't even a standard code which means MAILBOX FULL or
ACCOUNT NOT RECEIVING MAIL other than MAILBOX FULL, maybe by choice,
maybe non-payment, as specific as a site is comfortable with.

That's what I mean by standards and at least trying to focus on what
can be done rather than the endless retelling of what can't be done.

More specific and standardized SMTP failure codes are just one example
but I think they illustrate the point I'm trying to make.

Oh yeah here's another (ok maybe somewhere this is written down), how
about agreeing on contact mailboxes like we did with
postmaster@domain?

Is it abuse@domain or spam@domain or support@domain or
postmaster@domain (very commonly used) or ???@domain. Who cares? But
let's pick ONE, stuff it in an RFC or BCP and try to get each other to
conform to it.

5

abuse@domain is *already* specified (in RFC 2142).

Granted, separating reports of email abuse from those for other forms of abuse might be useful for large providers, but since we can't even get many domains even to set up the already-specified abuse@ address, much less read the mail we send to it, I'm not convinced that it would help. OTOH, many email providers seem to think it's my job to know what their internal organization is and re-route email to some spam-specific email reporting address. While that is just rude and ignorant behavior in my book, at least having a single standardized address would be an improvement...

6

Thank you. Perhaps that's why I prefaced that paragraph with:

  Oh yeah here's another (ok maybe somewhere this is written down), how
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  about agreeing on contact mailboxes like we did with
  postmaster@domain?

but you for some reason elided it.

Well, difficult to resist quibbling an example I suppose.

7

of abuse might be useful for large providers, but since we can't even
get many domains even to set up the already-specified abuse@ address, much less read the mail we send to it,

When someone like AOL offloads their user complaints of spams to all the abuse@ addresses instead of verifying that they actually are spams before sending off complaints, is it any surprise that everyone else is refusing to do their jobs for them?

The reason abuse@ addresses are useless is because what is being sent to them is useless.

George Roettger
Netlink Services

8

of abuse might be useful for large providers, but since we can't even
get many domains even to set up the already-specified abuse@ address, much less read the mail we send to it,

When someone like AOL offloads their user complaints of spams to all the abuse@ addresses instead of verifying that they actually are spams before sending off complaints, is it any surprise that everyone else is refusing to do their jobs for them?

I'm not sure I know what you mean. Are you talking about the optional feedback loop? When I was signed up for that I did get a bunch of bogus reports, but other than that I've never received a spam report from AOL at all.

The reason abuse@ addresses are useless is because what is being sent to them is useless.

I'm sure that a lot of useless reports come in--my servers never originate spam, but we still get the occasional bogus report due to forged headers. At the same time, I certainly send dozens of real spam reports every day and they all contain actionable information (that would be supplemented further if an actual human were to ask). What I've found is that "too big to fail" ISPs respond (if they accept the email at all!) with either an automated response or a canned response from a help desk monkey who is actually wrong close to half the time, while many boutique providers and most US-based .edu sites respond personally and cluefully. (Don't get me started about the US government, especially the military...)

My conclusion is that the problem is not crappy reports but rather under-investment in clue at big ISP help desks. All the fancy standards and tools in the world are not going to help this basic problem: stemming the tide of abuse from their networks is simply not a high enough priority for companies like Yahoo, Hotmail, AT&T, et al. Until they start losing money every time spam leaves their network, I don't see their behavior changing.

9

As one that works for a company that makes full use of complaints sent to it,
abuse@ addresses are not useless, far from it. Please don't get the idea that
because some think they're useless, it therefore is universal. We also get
100s of AOL feedbacks a day, which are filtered separately. Also not useless.
And we've also reported incidents to other companies' abuse functions, and had
them be resolved same-day because of it. Also, far from useless.

How about if you're not actively in an abuse function, you hold off on declaring
the function useless, cause the meme could catch on that it is, even if it's
not, and I've yet to see an automated filtering/blocking system fully replace or
completely obsolete a good trained network operator who understands what is and
is not abuse on the network.

-Dave D

10

I agree that they aren't completely useless. From our environment the abuse desks can be somewhat overwhelmed though. If you setup feedback loops for networks size of
1x /16
2x /17
2x /18
1x /19
to receive abuse complaints on dedicated / collocated customers you do get a some good complaints. Some of the time it is from compromised scripts, sometimes actual spammers, but most of the time it is from forwarded spam. This makes the abuse desk full of thousands and thousands of complaints. You can look in the headers of the spam complaints and see that it is forwarded spam, but it is still overhead. So signing up for a feedback loop for the entire network with something like Yahoo! can be burdensome and make abuse@ full of useless complaints. This isn't the problem I suppose in most environments, but it is in mine. Yahoo! blocking entire /24's are not necessarily a large problem, the larger problem is

A. they don't tell you when it is blocked (I don't believe it would be hard to email the abuse@ contact of the IP address range..)

B. their 'Bulk Mail Advocates' say they cannot tell what IP's are generating the /24 block once it is in place (perhaps it can be prior to the block?).

C. They offer no way to exempt certain IP addresses to be exempted from the /24 'de-prioritization'. This means the smaller companies who send maybe 3 or 4 emails to Yahoo a day are having difficulty and there's nothing you can do until the issue with the entire /24 is solved.

Administrators who actually find ways to get in touch with Yahoo to resolve issues are hindered by Yahoo's stance of 'It's coming from your network, you should be able to monitor it and figure it out'. In a dedicated/colo environment I don't think it is really reasonable to expect companies login to each server in a /24 to see who is sending mail to Yahoo. And even if they are sending mail to Yahoo were not psychic so we cannot tell what their users are marking as spam and what's not. I suppose the feedback loop would say that but...then abuse@ is flooded with complaints that are mostly mutual customers fault. Chances are if a server is sending spam to Yahoo they are sending it to quite a few other places as well which do actively report it.

-Ray

11

1. They are not complaints as such. They are what AOL users click report spam on

2. They are sent in a standard format - http://www.mipassoc.org/arf/ -
and if you weed out the obvious (separate forwarding traffic out
through another IP, and ditto for bounce traffic), then you will find
that - for actual ISPs - actual spam reports will far outweigh the
amount of misclicked reports.

3. As I said, its in ARF and that's machine parseable and you can get
stats from it.

12

See RFC 3463. Unfortunately it doesn't help much to solve the problem it
was designed for. I ranted about this a while back...
http://www.ietf.org/mail-archive/web/ima/current/msg00588.html

Tony.