verizon.net and other email grief

Hi,

trying to pin down why so much email isn't making it recently.

We see issues with various big ISPs.

The most obvious is none of the three UK ISPs I have ready access to can
connect to port 25 on relay.verizon.net. (MX for all the verizon.net email
addresses). We can ping it (I'm sure it isn't singular?), but we have no more
luck delivering email than contacting verizon technical staff, logs suggests
we are in day 3 of this. I'm now listening to hold music at International
rates - ouch.

Have a support call with GXN open on connectivity to hotmail, which seems to
vary with UK ISP, but is in general pretty poor from all of them.

In general I'd say off topic in Nanog, but there comes a point where you start
wondering if the routing is messed up.

Am I alone before posting more details.

  Simon

I think I can shine a little bit of light on what might be your
Verizon problem.

Summary:

Verizon has put in place an exceedingly stupid "anti-spam" system which
does not work, which facilitates DoS attacks, and which provides active
assistance to spammers. Verizon has been told all of this, and it's
been discussed on Spam-L. If there's been a response from Verizon,
I haven't seen it: and AFAIK the practice continues. Anyone trying to
deliver mail there might want to at least skim this to get an idea of
the issues they may bump into. Please note that in places this is
sketchy because it seems impossible to get Verizon to provide the
information necessary to make it otherwise (or correct any errors).

Details:

When an incoming SMTP connection is made to one of Verizon's MX's, they
allow it to proceed until the putative sender is specified, i.e. they
wait for this part of the SMTP transaction:

  MAIL From:<blah@example.com>

Then they pause the incoming connection. And then they start up an
outbound SMTP connection from somewhere else on Verizon's network, back
to one of the MX's for example.com. They then attempt to verify that
"blah" is a valid, deliverable address there. Since most people have
long since disabled SMTP VRFY, they actually construct a fake message
and attempt delivery with RCPT. If delivery looks like it's going to
succeed, they hang up this connection (which is rude), and un-pause
the incoming one, and allow it to proceed. If delivery looks like
it's going to fail, then they also hang up their outbound connection
(still rude), un-pause the incoming one, and reject the traffic.

This also means that if the MX they try to connect to is (a) busy
(b) down (c) unaware of all the deliverable addresses (d) something else,
that they'll refuse the incoming message.

It also means that if the address that's trying to send mail to Verizon
is something like "support@thuleracks.com", which is the address that
the people at Thule Racks emit support traffic from, but which doesn't
accept traffic, that Verizon will deny the message. (Yeah, this isn't
very bright on Thule's part, either.)

Whoops.

This is bad for a whole bunch of reasons: two of the more obvious ones
are (a) it's a pathetic "anti-spam" measure because ANY forged address
ANYWHERE will do, and (b) it doesn't scale. Add to that (c) it abuses
RCPT because apparently Verizon is unwilling to use VRFY and to accept
the decision of many mail server operators to disable it. Oh, and (d) the
behavior of their probe systems is nearly indistinguishable from that of
spam-spewing zombies, which don't obey the SMTP protocol either.

  [ (b) is also how it lends itself to DoS attacks. Sure, Verizon
  could rate-limit the rate at which they make outbound connections,
  but then attacker X could impose significant delay on mail
  from domain Y just by forging a boatload of messages purporting
  to be from addresses in Y to Verizon. If Verizon rate-limits
  their outbound connections, then any real messages from Y will be
  stuck in the verification queue along with a kazillion forgeries.

  And beyond that: other people are foolishly adopting this
  "callback" nonsense as well. Slashdot carried a note the other
  day about a program _designed_ to do this. This allows attacker
  X to forge messages from domain Y to idiots I1, I2... In, for
  a very large "n", and then stand back as all of them simultaneously
  try to connect to the MX's for domain Y.

  General principle: any "anti-spam" measure that generates more
  junk SMTP traffic at a time when we're drowning in it is probably
  a bad idea. ]

One thing that's not clear is whether or not Verizon caches any of
this information. Doing so might help cut down on DoS attack methods
that involve them, but of course it doesn't do anything about those
which leverage everyone else who's doing "callbacks".

And this is unfortunately, not the end of it.

A lot of people, including me, are blocking particularly problematic
spammer-controlled networks at (a) our border routers (b) our firewalls
or (c) our mail servers. In other words, we not only won't accept mail
from them, we won't even allow them to connect: we're blocking all IP
traffic from them. This prevents them from spamming (at least directly
from their own network space); it also prevents them from using their
resources to build lists of deliverable addresses to sell to other
spammers by poking at our mail servers.

Since Verizon is doing this probing *from their network*, spammers can
easily get around all of our blocking by getting Verizon to do it for
them. They can thus use Verizon to build/check their lists...and there's
no way for us to find out who's on the other side of these probes.

Which means that Verizon is running a free, anonymizing, spam support
service.

Note that refusing Verizon's probes, either accidentally or deliberately,
will of course cause Verizon to deny incoming mail. Which I suppose might
be okay for some folks, it isn't workable here.

So for now all we can do is explain to our users that it's causing problems
and deal with it. But if you look in your MTA logs and find things like:

Jul 15 07:24:51 <XXXXXXXXXXXXXXXXX@gsp.org>... User unknown
Jul 15 07:24:51 lost input channel from sc014pub.verizon.net [206.46.170.58] to MTA after rcpt
Jul 15 07:24:51 from=<antispam332078@west.verizon.net>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc014pub.verizon.net [206.46.170.58]

then you can see what they're doing.

---Rsk

While I can't speak to what Verizon is using, Both Exim and Postfix have the
very same feature called "address verification". Its in use at a number of
ISPs. My systems reject 1000's of messages every day because of
verification failures.

Roy Engehausen

That would be 1000's of other people's servers getting traffic from you
because someone forged their address in the spam. You are effectively
doubleing the total load spam places on the net.

This doesn't scale.

While I can't speak to what Verizon is using, Both Exim and Postfix have

the

very same feature called "address verification". Its in use at a number

of

ISPs. My systems reject 1000's of messages every day because of
verification failures.

i've never seen this done with postfix, but i know that exim's default
'address verification' for non-local addresses just checks that the domain
in the from is valid and that an mx record exists for it. they also have
what they call 'callout verification', which is equivalent to what is being
discussed, but the documentation makes the drawbacks painfully clear and
suggests that it only be used against hosts within the same organization.
i'm not a fan of exim, but it appears that although they've given users the
rope, they've been diligent enough to label it appropriately.

-p

It appears that they do some amount of caching.

      -Jeff

Christopher X. Candreva wrote:

That would be 1000's of other people's servers getting traffic from you because someone forged their address in the spam. You are effectively doubleing the total load spam places on the net.

This doesn't scale.

How so? Are you maybe confusing reject with bounce? If address verification takes place while the SMTP connection is still up, no forged adresses get messaged, at least not by the server doing the rejecting.

Verizon has put in place an exceedingly stupid "anti-spam" system which
does not work, which facilitates DoS attacks, and which provides active
assistance to spammers.

The technique discussed is called callback verification and I do not
agree that the technique stupid or provides assistance to spammers.
I do agree that some of the aspects in how this was implemented by
Verizon is not correct and causing problems.

Details:

When an incoming SMTP connection is made to one of Verizon's MX's, they
allow it to proceed until the putative sender is specified, i.e. they
wait for this part of the SMTP transaction:

  MAIL From:<blah@example.com>

Then they pause the incoming connection. And then they start up an
outbound SMTP connection from somewhere else on Verizon's network, back
to one of the MX's for example.com. They then attempt to verify that
"blah" is a valid, deliverable address there. Since most people have
long since disabled SMTP VRFY, they actually construct a fake message
and attempt delivery with RCPT. If delivery looks like it's going to
succeed, they hang up this connection (which is rude)

If I were them I'd do RSET first and then QUIT instead of direct hangup

, and un-pause the incoming one, and allow it to proceed. If delivery
looks like it's going to fail, then they also hang up their outbound
connection (still rude), un-pause the incoming one, and reject the
traffic.

Rejection of traffic is not approriate in this case, the greylisting
technique should instead be used and they should reply back with 4xx
error code (most likely 421 or 451), this allows for retry of email
later on instead of outright denial.

This also means that if the MX they try to connect to is (a) busy
(b) down (c) unaware of all the deliverable addresses (d) something else,
that they'll refuse the incoming message.

All good reasons explaining why error code should 4xx instead of 5xx as
I noted above.

It also means that if the address that's trying to send mail to Verizon
is something like "support@thuleracks.com", which is the address that
the people at Thule Racks emit support traffic from, but which doesn't
accept traffic, that Verizon will deny the message. (Yeah, this isn't
very bright on Thule's part, either.)

They are correct in this case. The address entered in RFC2821 MAIL FROM
is "Bounces-To" address and it must accept bounced email and as such it
must accept incoming emails. If the address does not accept traffic as
you indicated should not be used in MAIL FROM and different adddress from
local machine should be used. Please read email RFCs and then you'll
understand that Envelope MAIL FROM (despite its name) is not an address
that indicate source of email, the source of email address is actually
"From:" header (or in some cases Sender header - which one to use will
depend on your situation)

This is bad for a whole bunch of reasons: two of the more obvious ones
are (a) it's a pathetic "anti-spam" measure because ANY forged address
ANYWHERE will do

I agree that it requires verification that connecting machine has right
to use given mail-from address as part of its sending email. See SPF.

But for current situation it does work just fine and causes number of
emails with randomly generated emails to be stopped.

, and (b) it doesn't scale.

The scalability depends on implementation. Since we have Verizon
implementing it, I'm guessing it scales just fine based on the size
of their email network.

I'll also note that a lot of email is sent from the same address
and as long as they cache results of verification it'll not be
necessary to do verification each and every time.

Add to that (c) it abuses
RCPT because apparently Verizon is unwilling to use VRFY and to accept
the decision of many mail server operators to disable it. Oh, and (d) the
behavior of their probe systems is nearly indistinguishable from that of
spam-spewing zombies, which don't obey the SMTP protocol either.

Sometimes spammers probe various address before sending email to compile
new list of email addresses and they use similar technique, however those
are in fact distinguishable because you'll see them probing multiple
non-existing address usually in alphabetical order. If email is sent
from real working addresses, the callback verification would not look
like a spammer probing technique.

  [ (b) is also how it lends itself to DoS attacks. Sure, Verizon
  could rate-limit the rate at which they make outbound connections,
  but then attacker X could impose significant delay on mail
  from domain Y just by forging a boatload of messages purporting
  to be from addresses in Y to Verizon. If Verizon rate-limits
  their outbound connections, then any real messages from Y will be
  stuck in the verification queue along with a kazillion forgeries.

As with many techniques trying to address problems of current email
infrastructure with temporary solutions, it has weaknesses that made
those using vulnerable to specialized DoS attacks. This is nothing new
and that is why IETF requires "Security Considerations" section for
every draft. Obviously addressing possible security weaknesses is
a task for implementor of the technique and as you point out Verizon
could use rate-limiting to protect against ongoing attack.

I'll also note that rate-limiting is used to protect many kind of attacks
(including ip and web DoS) and in all cases it causes some legitimate
requests to be delayed along with large number of attacks sources. This
is considered to be acceptable temporary situation in order to protect
against larger problem.

  General principle: any "anti-spam" measure that generates more
  junk SMTP traffic at a time when we're drowning in it is probably
  a bad idea. ]

Callback verification if properly implemented will never generate more
junk SMTP traffic as DATA part of SMTP transmission never happens.

oh, so you would be ok with someone joe-jobbing you on their 1 million
messages/day spam run and getting 1 million 'verification' connections to
your mailserver farm?

-p

The other part is that you CACHE the answer you get (good, bad, or indifferent). I think that SPF+sender address verification is a GOOD thing when properly implemented. Yes it can be a bit of a hassle, but you shouldn't be sending mail you're not prepared to bounce.

That said, none of my sites are running a current enough version of Postfix to do this.

Far less traffic than the bounces would create at both ends. Yes this doesn't prevent it from happening if the address is real, but that's why I mentioned SPF in my previous email......That helps to verify the sender can send email for a given domain, and if that passes, then you want to see if the sender exists, if both pass then you can go on to other methods. OF course I'd first check blacklists before any of this, but that's my personal preference.

[...] they also have what they call 'callout verification', which is
equivalent to what is being discussed, but the documentation makes
the drawbacks painfully clear and suggests that it only be used
against hosts within the same organization.

No, that caveat is given for *recipient callforward verification*
which is dangerous if turned on blindly. I know, I tried it for a very
short while :slight_smile:

i'm not a fan of exim, but it appears that although they've given
users the rope, they've been diligent enough to label it
appropriately.

Sender callback verification is a different beast and is highly
effective against spam. It does of course not come without its price
of false positives caused by misconfigured senders. Unlike other
mechanisms, it at least doesn't inconvenience senders who haven't
botched their mail system.

The only false positives I see are things like web sites that mail
from a webserver role account which doesn't have a mailbox. Even so,
ecommerce sites are learning to not do this, and ordered goods usually
turn up regardless of whether or not an automatically-generated
confirmation email arrives.

That is already what happens when spammer forged your address - you see
1000's people sending you bounces and nastygrams. The real solution is
to use some form of protection for envelope mail-from address so that
it could not be so easily spoofed and forged. There are currently
several proposals on the table on how to do it and some of the proposals
are already being used on the internet in experimental ways:

SPF (dns records listing ips of mail systems that can send mail with given
      envelope mail-from domain).
  For more information see:
     http://spf.pobox.com
     http://www.openspf.org
     http://www.spfhelp.org

CSV with MPR records (similar SPF but provides list of mail-server hostnames
     that can use MAIL-FROM domain, the spoofing of mail-server names is
     protected based on EHLO by CSV):
  For more information see:
     http://www.csvmail.com/draft-otis-marid-mpr-00.html
     (and for CSV see http://mipassoc.org/csv/index.html)

BATV (replacement of your real mail-from address with special private
       connection-specific address - this allows to /dev/null bad
       bounces if they come back to you and you did not send the email).
  For more information see:
     http://mipassoc.org/batv/index.html

SES (predates BATV and similar technique, except that a HMAC
       encrypted address can confirmed by means of public server
       which allows email to be dropped at recepient instead of
       dropped at the source as being bad bounce as with BATV).
  For more information see:
     http://ses.codeshare.ca/

> Verizon has put in place an exceedingly stupid "anti-spam" system which
> does not work, which facilitates DoS attacks, and which provides active
> assistance to spammers.

The technique discussed is called callback verification and I do not
agree that the technique stupid or provides assistance to spammers.
I do agree that some of the aspects in how this was implemented by
Verizon is not correct and causing problems.

<snip>

But for current situation it does work just fine and causes number of
emails with randomly generated emails to be stopped.

Erm. Yeah, it stops them from being delivered to Verizon by shifting
half the cost of verification onto the victims.

>, and (b) it doesn't scale.

The scalability depends on implementation. Since we have Verizon
implementing it, I'm guessing it scales just fine based on the size
of their email network.

See above. It doesn't scale when /everyone else/ starts doing it.

Callback verification if properly implemented will never generate more
junk SMTP traffic as DATA part of SMTP transmission never happens.

By the time Verizon's callback servers hang up on us they've already
generated more junk SMTP traffic, wasted our resources to protect their
customers, and aided spammers doing list validation. Your claim that
dictionary attacks are always alphabetical is pretty weak and brings
nothing to bear on the actual problem - that by rejecting mail from a
given address because of (possibly spurious) "verification", they are
actually giving the spammers a tool they can use to cull bad addresses
from their own lists.

The only positive thing I have to say about Verizon's callback scheme is
that so far it has not been seen here more than 6 times in a single day
in the past two months. So they must be doing some caching, given that
at least one of the domains we host has been under joe job outscatter
attack for several months running now.

Yes - *you* know that the address you put in the MAIL FROM: should be one
that you know is willing to accept any bounces you may be about to
generate, and *I* know that.

The problem is that the spammer (I include A/V companies that still
generate rejection notices) knows that as well - and doesn't feel
the same need to follow the rules and be a good netizen.

Yes, it's a royal pain trying to follow a protocol when you know up front
that there's a 70% to 90% chance that the other person is intentionally
failing to follow it....

It does not appear that they are caching it, here is a sample from my log
file:

Dec 6 19:18:15 white sm-mta[25976]: iB70IF6n025976: <lopqp@abc.net>... User unknown
Dec 6 19:18:15 white sm-mta[25977]: iB70IF6n025977: <lopqp@abc.net>... User unknown
Dec 6 19:18:16 white sm-mta[25976]: iB70IF6n025976: from=<>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc006pub.verizon.net [206.46.170.182]
Dec 6 19:18:16 white sm-mta[25977]: iB70IF6n025977: from=<>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc019pub.verizon.net [206.46.170.68]
Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: <lopqp@abc.net>... User unknown
Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: <lopqp@abc.net>... User unknown
Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: lost input channel from sc006pub.verizon.net [206.46.170.182] to MTA after rcpt
Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: from=<antispam245967@west.verizon.net>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc006pub.verizon.net [206.46.170.182]
Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: lost input channel from sc019pub.verizon.net [206.46.170.68] to MTA after rcpt
Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: from=<antispam830748@west.verizon.net>, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc019pub.verizon.net [206.46.170.68]

What happens when verizon tries to send email to somebody who does the same
type of check, does this not create an infinite loop?

K

Krzysztof Adamski wrote:

Well when I tested it (3 hours ago) I connected to them manually while
watching my incoming milter log. Indeed they visited immediate and
verified my address (with an rcpt to command) and when that succeeded,
I got the "sender ok" in my telnet session. I then broke the
connection and tried again. The second time they did not contact my
server and gave me a "sender ok".

It is now about three hours later and I just tried again and they did
not contact my server. Now this is the "success" case where the
mailbox exists. I don't know if they cache the non-existence of a
mailbox.

      -Jeff

Reply (*long* reply) being sent off-list. If anyone else wants to
see it, rattle my cage.

---Rsk

I don't think this is the issue, as we aren't getting that far in the
transaction.

Inspection suggests that the anycast announcements in the UK were pointing to
a server that wasn't accepting email.

I believe here the problem is using anycast, and not providing a backup system
not using anycast. The previous case I'm aware of was when bits of the NE USA
lost ".org" because they only had anycast DNS servers (and still do AFAIK),
and the announcement messed up.

Whilst I plead ignorant of the technical details of anycast, strikes me that
it is clearly more complex, and thus more prone to failure, and these
failures are potentially less obvious.