Email peering (Was: Economics of SPAM [Was: Micorsoft's Sender IDAuthentication......?]

Although I agree that email peering is a seriously bad idea, I don't
think that the analogy to uucp is correct. Uucp users had to know
explicit paths; today, we'd use routing protocols. (I tried the
equivalent for uucp back in 1982, but the map data -- think routing
registry -- was of too low quality. Hmm -- today's routing registry
isn't very complete, either. But we have BGP, which uucp didn't have.)

Uucp also relied on relative names, rather than absolute domain names.
This scheme would retain domain names.

The other big difference from early uucp is the presence of business
agreements. A lot of uucp email (and netnews) traffic was, shall we
say, not always carried in accord with corporate goals. People today
are accustomed to paying for basic Internet services such as access;
that was rarely possible in uucp days. (For more details, see
http://www.cs.columbia.edu/~smb/papers/pathalias.paper.pdf by myself
and Peter Honeyman, published in 1986.)

There are, however, three very big problems. First, it forces people to
pay for services that they don't pay for today. I'm not talking about
Jo[e] Consumer, I'm talking about a modest-sized business or ISP. They
have the ability to deliver email today and will resist being told to
pay. Nor will paying stop them from receiving spam -- at best, they
see less spam if *you* pay. In other words, the incentives are
backwards.

A second problem is that multi-hop email seriously reduces reliability.
That is indeed a lesson we learned in uucp days. It's mentioned in the
paper; it was present even more explicitly in the original pathalias
software package I released.

But the biggest issue is control: if you have to pass email through a
site, that site controls whether or not it can be delivered. Yes, that
might stop (some) spam. Might it also stop unpopular political ideas,
or provide a facility for government (or whomever) to profile you?
Maybe my upstream email provider, 3 hops away, is Google; does that
mean I've consented to let Google associate keywords in my email with
my email address? (I won't reprise the debate about gmail and privacy,
save to note that as of this morning, the Google web page on *sender*
privacy still doesn't address that point.)

    --Steven M. Bellovin, http://www.cs.columbia.edu/~smb

You're right -- I left out the routing table bit, which also existed some
time ago. BITNET used the bitnet.links file; here's an old one still
accessible for viewing:

    http://web.mit.edu/afs/athena/reference/net-directory/host-tables/bitnet.links

Similar concept, same scaling problems; it just hides the explicit routing
from the user (as would any modern "peering" system, presumably).

And just to add to it all, you might want to compare to another similar
system - FidoNet with its nodelist routing files.

Randy Bush is probably right person to ask but I have had a feeling in 1994 that its routing system would not be able to deal with all the people that want to be connected to global email system. But of course we end with everyone switching to internet and its no longer a problem (and fidonet zones now are either smaller or same size as 10 years ago).

And also consider that fidonet does have hierarchical address which
makes routing decisions easier. With internet and majority of users
using email addresses in TLDs (or country SLDs which is the same),
there is no such hierarchy available.

So my opinion is that email peering is not workable as far as solution
for entire internet. But some of it may be of use in very limited scale
between certain very large providers as internal whitelisting system.

Similar concept, same scaling problems; it just hides the explicit

routing

from the user (as would any modern "peering" system, presumably).

Then you are presuming wrongly. Nowhere in what I wrote have
I suggested any changes in the existing email technology. I am
not suggesting that we drop SMTP in favour of your favourite
old dusty protocol. I am suggesting that we need a system of
accountability for people who run Internet email servers based
on contracts and SLAs, i.e. peering agreements.

I haven't specified how it would be implemented because I expect
that the companies negotiating such agreements would specify this
in some kind of operational best-practices document.

One way that it COULD be implemented is for people accepting
incoming email on port 25 to check a whitelist before accepting
email. Only operators who have signed a peering agreement would
be on the whitelist. Presumably, the whitelist would be served
up by your regional association and they would have some means
of relaying queries (or synchronizing their database) with the
other 4 regions.

Let's face it, people have described a lot of best practices
for running SMTP based email services but there has never
been a concerted effort to implement these in some methodical
way. It has always been a case of preaching to the converted
at NANOG or on some lists. And it just does not scale!

--Michael Dillon

There are, however, three very big problems. First, it forces people to

pay for services that they don't pay for today.

Businesses often pay, not for services, but for accountability.
They want someone else to take responsibility for a problem
even if it costs them more money than taking that responsibility
on themselves. Insurance, maintenance contracts, etc.

Today, if Joe Business gets lots of spam, it is not his
ISP's responsibility. He has no-one to take responsibility
for this problem off his hands. But if he only accepts
incoming email through an operator who is part of the
email peering network, he knows that somewhere there is
someone who will take responsibility for the problem.

That is something that businesses will pay for.

But first, ISPs have to put their hands up and take
collective responsibility for Internet email as a service
that has value and not just as some kind of loss leader
for Internet access services.

--Michael Dillon

Similar concept, same scaling problems; it just hides the explicit

routing

from the user (as would any modern "peering" system, presumably).

<snip>

One way that it COULD be implemented is for people accepting
incoming email on port 25 to check a whitelist before accepting
email. Only operators who have signed a peering agreement would
be on the whitelist. Presumably, the whitelist would be served
up by your regional association and they would have some means of relaying queries (or synchronizing their database) with the
other 4 regions.

DNSWL -- this is already being done. It is not widely viewed as being in any way similar to a peering concept. What would be more similar would be a consortium of large providers providing such a whitelist. That would be something I would welcome.

I would settle for having aol,msn,yahoo,earthlink,cablevision or any half dozen providers making public THEIR whitelists.

The problem is that there does not appear to be any incentive for them to do so -- fee or no fee.

In fact, I would encourage anyone planning on ragging on DNSBL's to put up AND shut up, namely operate a DNSWL.

Existing public whitelists include:

exemption.ahbl.org
bondedsender.org
habeas.com

To use it with sendmail:

jlewis's http://njabl.org/dnswl.m4
http://groups-beta.google.com/group/comp.mail.sendmail/msg/a26d1cbd1c739626

To use it with spamassassin:

header XXX_DNSWL eval:check_rbl('xxx-firsttrusted', 'xxx.ttec.net')
score XXX_DNSWL -5

Anyone else with a public DNS whitelist?

<snip>

Something that is already being setup, and that tends to add a slight
amount of reputation to any authentication schemes that might be used,
is a "feedback loop"

Kind of like what we do, or what AOL does (http://postmaster.info.aol.com/fbl/)

Not public as such, but well it is as much like peering as anything in
the smtp email world can be. [e&oe gateways to uucp and x400]

Michael.Dillon@btradianz.com said:

That is something that businesses will pay for.

But first, ISPs have to put their hands up and take
collective responsibility for Internet email as a service
that has value and not just as some kind of loss leader
for Internet access services.

Many large organizations already have already, in a case by case way, set
up private mail peering with others they exchange large volumes of mail
with. This "trusted traffic" is often able to bypass the expense and delay
of the spam-filter farm, making the cost and hassle of a parallel mail
infrastructure worthwhile to them, and everyone is happy.

There is no reason you can't pick another port, modify one of the many
FOSS mail servers out there to do whatever it is you are proposing, and
start providing this kind of thing on a more formal basis. Call it an
email toll-road.

(hmmmmm, would a toll-road be troll free? I might pay for that).

If you are able to create a solution that works, and that people will pay
for, then you'll be happy. Since it works in parallel, it won't disrupt
anyone who doesn't want to play along, so they won't be anymore unhappy
than they are now.

I don't think what you have been talking about so far will work, and I
don't think I'm alone in that. But hey, prove me wrong, and maybe someday
I'll be writing a check to you to make sure I get email every day.

Best,
Ben

Many large organizations already have already, in a case by case way,

set

up private mail peering with others they exchange large volumes of mail
with. This "trusted traffic" is often able to bypass the expense and

delay

of the spam-filter farm, making the cost and hassle of a parallel mail
infrastructure worthwhile to them, and everyone is happy.

Sounds good.

I don't think what you have been talking about so far will work, and I
don't think I'm alone in that.

That's strange because you just finished describing how
SOME companies are already engaging in email peering on
a piecemeal basis. And how these companies ARE finding
this to be beneficial in reducing costs. So please explain
why my suggestion about widespread email peering agreements
won't work?

And please don't suggest that webs of trust are not
scalable. Given the techniques of scaling that we have
in the 21st century, I simply don't believe that.

--Michael Dillon

Michael.Dillon@btradianz.com said:

That's strange because you just finished describing how
SOME companies are already engaging in email peering on
a piecemeal basis. And how these companies ARE finding
this to be beneficial in reducing costs. So please explain
why my suggestion about widespread email peering agreements
won't work?

Because I don't think "some companies" == "the entire population of email
users", or even a sizable (ie widespread) part of that population.

A large number of people are fine with the current system, and thus won't
pay more for something else. Me, for example.

Those who are unhappy will pay more for a better solution, and some small
number with really deep pockets may be at that point where they will pay
for something like "business class" email, in addition to the "tourist
class" email they already have.

You seem to repeatedly describe a solution that becomes so big that it (at
least substantially) replaces 25/SMTP. That's what I don't think will
work, or is needed.

And please don't suggest that webs of trust are not
scalable. Given the techniques of scaling that we have
in the 21st century, I simply don't believe that.

I don't think either are relevant to this discussion. In the utility model
you seem to talk about (and that I was talking about) all you care about
is the provider. If you contract with them for traceable, trusted spam
free email, and they give you something less than that, they pay a
penalty. The utility knows, and has a contractual relationship with, each
endpoint, and presumably can keep track of traffic in its own network.
Problem solved.

And the whole thing doesn't need to scale, because there are a severely
limited number of companies that would be willing to pay the costs for
such a service. But they are out there, and one might be able to make a
business out of it.

Best,
Ben

> Similar concept, same scaling problems; it just hides the explicit
routing
> from the user (as would any modern "peering" system, presumably).

Then you are presuming wrongly. Nowhere in what I wrote have
I suggested any changes in the existing email technology. I am
not suggesting that we drop SMTP in favour of your favourite
old dusty protocol. I am suggesting that we need a system of
accountability for people who run Internet email servers based
on contracts and SLAs, i.e. peering agreements.

In between the choice of accepting mail from *anybody* by default which we
have now and the choice of accepting mail from *nobody* by default that
explicit peering agreements represents there is another solution; which is
to accept mail only from IPs that have *some relation* to the sender's

From domain, for example by MX record or by reverse DNS (we implemented

that test and call it MX+).

Here is a downloadable reference implementation for use with procmail:

http://mxplus.org/

The example program mxplus is code that was carved out of the mail server
software we use and made standalone. It's an antispam option that works
well for many users. The example includes sender email address
validation, which is another test like MX+ that works well for most users
and breaks under usually acceptable circumstances when senders do bad
things like send email with an invalid From address. YMMV.

Mike.

+----------------- H U R R I C A N E - E L E C T R I C -----------------+

In between the choice of accepting mail from *anybody* by default
which we have now and the choice of accepting mail from *nobody* by
default that explicit peering agreements represents there is another
solution; which is to accept mail only from IPs that have *some
relation* to the sender's From domain, for example by MX record or by
reverse DNS (we implemented that test and call it MX+).

This has the same problem as all of the other duct tape authorization
schemes -- it breaks a lot of valid e-mail, so that you have to
maintain a painfully large manual exception table, or write off a lot
of mail that your users will not forgive you for losing, or more
likely, both.

In this particular case, the biggest issue is forwarders, commercial
ones like pobox.com, associations like the ACM and IEEE (I get some
odd mail being uucp at computer.org), and large numbers of colleges
and universities which let graduates keep their email address. In all
of those cases, the users send mail from their own ISPs, whatever they
are, inbound mail is forwarded back to the ISP accounts, and there is
no way to enumerate the valid sources of mail.

There's also plenty of domains where the inbound and outbound mail
servers are different, and neither one matches the domain name of the
mail. For example, I host about 300 small mail domains on a pop
toaster here. The MX is mail2.iecc.com, and the outbound host that
many but not all of them use is xuxa.iecc.com. (Mail for iecc.com
itself is on another host.) The IPs all happen to be in the same /24,
but guessing whether two IPs are "close enough" is a poor way to
authenticate or authorize anything.

Before you point out that they could change the way those systems work
to be compatible with your scheme, well, duh, sure. But if you're
going to make people change their existing working mail setups,
there's little point in going through the vast cost of a widespread
change for such a marginal benefit. Read archives of SPF mailing
lists for endless flamage on this topic, since SPF has the same
problem.

Regards,
John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Information Superhighwayman wanna-be, http://www.johnlevine.com, Mayor
"A book is a sneeze." - E.B. White, on the writing of Charlotte's Web

>In between the choice of accepting mail from *anybody* by default
>which we have now and the choice of accepting mail from *nobody* by
>default that explicit peering agreements represents there is another
>solution; which is to accept mail only from IPs that have *some
>relation* to the sender's From domain, for example by MX record or by
>reverse DNS (we implemented that test and call it MX+).

This has the same problem as all of the other duct tape authorization
schemes -- it breaks a lot of valid e-mail, so that you have to
maintain a painfully large manual exception table, or write off a lot
of mail that your users will not forgive you for losing, or more
likely, both.

In this particular case, the biggest issue is forwarders, commercial
ones like pobox.com, associations like the ACM and IEEE (I get some
odd mail being uucp at computer.org), and large numbers of colleges
and universities which let graduates keep their email address. In all
of those cases, the users send mail from their own ISPs, whatever they
are, inbound mail is forwarded back to the ISP accounts, and there is
no way to enumerate the valid sources of mail.

This gets into the discussion of what percentage of mail a user gets that
is like this. It varies widely. From our measurements for most users
this is less than 1 percent. For me or you its definitely much higher
than 1 percent (I definitely agree with you).

There's also plenty of domains where the inbound and outbound mail
servers are different, and neither one matches the domain name of the
mail. For example, I host about 300 small mail domains on a pop
toaster here. The MX is mail2.iecc.com, and the outbound host that
many but not all of them use is xuxa.iecc.com. (Mail for iecc.com
itself is on another host.) The IPs all happen to be in the same /24,
but guessing whether two IPs are "close enough" is a poor way to
authenticate or authorize anything.

In between the two extremes of spam growing at the rate of Moores law and
not using email anymore there are all sorts of solutions. Pick a solution
or solutions that you like, or not. Virtually all of them will result in
some sort of reduction in the current ability of anybody being able to
send mail as anyone from anywhere.

Before you point out that they could change the way those systems work
to be compatible with your scheme, well, duh, sure.

Nope wasn't going to. They'll break when sending to people who filter or
reject based on MX+.

But if you're going to make people change their existing working mail
setups, there's little point in going through the vast cost of a
widespread change for such a marginal benefit. Read archives of SPF
mailing lists for endless flamage on this topic, since SPF has the
same problem.

Oh yeah. The only different here is that this is simpler, doesn't require
any interpretation of the SPF/Microsoft policy/patent wars, and doesn't
involve a complex virtually turning complete macro language embeded in
DNS. You don't have to understand any new DNS entries because there are
no special DNS entries with MX+.

With regards to the marginal benefit, this a measurement thing that only
you can determine for your specific userbase.

We can measure the results from MX+ and greylisting for our users. For
the people that use it, they are really happy. Perhaps they correspond
with more users of large ISPs and companies that satisfy the test
criteria. If you assert that it won't work for your users, that's
completely within the realm of normal decision making, who am I to tell
you what is going to work for your users. You only have so much time in
the day, you have to make your own judgement calls for whatever techniques
to make available to your users.

Mike.

+----------------- H U R R I C A N E - E L E C T R I C -----------------+

This has the same problem as all of the other duct tape authorization
schemes -- it breaks a lot of valid e-mail, ...

In this particular case, the biggest issue is forwarders, ...

This gets into the discussion of what percentage of mail a user gets that
is like this. It varies widely.

Agreed. Around here (Ithaca NY) it's probably on the order of 20% due
to all of the Cornell grads who still use their cornell.edu address.

Pick a solution or solutions that you like, or not. Virtually all
of them will result in some sort of reduction in the current ability
of anybody being able to send mail as anyone from anywhere.

Right. That's why for widespread deployment we have to look at the
small minority of schemes that don't break legitimate mail. That's
why I'm looking at CSV, which makes it easier to assign reputation to
sending hosts, and domainkeys (or whatever it's called when they're done
mixing in IIM) which if sensibly used makes it easier to whitelist mail
from people you like.

R's,
John

Folks,

DNSWL -- this is already being done. It is not widely viewed as being in
any way similar to a peering concept. What would be more similar would
be a consortium of large providers providing such a whitelist. That
would be something I would welcome.

To repeat what John Levine said, and that I suggested in my posting "Informal
email peering" please take a look at CSV <http://mipassoc.org/csv> as a
candidate mechanism for determining the operations-related identity to assess,
and for a means of querying a third party to obtain an assessment.

CSV is simple, uses efficient DNS records, and mostly importantly uses
operations identities rather than content origination identities.

Several schemes that have some popularity use a path registration approach (SPF,
Sender-ID) which ties an origination identifier (rfc2822.From, rfc2822.Sender,
or rfc2821.MailFrom) to the MTAs along the transmission path.

For you ops folks, think of this as requiring pre-registration of all source
routes to all recipients. For you architecture freaks, think of it as a really
spiffy layer violation.

By contrast, CSV uses identities that are directly tied to the MTA that
is being assessed.

Once you have a validated identity, you need a scalable means of
assessing it. The combinatorial explosion with email makes pair-wise
agreements unscalable. Hence, some form of third-party assessment
schemes is needed.

And that is what motivated the idea for <http://mipassoc.org>. Develop
a common set of best practises, and have organization commit to
supporting them.

  d/

Today, if Joe Business gets lots of spam, it is not his
ISP's responsibility. He has no-one to take responsibility
for this problem off his hands. But if he only accepts
incoming email through an operator who is part of the
email peering network, he knows that somewhere there is
someone who will take responsibility for the problem.

That is something that businesses will pay for.

Just look at the Postini numbers...

Pete

My e-mail is alex@relcom.net, but I send it when I am on DSL with EthLink
(and thru Earthlink SMTP). And it is 100% valid situation.

Sure is - which is one reason why spf just is not going to work for you
CSV on the other hand makes no claim about the sender - it looks at
the sending host

Please let me borrow Ben's point and expand on it.

Spam as it's usually discussed (spam propagated via SMTP) is only part of
the spam problem. We've seen Usenet spam, chat room spam, http referrer
log spam, blog spam, and so on. And all of those bundled together and
labeled as "spam" are only part of the overall network abuse problem --
which also involves phishing, zombies, DoS attacks, spyware, etc.
And these are all (increasingly) interelated problems, e.g. spam is used to
phish people to sites which forcibly download spyware, and so on.

We could (and some already have) spend an enormous amount of time devising
very clever "solutions" to these and deploying them. But as we've seen,
doing so usually results only in a shift in the nature of the abuse, not
an overall reduction in it.

So even if we had The Perfect Solution to SMTP spam and it was globally
deployed tomorrow and had no adverse side-effects...we'd buy ourselves
a brief respite, no better.

I'm not saying some of the technical approaches aren't clever. They are.
But none of them are going to solve the problem for any acceptable value
of "solve", not because there's anything wrong with them per se, but
because they're technological attempts to solve the problem at its
end points -- rather than its source points.

"The best place to stop abuse is as near its source as possible."

Meaning: it's far easier for network X to stop abuse from leaving its
network than it is for 100,000 other networks to defend themselves from it.
Especially since techniques for doing so (for instance, controlling
outbound SMTP spam) are well-known, heavily documented, and easily put
into service.

The problem is that network X, for many values of "X" (see the data
compiled by Spamhaus or SPEWS or any number of others) hasn't done so.
Whether that failure is due to incompetence, greed, laziness, negligence
or anything else is an interesting question...but really doesn't matter,
because regardless of the cause, the fastest way to get it fixed is to
make it X's problem...*not everyone else's*. (It's often impressive
how fast X can move--despite protestations otherwise--when this situation
is created.)

Those who have been around a long long time know that this is how it
used to be. If your network started spewing crap, and didn't stop spewing
crap in a fairly timely manner, you got a phone call or email explaining
that someone had their hand on your plug and was going to pull it.

The point? The point is that there is no need for any new technology
to deal with the spam/abuse probem. What there is a desperate need
for is the *will* to use the technology we already have -- to shift
the burden of dealing with abuse onto those who are permitting it
to originate from their network. This can be done in a number of
ways: using DNSBLs, firewalls, routers, whatever.

Because if it's not done, then Network X, for many values of X,
will be perfectly happy to watch everyone else innovate and scramble
and spend money to defend themselves *as long as X doesn't have to*.
As we've seen. For many years. Over and over and over again.

After all, why should they? There's nothing in it for them and no
downside if they don't.

  "[...] if you give people the means to hurt you, and they do it,
  and you take no action except to continue giving them the means
  to hurt you, and they take no action except to keep hurting you,
  then one of the ways you can describe the situation is "it isn't
  scaling well."

                --- Paul Vixie

So either the collective "we" has the will to stop putting up with this
nonsense -- or we don't. If it's the former, then we already have all
the tools we need. If it's the latter, then nothing we come up with,
no matter who clever it is, is going to make any real difference.

---Rsk

Rich Kulawiec wrote:

"The best place to stop abuse is as near its source as possible."

Meaning: it's far easier for network X to stop abuse from leaving its
network than it is for 100,000 other networks to defend themselves from it.
Especially since techniques for doing so (for instance, controlling
outbound SMTP spam) are well-known, heavily documented, and easily put
into service.

The problem with countermeasures that would actually hurt the source of junk heavily enough would also have to hurt "legitimate" traffic making you an immediate lawsuit magnet. If that would not be the case, or some larger parties feel they could stand despite this fact, the problem would be fairly straightforward to reduce to a fraction in a few months time.

Pete