Is anyone else getting this message on every nanog post today?
"This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing>"
I don't know if this link itself is malware, as it goes to the MS store, or if something is broken in the Nanog Mail machine.
If it's just me, never mind. I'll figure it out.
-mel beckman
Usually mailing lists act like e-mail spoofers as far as SPF and DKIM is concerned. These two systems above try to minimize spoofed e-mail by doing the following:
SPF: Each domain adds a list of IP Addresses that are allowed to send e-mail on their behalf.
DKIM: Each email sent by an "original" mail server is cryptographically signed with a key available, again, in the DNS.
When you send an e-mail to a list, you send it to the mailing list mail server. After that, of the server forwards that e-mail to the recipients, its original address is shown, therefore if Outlook checks for SPF records, that check will fail. An easy way to get around this is for the list to change the From field to something else, like "Mel Beckman via NANOG" and a local email address.
However, when you send that email, it may also be signed with DKIM: any change in subject (say "[NANOG]" is added) or the body (say "You received this email because you subscribed to NANOG" is appended) will also cause that check to fail.
Typically the behavior of the recipient if one or both of these checks failed is described in yet another DNS record, called a DMARC Policy. Some set this to very strict levels (reject e-mail / send to spam), some others to warn the user (like what you saw?), and some others, knowing this happens, to ignore/notify.
This message probably appears because of the above SPF / DKIM / DMARC combo but I can't be 100% sure from the provided info.
In any case, this is likely not your fault. If you want to be sure, verify the contents of the e-mail against the public NANOG archive which is available over HTTPS. My guess is that nothing has been changed.
Thanks,
Antonios
Antonia's,
Thanks for the very clear explanation. I use DKIM and SPF, but didn't know about this corner case. I'm surprised the SPF, etc architects missed it, or seem to have. In any event, I seem to be getting all the messages.
-mel beckman
I don't think they did miss it per say. SPF is specifically meant to say where senders are allowed to send from. Mailing lists (in some configurations), forwarders, et. al. (inadvertently) violate this when they re-send the message with the original sender from a not-explicitly-allowed source.
Sender Rewriting Scheme is a way that these forwarding services can re-write the SMTP Envelope From address to not run afoul of SPF (et al).
Mailing list managers, in particular, can also change the message in a few different ways to avoid some of these pitfalls.
- Remove all but a subset of headers.
- Alter the RFC 822 From: header such that the message appears to come from the mailing list its self.
I also strongly recommend that mailing lists be viewed as an entity unto themselves. I.e. they receive the email, process it, and generate a new email /from/ /their/ /own/ /address/ with very similar content as the message they received.
I strongly encourage mailing list admins to enable Variable Envelope Return Path to help identify which subscribed recipient causes each individual bounce, even if the problem is from downstream forwards.
The problem with this is that it takes more processing power and bandwidth. Most people simply want an old school expansion that re-sends the same, unmodified, message to multiple recipients. - That methodology's heyday has come and mostly gone.
Hello,
Both SPF and DKIM are meant to be checked against the domain in the
envelope sender (SMTP protocol-level return address) which the NANOG list
sets to nanog-bounces@nanog.org. Checking against the message header "from"
address is an incorrect implementation which will break essentially all
mailing lists.
Regards,
Bill Herrin
Bill,
If that's the case, then Microsoft appears to be at fault here. I'll try opening a ticket (I know. Windmills 
-mel
That may be what the original intent was.
Every SPF implementation I've seen has checked the SMTP envelope FROM address /and/ the RFC 822 From: header address.
Granted, that does not mean that it's the correct behavior.
Hi Grant,
The gold standard, Spamassassin, does not. Indeed, the message to which I
reply was scored by spam assassin as "SPF_PASS" even though you do not
include NANOG's servers in the SPF record for tnetconsulting.net.
Regards,
Bill Herrin
Indeed, in more detail (which I omitted for simplicity), these checks are performed in a series of headers, the last of which is the From: header. I think the “envelope-from” is either the first or the second in this 5-point list.
That said, there are a lot of implementations out there that do not respect that and treat the From address as the sender whose honesty must be verified. Every time I send mail to a mailing list from my own domain, due to DMARC I get back several reports of SPF and DKIM fail, mainly because the mailing list messed up something.
The gold standard, Spamassassin, does not. Indeed, the message to
which I reply was scored by spam assassin as "SPF_PASS" even though
you do not include NANOG's servers in the SPF record for
tnetconsulting.net.
The message from Mr. Taylor (to which Mr. Herrin is replying) arrived
here with:
Return-path: <nanog-bounces@nanog.org>
So an SPF implementation that checks either or both of the (rfc2821
envelope from / rfc2822 header from) domains will pass.
The original was DKIM signed by d=tnetconsulting.net (c=simple/simple -
you might want to change that) but of course that signature was broken
by the nanog list handling.
Actually, my problem is not so much processing power and bandwidth,
but that every time I've encountered this problem I found a morass
of painfully broken, horribly documented, super-complex software.
With sendmail/postfix you can edit an alias file and say:
bob: joe, tim, alex
And boom, done. If I could enable some feature/module/whatever in
either one with a line or two of config to make that do Variable
Envelope Return Path I would, but every solution I know of requires
setting up a complex milter, running some external daemon, which
often depends on 3 different interpreted languages to be installed
and so on down a dependency hell.
While I haven't looked at real mailing list software recently
(e.g. mailman) when I last did they didn't suport this either and
it took a pile of 3rd party hacks to make it work.
Why o why in 2017 can this not be a checkbox, a line of config, or
so on.
For that matter, setting up DKIM is horrendously complicated for
no good reason...
This is incomplete.
TL;DR: SPF checks the envelope sender. DKIM doesn't check anything except to
test that parts of the message haven't been altered. DMARC adds policy to both
to check them against the header From:. Mailing list software may not work
with DMARC-reject senders (but Nanog does).
* Grant Taylor via NANOG:
Thanks for the very clear explanation. I use DKIM and SPF, but didn't
know about this corner case. I'm surprised the SPF, etc architects
missed it, or seem to have. In any event, I seem to be getting all
the messages.I don't think they did miss it per say. SPF is specifically meant to
say where senders are allowed to send from. Mailing lists (in some
configurations), forwarders, et. al. (inadvertently) violate this when
they re-send the message with the original sender from a
not-explicitly-allowed source.Sender Rewriting Scheme is a way that these forwarding services can
re-write the SMTP Envelope From address to not run afoul of SPF (et al).
SPF (in its specified form) is very compatible with the way people
have been running mailing lists since the mid-to-late 90s because the
mailing list specifies itself as the SMTP envelope from address. This
has the added benefit of enabling bounce processing.
Unfortunately, the envelope from address is used for generating
bounces only. Mail clients just show the header. That's why some SPF
variants (like the one proposed by Microsoft) apply SPF rules to email
headers as well, although this wasn't part of the original SPF spec
(because it breaks too much). DKIM was designed from the start to
(optionally) break mailing lists, and some mail providers now publish
sender domain policies which other mail providers enforce. In effect,
this requires pervasive header rewriting (within the mailing list
software) before the message can be sent over the mailing list,
obfuscating the true sender.
It's a very unfortunate situation. The mailing list software could
put the original sender address into a new header, and if that header
is standardized, mail clients could just display that. But then,
certain sender domains would just sign the absence of that header, and
we are back to square one. Presumably, we could use a randomized
header, so that at least DKIM protocol changes are required, but it is
basically an arms race at this point.
While I haven't looked at real mailing list software recently
(e.g. mailman) when I last did they didn't suport this either and
it took a pile of 3rd party hacks to make it work.
The latest versions of Mailman (2.1.23 and 3.0.0) both work reasonably well out-of-the-box with SPF, DKIM, and DMARC. Some additional configuration tuning might be necessary for additional compatibility. However, those features are still available in an out-of-the-box configuration, they’re just not enabled by default because they might cause more problems than they would solve for certain types of typical installations. So, if you want those features, you need to turn them on.
IMO, Mailman3 works better out-of-the-box with SPF, DKIM, and DMARC as compared to Mailman 2.1.x, but that codebase is still pretty fresh. We’re now using it by default for mailing lists hosted on python.org, but we have not yet converted any of the older Mailman 2.1.x lists over to Mailman 3. We haven’t noticed any major problems yet with the latest version of Mailman3, but we still want to be careful in our testing.
For that matter, setting up DKIM is horrendously complicated for
no good reason…
Sites like DMARCian help with that process to a degree, but there’s still a lot of complexity there that I would like to see handled automatically.
Unfortunately, that’s kind of the nature of the beast right now with these tools. The technology is still complex and difficult to configure, and it’s easy to set things up in a way that you wind up shooting yourself in the foot — and possibly with a large thermonuclear device.
No provider is immune to these mistakes, and some providers are more likely to make big mistakes than others.
Hi Alan,
I accept your explanation as the correct one.
Regards,
Bill Herrin
The purpose of SPF is to REJECT messages before the data phase. This cannot be done if you are checking the RFC-822 From: header since that requires accepting the message and invalidates the entire purpose of SPF.
I have never seen an SPF implementation that uses the RFC-822 header From. Doing so would be pointless.
So for DMARC+SPF to pass not only must the message come from a source
authorized by the envelope sender domain, but that domain must be the
same domain (or parent domain or subdomain) of the header From:
address.
For DMARC+DKIM to pass, the DKIM signature must pass and the DKIM
signing domain must be the same domain (or parent domain or subdomain)
of the header From: address.
Again, DMARC requires only one or the other mechanism to pass. So
messages forwarded intact should be OK if they have an aligned DKIM
signature.
Brad Knowles wrote:
...and it's easy to set things up in a way that you wind up shooting
yourself in the foot -- and possibly with a large thermonuclear
device.
For an example of that (unless I am misunderstanding something), we
have:
--> Hello marketo-email.box.com [192.28.147.169], pleased to meet you
<-- MAIL FROM:<$MUNGED@marketo-email.box.com>
<-- RCPT TO: ...
dkim pass header.d=mktdns.com
rfc2822 from header = $MUNGED@email.box.com
dig _dmarc.email.box.com txt +short
"v=DMARC1; p=reject; ..."
dig email.box.com txt +short
"v=spf1 ip4:192.28.147.168 -all"
So given the dmarc reject policy, it needs to pass either spf (which
fails 192.28.147.168 != 192.28.147.169), or dkim (which fails since it
is not signed by anything related to email.box.com.
Am I missing something, or is that just broken?
That appears to be broken. The -all on the SPF record alone breaks it, since
receivers should refuse it at that point. But yeah the DMARC is also broken.
Interestingly, the mail I've seen recently from email.box.com has multiple
signatures, one of which is from email.box.com. And it originated from
192.28.147.168. Weird.
> For an example of that (unless I am misunderstanding something), we
> have:
>
> --> Hello marketo-email.box.com [192.28.147.169], pleased to meet you
> <-- MAIL FROM:<$MUNGED@marketo-email.box.com>
> <-- RCPT TO: ...
>
> dkim pass header.d=mktdns.com
> rfc2822 from header = $MUNGED@email.box.com
>
>
> dig _dmarc.email.box.com txt +short
> "v=DMARC1; p=reject; ..."
>
> dig email.box.com txt +short
> "v=spf1 ip4:192.28.147.168 -all"
Well you should be checking the correct TXT record for SPF.
dig marketo-email.box.com txt +short
"v=spf1 ip4:192.28.147.168 ip4:192.28.147.169 -all"
Hm, a closer reading of rfc7489 sheds some light on this:
Would dmarc-spf consider marketo-email.box.com to be 'aligned' with the
from header email.box.com domain? It is neither a child nor parent of
email.box.com.
The _dmarc txt record for email.box.com has no aspf: tag, so we should
be operating in spf/dkim relaxed alignment mode.
rfc7489, when discussing relaxed identifier alignment, says the
"Organizational Domain" of the identifiers must match. But there is no
explicit example of that. Instead, the examples talk about one of the
identifiers being a parent of the other identifier.
The envelope from marketo-email.box.com and the 2822 header from
email.box.com have the same box.com organizational domain. If we ignore
the examples in rfc7489, it looks like this is NOT broken.
I am probably not the only one that wrote code matching on the
parent/child relationship of the identifiers, rather than computing the
Organizational Domains and matching those.
As Mr. Hodgson pointed out, box.com has very recently started sending
mail with multiple dkim signatures, header.d=email.box.com and 2822
header from = email.box.com.
Now off to fix my code.