Unable to email anyone from my primary domain name; thanks Google Mail and G Suite.

Mailman List Mirror (Read Only)
1

Dear NANOG@,

I’m not sure where else to post this, and this is not really new, either, but I think I have a new take here.

I use my own personal domain name for various UNIX stuff, including sending log-related things to myself out of cron, which end up in my own Gmail.com account, either directly, or through forwarding (w/o SRS). (I do not use G Suite for my own domain name, for obvious reasons; just the consumer-based gmail.com email address from the old times of invitation-based registrations.)

Over the years, I sometimes had certain messages rejected by Gmail, but it was a very low rate of rejection (less than 5% for any mail I cared about), and wasn’t a major problem (usually only some automated messages would be rejected).

A couple of months ago, I setup some new scripts that would send me new nightly emails. It’s all plain text, but had a few dozen of domain names present (it’s logs). Absolutely no links, just plenty of domains which I don’t control. So, Gmail has been presenting most of these messages with their red warning label that the email contains malicious links, even though all of these emails contained zero links, zero URLs to any of these unknown domain names, zero URL schemes, zero “http://”, zero “https://” etc. You get the idea.

Since about a few weeks ago, I am now seeing at least a 95% rejection rate for my domain name, for ALL email, including the forwards. Including emails which I send to myself from within Google, and which get forwarded back to Gmail by my UNIX machine (which is not known to break Gmail’s DKIM, either, although it’s also difficult to test, because when it does get through, it’s automatically marked as a duplicate by Gmail, so, you don’t get DKIM status from Gmail by looking at the headers, since you only get to examine the original copy that was sent, not the forwarded duplicate that was subsequently accepted). I.e., emails with a passing DMARC still get rejected.

The funny thing is, Google doesn’t actually blacklist my primary IPv6 address in my own /48 from which all of my messages originate; even though the rDNS resolves to a subdomain on the very same domain name which they’ve blacklisted “due to the very low reputation”. They’ve blacklisted just the main domain name that I use for my own non-Gmail-hosted mail. Sending the same messages into my Gmail.com from a different domain name in MAIL FROM, which is served from the same zone file DNS-wise (e.g., an SPF pass), through sendmail’s -f option, or with Mutt, makes the messages go through (even with rDNS being “low reputation”); sending it from my primary domain name in MAIL FROM results in the following:

DATA
<<< 550-5.7.1 [2001:470:xxxx:: 19] Our system has detected that this message is
<<< 550-5.7.1 likely suspicious due to the very low reputation of the sending
<<< 550-5.7.1 domain. To best protect our users from spam, the message has been
<<< 550-5.7.1 blocked. Please visit
<<< 550 5.7.1 https://support.google.com/mail/answer/188131 for more information. 135si403977wma.43 - gsmtp
554 5.0.0 Service unavailable

The support article suggests using Postmaster Tools; great, never heard of it, sounds cool; let’s verify our domain, and try it out, hopefully, there’s a solution right there.

However, after verifying my domain name through DNS for Postmaster Tools, it is revealed that Postmaster Tools cannot tell me anything at all, with all tabs and screens being 100% blank, allegedly because I’m not actually a mass email sender (I don’t send hundreds of emails a day or whatnot), and they’re too afraid that I’ll figure out why my mail doesn’t actually go through, instead of signing up for G Suite.

Right now, I’ve had a business need to reply to a work-related email from some other business.

This is what I got after sending my reply from my primary domain name through mutt — a nice double rejection by both the G Suite and Gmail in a single bounce generated by my server:

----- Transcript of session follows -----
… while talking to aspmx.l.google.com.:

DATA
<<< 550-5.7.1 [2001:470:xxxx:: 19] Our system has detected that this message is
<<< 550-5.7.1 likely suspicious due to the very low reputation of the sending
<<< 550-5.7.1 domain. To best protect our users from spam, the message has been
<<< 550-5.7.1 blocked. Please visit
<<< 550 5.7.1 https://support.google.com/mail/answer/188131 for more information. z11si12494671wrw.137 - gsmtp
554 5.0.0 Service unavailable
… while talking to gmail-smtp-in.l.google.com.:
DATA
<<< 550-5.7.1 [2001:470:xxxx:: 19] Our system has detected that this message is
<<< 550-5.7.1 likely suspicious due to the very low reputation of the sending
<<< 550-5.7.1 domain. To best protect our users from spam, the message has been
<<< 550-5.7.1 blocked. Please visit
<<< 550 5.7.1 https://support.google.com/mail/answer/188131 for more information. 135si403977wma.43 - gsmtp
554 5.0.0 Service unavailable

Changing MAIL FROM into a non-primary domain name (served out of an identical zone file, basically) gets the message accepted, without DKIM, without the 4-minute delay that many “suspicious” messages have had for months now, from the very same IPv6 address with the rDNS pointing to the domain name with “the very low reputation”, and it shows up in both my own Gmail as well as, presumably, in the G Suite account of the business partner I was replying to. (Note that this trick where the rDNS domain gets ignored works only for new emails with a passing SPF; I presume the rDNS still prevails in bringing the “low reputation of the sending domain” for forwards, as they don’t seem to succeed any longer now.)

There are a number of possible tl;dr: takeaways here:

  • don’t spread the monoculture — don’t use G Suite for your organisation;

  • don’t send crontab output to your Gmail from your primary domain name;

  • don’t use Gmail.

What is my own takeaway here?

  • Without being an anti-Google zealot, from a purely practical perspective, since my Gmail account no longer contains the mail I care most about, as it gets rejected on the SMTP layer, I’ll have fewer reasons to use it.

  • I’ll now have no other choice but to modify my setup to stop forwarding to Gmail, because my business contacts don’t need to see all these bounces that are now taking place.

  • Since so many businesses are G Suite useds, I’m still looking for a solution to get rid of the “the very low reputation of the sending domain” from a domain name I’ve been using since 2007, and which I’m 100% convinced was blacklisted by Google entirely for me sending crontab with system logs (zero HTML, zero URLs) to my own Gmail. I have SPF and DMARC all setup and passing since years ago, which doesn’t stop this issue from occurring. Merely switching the name of the domain in From and MAIL FROM to any other domain I own (which points to the very same MX) appears to be my workaround for now.

Some possible suggestions for Google, if I may:

  • Maybe don’t convert schemeless domain names which are non-URLs into “malicious” URLs? (They already do seem to block them from being presented as links in the UI in such an instance, but there’s little reason for trying to convert these domain names into links in the first place.)

  • Maybe don’t consider harmless plain text emails with a bunch of domain names to contain malicious links when they don’t?

  • Maybe don’t assume everyone with a domain name runs a G Suite? (Their whole troubleshooting guide is built around it.)

  • Maybe don’t assume everyone with a domain name sends hundreds of emails from their domain name per day? (They seem to limit Postmaster Tools based on such a criterion.)

  • Maybe don’t blacklist a domain name for sending harmless logs to a Gmail account that lists said domain name as an alternative From address?

Cheers,
Constantine.

2

zip up the log before you send it. -Joe

3

I’d recommend posting this over on the mailop list as well. Lots of discussions about issues like this there.

I too send myself various cron/*nix emails. The difference is I send to my own domain on my own server so I don’t see the issues you do. (Or rather if I do, I can control them)

Funny enough, I had a script that shot off an email with the malicious domains (blacklist) it had updated for a squid proxy that I run. I had to do some rejiggering to get this through, if I recall Spamassassin specifically viewed it as highly radioactive.

Bigger picture, I think that (unfortunately) we will see more and more problems like this. With the large providers running so much (as you mentioned - “monoculture”), and their services tending toward the “black box”… I don’t know what the answer is.

4

DNUESTTM<TM> (Do Not Use Entertainment Systems for Things That Matter)

5

This is a perfect example of why service providers, IT consulting outfits, etc. really need to either run their own mail infrastructure or contract with an outfit that can handle their unique requirements. Google, etc. is obviously going to have things very much set up for their normal use case, and their infrastructure is so large that I'm totally not surprised that the sender RHS reputation system doesn't talk to the user configuration's "alternative FROM address" nor interact with the UI's presentation (or lack thereof) of URI-like text as clickable links.

I've had this discussion with at least one service provider I help manage. There was a desire to use G Suite for email since it was simple. Glad I was not crazy in counseling them that it was a potentially bad idea (for this and other reasons).

Another bizarre example: I have a client who uses mimecast for their email. I simply cannot email them attachments whose names end in .7z. No amount of whitelisting my address, FROM domain, MX, etc. will make them get through to the inbox. They get silently spambinned every time. Send the same content in a .zip (even if it's actually a 7zip file), and it's fine. Somebody probably bolted a filter on the front of everything to handle some attack years ago that just says ".7z bad, umkey?".

6

Dear NANOG@,

[...]

Over the years, I sometimes had certain messages rejected by Gmail, but it
was a very low rate of rejection (less than 5% for any mail I cared about),
and wasn't a major problem (usually only some automated messages would be
rejected).

[...]

I have noticed that you do cc to yourself. I used to do it, too, but I
also have impression more of my mails get through since I started to
bcc myself, which does not show on the other side and does not trigger
malicious safeguards.

Not that it is going to help much. It looks like gog put a lot of
control into hands of software and then lost control of the
software. No point to try contacting them from rejected address or
trying to mention the address while contacting from passable one.

7

[ I'm just going to focus on one point. ]

it is revealed that Postmaster Tools cannot tell me anything at all, with
all tabs and screens being 100% blank, allegedly because I'm not actually a
mass email sender (I don't send hundreds of emails a day or whatnot), and
they're too afraid that I'll figure out why my mail doesn't actually go
through, instead of signing up for G Suite.

There is a persistent mythos -- a worst practice, actually -- among many
operations that obfuscating the reasons why messages are rejected is useful.
This is wrong.

Consider: either the sender is benign (as in this case) or they are not.

If they are benign, then denying the information necessary to understand
and solve the problem helps no one. It's also counter to the decades of
cooperation and mutual assistance that have built the Internet.

If they're not benign, then either they don't care enough to acquire
this information or they do. If they don't care, then providing the
information doesn't hurt, because it'll be ignored anyway. If they do
care, then they WILL get it, whether by conducting research or by
breaching security or by the simpler/cheaper path of paying someone
on the inside off. (If you're going to tell me that everyone who
works AT Google is working FOR Google, then I'm going to tell you that
you're naive and clueless. If I were in the large-scale spam-for-hire
business, I'd have already planted my own people there a long time ago.)

Best practice when rejecting mail traffic is to (a) provide at least
a semblance of a reason why and (b) a remediation path that includes
escalation to real live human beings. All mail systems (except for
the edge case of those which accept everything) make rejection errors
and that must be accounted for in design and operation.

---rsk

8

“Trust Andrew™ when I say this.”

Disable your IPv6 access to their mail server.

At Google, something hasn’t worked, well since the beginning of time, when it come to propagating your domain reputation to handling incoming emails using IPv6.

I just had the case last week when a customer account go abused and dropped their domain reputation to 0 for GMail/GSuite. Nothing worked until I made outgoing emails connection “icmp unreachable” thru IPv6.

Example with ipfw.

ipfw add [rule number] reject ip6 from me6 to any 25
ipfw add [rule number] reject ip6 from me6 to any 587

Good luck.

9

My company uses G Suite Business, and I've seen the exact same thing with my cron jobs, and it's getting worse over time. I have taken the following steps:

- Use the smtp relay address that Google provides
- Whitelisted my server's IP addresses to relay email
- Added my server's IPv6 and v4 addresses to the Email whitelist section in the admin console

And yet, the problem persists. Emails going to SPAM, getting the red label that you talked about, etc. And it's inconsistent, too - one day the cron output from Server A comes through without issue, the next day it's labeled as dangerous.

It's very, very frustrating.

10

This was recently thrashed out in the MailOp mailing list, https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop.

I can point you to the beginning on the email chain, but it became a mess - https://chilli.nosignal.org/cgi-bin/mailman/private/mailop/2019-October/014854.html (And yes, please ignore the bad certificate……)

The link is private, so you’ll have to sign up first. The ‘google’ insights come from the author Brandon Long.

From Google’s PoV, there wasn’t enough good signals from his emails being sent and his messages being acted upon within his mailbox for Google to determine it legitimate. Google watches everything, from the moment the email touches their edge until the email is ultimately disposed. Everything in-between provides signals to Google if that email was something that the receiver wanted. Think of things like never reading the email and auto-filtering it (tagging, I guess for gmail), for example, are negative signals.

I suspect that by changing your 5321.MailFrom, you changed the signal calculus, for now. I bet in a bit, provided that you don’t change any other behaviors, that these emails will eventually be rejected too.

This is done by all the big players, but Microsoft is the most aggressive.

The recommendations for this guy were to have his email relayed through a bigger company before heading out to Google, especially if he wasn’t going to dump OVH. For example, OVH maintains email relays that have way better reputations than the space used by customers (this was a recommendation by someone else, I have no clue personally).

Also, free is free. If you pay for G-Suite, then the admin gets a LOT of extra bonuses that anyone would expect out of a paid mailbox. I don’t know about G-Suites (wouldn’t touch the stuff personally), but you can get a O365-hosted exchange mailbox for like $5/month these days with all the aliases you need and all the post-processing transport rules you want. In line with the paid Microsoft mailbox – an email does not get delivered for no reason except in the rarest of cases. The same is not true with the free mailboxes hosted by Microsoft or Google.

And if having an account opened with one of the “big guys” ruffles your feathers the wrong way, try a smaller paid provider. I’ve used Hushmail personally for the last 8-10 years.

Some other pieces of insight provided by Brandon in that thread that are counter-points to your suggestions –

A lot of email that they tag as spam or straight-out block in the same way yours are now is due to 419 scams and spearfishing. Both are low-volume, may not contain links and can be sent text, so they share many of the same signals that your emails do today or that you propose as getting a free pass.

TL;DR for this guy - his domain is a free one off of eu.org, he’s coming out of OVH network space and he sends a few messages a day, so the OVH signaling eventually won. As a paraphrase from that thread, 1 out of 10000 messages aren’t spam from that netblock in Google’s eyes. And sign-up for the MailOp list if you want to discuss this further in a better forum.

Good Luck!

11

I use my own personal domain name for various UNIX stuff, including sending
log-related things to myself out of cron, which end up in my own Gmail.com
account, either directly, or through forwarding (w/o SRS). (I do not use G
Suite for my own domain name, for obvious reasons; just the consumer-based
gmail.com email address from the old times of invitation-based
registrations.)

Too bad you don't use G Suite as it allows you to notate incoming relays where Gmail does not. It is possible that authenticating (logging in as you@gmail.com) would help you get them delivered to your INBOX.

A couple of months ago, I setup some new scripts that would send me new
nightly emails. It's all plain text, but had a few dozen of domain names
present (it's logs). Absolutely no links, just plenty of domains which I
don't control. So, Gmail has been presenting most of these messages with
their red warning label that the email contains malicious links, even
though all of these emails contained zero links, zero URLs to any of these
unknown domain names, zero URL schemes, zero "http://", zero "https://"
etc. You get the idea.

Many an MUA would convert the anything.knowntld (and other) strings (some don't even check for known TLDs) into clickable links if no adjacent URL was present so it seems that so far as Gmail is concerned those strings are something you might eventually be able to click thus they are URLs.

Since about a few weeks ago, I am now seeing at least a 95% rejection rate
for my domain name, for ALL email, including the forwards.

My experience says that: their system has learned that your system(s) continued to send messages that their user (yes you, but they don't know that) did not want, i.e., you left it marked as SPAM or deleted it without reading the message, or at least not enough was noted as not SPAM *and* read (aka displayed, and not for half a second either) so as to influence their AI, which makes mistakes and will never correct them if not fed correcting info.

/mark

12

You’re assuming that IPv6 is at fault, but as I’ve already mentioned, if I change the From and MAIL FROM to one of the other domains with a DNS zone similar to the primary one with crontab-acquired “very low reputation”, without changing anything else, then the identical messages do get through at the SMTP stage — and show up directly in Inbox — i.e., don’t even end up in the Spam folder, either.

So, sorry, but I’m not going to go around blocking my IPv6 for no reason.

C.

13

[…]

I suspect that by changing your 5321.MailFrom, you changed the signal calculus, for now. I bet in a bit, provided that you don’t change any other behaviors, that these emails will eventually be rejected too.

Of course. This is just a tell-tale about sending yourself the output of crontab runs towards Gmail — apparently, it may result in the whole domain becoming blacklisted.

[…]

This is done by all the big players, but Microsoft is the most aggressive.

Microsoft and Outlook are kind of irrelevant nowadays. Even the big enterprise companies are often on G Suite nowadays.

[…]

Also, free is free. If you pay for G-Suite, then the admin gets a LOT of extra bonuses that anyone would expect out of a paid mailbox. I don’t know about G-Suites (wouldn’t touch the stuff personally), but you can get a O365-hosted exchange mailbox for like $5/month these days with all the aliases you need and all the post-processing transport rules you want. In line with the paid Microsoft mailbox – an email does not get delivered for no reason except in the rarest of cases. The same is not true with the free mailboxes hosted by Microsoft or Google.

You’re ignoring several issues here:

  • First of all, it doesn’t seem like acceptance rules are different between Gmail and G Suite. TBH, I think that’s actually a good thing, because it means that there’s potentially a higher likelihood that if you can see the message in your own Gmail, then so could your business partners in their G Suite.

  • I am actually not just a free used of Gmail, but a paying customer. They’ve tricked me into believing that I’ll have unlimited storage space; yet then the quota stopped growing at 15GB, but my mailing list archives did not, so, now I’m forced to pay 1,99 USD/mo for continued ability to receive the mail.

  • You’re assuming that all those controls within G Suite actually work. It’s been mentioned elsewhere in this thread that they don’t actually seem to work, after all. See https://mailman.nanog.org/pipermail/nanog/2019-October/103842.html. Not to mention that the bigger issue is that G Suite has a pretty good monopoly on corporate mail nowadays, so, even though I could rather easily abandon my own Gmail account, I cannot quite stop dealing with other people’s G Suite accounts any time soon.

C.

14

[ Again, just commenting on one point. ]

My experience says that: their system has learned that your system(s)
continued to send messages that their user (yes you, but they don't know
that) did not want, i.e., you left it marked as SPAM or deleted it without
reading the message, or at least not enough was noted as not SPAM *and* read
(aka displayed, and not for half a second either) so as to influence their
AI, which makes mistakes and will never correct them if not fed correcting
info.

[ Note that the proper term is "spam": it's not an acronym and should
never be fully capitalized. ]

It is a worst practice in mail systems engineering to allow input from
putative users into any decision-making process *absent* manual review
of each piece of data by very clueful humans, e.g., curated abuse reports.

Consider: either that input is coming from the person who owns the
account or it's not.

If it's coming from a person, then there is an extremely high probability that
it's coming from someone who doesn't have the slightest idea what they're
doing. (If users, en masse, knew what they were doing and were even
modestly diligent, then spam and phishing would not be serious problems.
But they are; they flourish because users are careless, lazy, stupid, etc.
They've been proving this daily by the hundreds of millions for decades.)
Systems which do this are built on the laugable assumption that the
aggregate opinions of a hundred million idiots are somehow magically
more valid than the opinions of one.

If it's not coming from a person, then it's coming from the new owner
of the account. That new owner is hostile by definition, so allowing
input from them is an even worse idea than allowing input from users,
who may only be hostile most of the time. (Keep in mind that email
accounts are compromised by the billions -- e.g., Yahoo -- and that
systems are likewise so. And of course anyone who has compromised a
system can avail themselves of any/all email credentials stored on or
traversing that system. Those who doubt the scale on which both are
happening are invited to peruse the darknet marketplace of their choice
and observe that accounts/systems are for sale in bulk quantities from
a wide variety of sellers.)

---rsk

15

Please post your password to nanog@. Consider: either we’re all benign, or we’re not. And if we’re not, either we’re too lazy to read all the messages to the list, or we’re willing to rubber-hose the password out of you. Posting your password to the list is the most logical way to avoid the hose.

You do want to avoid the hose, don’t you? :wink:

Damian

16

I thought we were all educated people on this ML. Please avoid sophism.

17

Their system, their rules. Their size makes it all but a certainty that plenty of automation will be involved and their modus vivendi is that they be AIs, no matter what you or I might think. Further they do have humans in the loop, their user individually and their users collectively which want messages from others not directly in the loop it just is not quite how we might like it to be when things subjectively "are wrong", as in this case.

/mark

18

Hi,

This is not an assumption, it is my experience.

Sorry it didn’t fit your case.

19

��� Hi,

��� This is not an assumption, it is my experience.

Mine as well. My mail server's PTR records are identical for IPv4 and IPv6. IPv6 fails and IPv4 is fine. I disabled IPv6 for gmail.com.