Mail Sending Self-Test Platform


after our paper on mail sending configurations some time ago [1], we
now glued that together into a self-service site:

Even though some of you might have already seen this at the APNIC blog
or on other lists, i though i'd still drop it here as well, especially
given that it includes a full IPv6 readiness test for mail infra
(recursive DNS, authoritative DNS, and v6 mail delivery ;-)).

I'd be happy to hear your feedback, especially if things do not work as
expected (then, your test ID and ideally stored emails would be really
helpful, so i can double check what went wrong). More details below
(also see [2] for an overview of the tests we run).

Please note that setting up the tests (as we have to configure vhosts
for some MTA-STS cases etc.) takes some time on our site. The test-site
should periodically reload and provide the status. As we use JS for
that part, please reload it manually every few minutes if you block JS.

I would also be interested to hear what you think should be included in
addition to the current tests; Some of our plans for the future below
as well.

We already found some interesting bits, like Vultr having lame v6
delegation for their AuthNS servers' domain, making their domain and
rDNS non v6-resolvable [3], or mail-in-a-box using relaxed/simple for
DKIM, which breaks signature validity on long To: headers [4].

If you want me to block certain domains/IPv(4|6) ranges or ASes for the
service to keep your users from using it, please let me know and i will
implement it asap.

# Overview

The site tests a variety of parameters about mail sending (details:
[2]), by evaluating mails a user sends us in reply to a message that
can be requested on the site. Simply requesting a message on the site
does not start any evaluation or measurements, i.e., you have to send a
reply-all to a requested message to start the evaluation. After the
test, you can also delete the test or download a copy of your data, and
if you opted to do so, a copy of the emails as we received them.

# Method

Users send emails to us (in reply-all to a message we sent, for easier
usability).Target MXes on our site are either configured in a fully RFC
compliant way, or intentionally misconfigured in a common way (broken
DNSSEC for the domain, for example, or a broken DANE record, again, see
[2] for an overview). Thereby, we can determine the
configuration/support of features by the senders' MXes.

# Possible Harm

The worst thing that should happen to mail operators is finding
specific undeliverable mails (broken DNSSEC if DNSSEC is validated, v6
only DNS if no IPv6 DNS resolution is supported) in their mail queue,
or users receiving bounces (see FAQ on the page). So far the system has
been tested with several hundred mail-setups, without encountering
issues. If you encounter other unintended side-effects, please reach
out to or me directly off-list.

# Test details

Specifically, we are testing:
- v4 Mail sending
- v6 Mail sending
- Greylisting support

- Transport encryption (Plaintext/TLS cipher strength and version
support, opportunistic encryption)
- DANE support outbound, i.e., if you honor DANE
- MTA-STS support outbound, i.e., if you honor MTA-STS (incl. whether
you prefer DANE over MTA-STS, as it should be)
- Sending of TLS reports and whether these are standard compliant, also
providing a copy of the received report (Even though so far only three
operators delivered TLS-RPT).

- DNS resolvers used by the mailserver
- Whether the mailers' DNS servers resolve via IPv6
- Whether the mailers' DNS servers validate DNSSEC

- If all names relevant for email delivery resolve via IPv6
- If all names relevant for email delivery are DNSSEC signed

- Mailer-basics (ehlo=rdns=fcrdns), SPF (from/envelope)
- SPF policy size (by retrieving your policy, up to 20 referrals deep)
- DKIM (also checking key-sizes, algorithm mismatches and
canonicalization issues; These are surprisingly common, i.e., simple
being set breaking only in cases not caught by standard dkim testers)
- DMARC (policy validity, conformance of sent mails, displaying
implicit defaults)
- DMARC report deliverability by sending one(1) valid DMARC report for
a received email, including authorization of out-of-domain RUA/RUF
(also providing a copy of any bounces received)

- Full IPv6 readiness (join over three previous IPv6 tests)

- Whether any headers that should not be duplicated are duplicated
- Whether any mandatory headers are missing
- If From/Envelope match

# Planned tests

- evaluate outbound spam filtering
- evaluate received_from stripping
- test for inbound-link parsers
- cluster by provider (via MTA sets), not domain
- Add DANE checks for senders' MXes
- test for host-addresses in SPF prefixes
- add MTA-STS check for senders' domains
- add test for handling more than 5 MX with only the lowest-prio one(s)
being reachable
- requesting-domains' MX aspects (null MX, IPv4/6 addr in MX record,
non-reachable MX, broken MTA-STS/TLSA, broken DNSSEC, validation of our
setups SPF/DKIM (via DNS server logs, v6 resolvability), etc.).

With best regards,

[2] Email Delivery Report
[4] [DKIM] Use of relaxed/simple cannonicalization can lead to invalid DKIM signatures for long header lines · Issue #2239 · mail-in-a-box/mailinabox · GitHub