Another tip for domains that don't send email: set a null mx record. This explicitly tells sending servers that your domain does not receive mail, and isn't just broken. Create an mx record priority 0, with a dot (.) for the value.
I'd like to add that this is because with no MX record the MTA may still attempt to deliver email to the IP from the domain A/AAAA record.
This is a lesser known feature of SMTP, it is specified in rfc5321 section 5:
If an empty list of MXs is returned,
the address is treated as if it was associated with an implicit MX
RR, with a preference of 0, pointing to that host.
>Another tip for domains that don't send email: set a null mx record. This explicitly tells sending servers that your domain does not receive mail...
Most of my domains don't send mail, however I do prefer to receive mail to the standard aliases (webmaster@, abuse@, etc) and often a catchall. Even for domains I've got parked and awaiting deployment.
Am I thinking too much, not enough, or is the opening phrase of the advice meant to be "Another tip for domains that don't intend to receive email"?
Oh wow, TIL. Does that work? It says the draft expired in 2014.
"[T]he SMTP client first looks up a DNS MX RR and if that is not found it falls back to looking up a DNS A or AAAA RR.
Many domains do not accept email, but do have A or AAAA records. If they have no MX records, senders will attempt to deliver mail to those A or AAAA records. "
Nothing like a good opt-out to make you feel like you're signing up to a Do Not Call list.
I guess we can't blame email too much, it's an incredibly old protocol. But at some point, some quirks have got to be worth rethinking, right?
Maybe we can deprecate the fallback from MX to A/AAAA for the next, oh I don't know, two hundred years before we can move on.
I did run my own server with properly setup DKIM and SPF and all. Gmail and Hotmail would still mark my emails as spam whenever they felt like it. Sometimes, even for emails that were a reply to an email coming from their own servers. I almost lost a client because of that once. Maybe it was because of the IP range (I was in a cheap Scaleway server)?
At some point I just accepted reality and paid for a third party email server (Runbox) but the situation with the monopoly of these giants marking as spam anything they don't manage made just really sad.
HN often talks about running your own email server like it's this wonderful example of a decentralized service that anybody can take part in. That's kind of true, but there's this whole other aspect that gets ignored.
Email is not this wonderful, democratic system of equals. Email is more like a school playground where a handful of enormous bullies control everything. You either learn to work with them, cater to their demands, or you'll never be able to visit large sections of the playground.
Gmail is relatively easy to appease, mostly just configuring your nameservers and email headers to Google's standards, then slowly building "reputation". Hotmail will block an entire domain (IP?) randomly every few years, then you fill out one of their infamous forms and pray they unblock you quickly. Yahoo (technically Verizon now) is even more difficult to work with, they block you out of nowhere and I've never figured out how to appeal. Seems like they do some kind of ratelimiting for smaller email servers, I don't know if it's some kind of punishment or just by default.
Then there's the dozen other major email providers. Some easy to work with, others not so much.
> Gmail is relatively easy to appease, mostly just configuring your nameservers and email headers to Google's standards,...
Do you mean SPF, DKIM and DMARC or something else? I will check the records again, just in case, but afaik they are perfect and Gmail still often classifies my mail as spam. It is annoying as hell (and your comparison to playground bullies is very apt).
SPF, DKIM and DMARC, but there's other factors as well. Gmail will usually tell you exactly why something was flagged spam. If you go to "Show Original" SPF, DKIM and DMARC should all show "PASS" if everything is configured correctly.
Hmm, not always. I've just setup my own mail server, I PASS SPF, DMARC and DKIM according to Gmail headers and yet my email is still classified as spam, as determined by "Gmail magic". I believe this is because the domain is relatively new and Gmail treats you as guilty until you prove your innocence (ie. Enough people mark my emails as not spam)
"Gmail and Hotmail would still mark my emails as spam whenever they felt like it. Sometimes, even for emails that were a reply to an email coming from their own servers. I almost lost a client because of that once. Maybe it was because of the IP range (I was in a cheap Scaleway server)?"
Like you, I have properly set up DKIM and SPF and blah blah.
I also have a 12 year "clean" IP from he.net.
Sometimes gmail will mark something I send as spam that is, like yours, a reply to their user and is a recipient that I have a years long history of corresponding with.
That's frustrating but what really makes me angry is that there is no indication given that this occurs. Why can't google send a proper bounce or warning message ... or even an error code that I could at least see in my logs ?
In the arms race between spammers and (providers) are error indications exploited in some way ?
That's so frustrating. The biggest AI company in the world and their anti-spam system is so stupid? Really lowers your expectations on state of the art tech.
They're not stupid. It's part of a sophisticated long-term strategy. They deliberately make things complicated to maintain so that you give up and use their services. This strategy is working. Many small and mid-sized businesses today gave up on email maintenance altogether and outsourced it to either Google or Microsoft. Many individuals end up using Gmail, because it seems more "reliable" now that so many other people are on Gmail.
I have a similar impression from dealing with deliverability issues.
Relevant providers usually just require a few days of sending legit mail and occasionally asking to be unblocked, but there seems to be no chance getting into the inbox on GMail and Outlook with a "too low" sending volume.
If it wasn't for regulations, I would have already given up and switched to GMail.
> I did run my own server with properly setup DKIM and SPF and all
Ah. But the million dollar question is WHERE did you run that server?
If you ran it off your home connection then there's your problem.
Pretty much everyone on the receiving end (ISPs, corporates etc.) will either block or assign a low reputation to home ISP IP ranges.
So basically you can do all the DKIM and SPF you like on your home MX server, but you're going to have the high hurdle of your untrustworthy IP range to fix first !
EDIT: Just saw you said "cheap Scaleway" ... well that's a similar sort of problem. Perceived "spammer friendly" hosts tend to be assigned a low rep in spam scanning. You won't be auto-blocked like residential IPs are, just a low rep set. (for any lawyers reading, I am NOT making any comment on Scaleway here, I am speaking in general terms !)
I have one domain that uses Google's G suite. But sometimes when I send mail from this domain to a gmail address it gets marked as spam. It doesn't leave Google's network or cloud of servers, but they flag it.
My experience is similar. I’ve been running my own for about 12 years, for my own own personal email + a few other individuals and a couple of very small companies. I also use postfix + Dovecot (I'm impressed by the quality of those programs). I replaced Spamassassin a few years ago by a handful of procmail rules, which reduces CPU usage a bit.
My setup handles a couple of hundred email per day, and I rarely have to pay any attention to it.
That was a good article, and your site seems to have a lot of useful information. Does it have an RSS feed?
Thanks for the suggestion. A dozen or so spam emails still get through to my inbox every day. But that’s still better than using something like Gmail and missing real emails because of false positives.
Thanks for sharing. I keep getting closer and closer to wanting to stand up my own mail server, but research always leads to "it's not worth it" posts, and it's nice to have a realistic point of view that is also pretty up to date. I also didn't know about mail-tester.com, which I'm adding to my toolbox.
It is not worth it if you need to send marketing campaigns in hundreds of mails per day or transactional email in hundreds that really needs to be delivered to corporate email boxes of "joe@megacorp.com".
For my personal e-mail most of the time I am just on receiving end. For receiving e-mail I don't see much problem. For receiving email only you don't need to setup any of the DKIM/SPF/DMARC stuff.
Occasionally I send some e-mail out but you are not going to be marked as spammer if you sent 1 e-mail a week from your IP address. Most likely it will be delivered and accepted by whatever provider receiving person is using, might be marked as spam - but well you probably can call up that person and ask them to check spam folder.
For spam - don't put your email into every web page that asks you to do it and you should be good. My gmail account that is now ~20 years old started to receive loads of spam only last year when email lists from various leaks started to go around internet. My other email accounts that get loads of spam are on shitty providers that send their own spam.
The realization that my Gmail is nearing 20 years old was not a reminder I was prepared for today. If it's that old, how old I must be now. :-\
I'm similar to your boat, so this is just one more reason I'm going to give it a go, I think. It'll take time to move the stuff I care about off Gmail anyway, so I can sort out the spam stuff as I go. And I'd really like the flexibility of controlling my mail in ways I can't really right now with a hosted provider anyway.
For a time i DID host my own emails. Out of my house, with a business line and static IP.
Google Apps was free at the time and I registered my domain and over time that has become the easiest (its still free for me because i grandfathered in back in like...2014.)
With that said, more and more I want my own privacy and control. And VPS services are cheap. Im leaning toward moving to Mail in a Box or something similar to what OP talked about in his blog. Its really just me and my wife so not much volume. Could probably host it on Linode no problem.
Shameless plug -- for people who want to automate this process, a guide that does this with Pulumi and SES in particular (while leaving space for an cloud-external self-hosted SMTP server):
I'm a big fan of Pulumi and have used this setup for 2 projects so far (it's part of a bigger framework of repeatable infra for SaaS projects I've been hacking on), and I can say it definitely delivers (pun intended).
Code in the blog post might be a little stale -- I've just put an update in there to note that SES recently changed to using DKIM records primarily for domain verification, so the aws.ses.DomainIdentityVerification (Pulumi object) is no longer needed. One of the nice things about using Pulumi to do this is that since AWS allows you to use pre-made DKIM keys now, I read one right off disk (look into how to make a dkim key[0], it's interesting), and use it for both my personal email (I run maddy[1]) tied to the domain and SES.
FWIW, there's also BIMI[1], an industry standard that attempts to increase the wide adoption of email authentication with brand-specific indicators[2] next to properly authenticated messages & requires senders to publish these SPF, DKIM, and DMARC records.
Controversial and technically dubious. The whole premise of the BIMI standard hinges on the adoption of X.509 certificate authorities which will issue certificates (called “VMC”) for a graphical logotype. However, the laws regarding trademarks are restricted in many scopes; legal juristictions and business area (trademark “class”), to name a couple. No VMC issuer could really do anything sensible with similar trademarks which are registered by multiple entities in different juristictions and/or different business areas. The best-case scenario is that huge players like FAANG, CNN, etc. get VMC certificates and the rest of us get nothing. Which is, I guess, why the big players are toying with the idea and are, slowly, tentatively, pushing for it.
Thousands and thousands of dollars for a BIMI certificate is just ludicrous.
Certificates are certificates are certificates.
They're 1KB files full of random numbers. What you use them for shouldn't alter their price... which ought to be zero.
The facts that a handful of near-monopolistic CAs have managed to control protocols, set standards, and generally funnel money into their coffers that need not have been spent at all is one of the worst examples of naked capitalistic greed I have ever seen.
From what I can see, it is the very same CAs who used to peddle normal certificates (and are currently mostly failing to sell EV certificates) who are now desperately trying to save their business by pivoting to convince businesses that VMC certificates are the new big thing.
> Thousands and thousands of dollars for a BIMI certificate is just ludicrous.
VMCs are between 800 and 1500 USD
> Certificates are certificates are certificates.
The x509 certificate is, but the process to verify your ownership of the supplied 'mark' (your logo) is not.
The VMC issuing procedure is very involved, it requires the CA to confirm with your local trademark office that your company owns the trademark. For VMC the CA is also required to verify (by phone) that the person who requested the VMC does indeed work for the company, they are also required to verify that the person who requested to certificate is allowed by the organization to do so.
The CAs are actually doing work here, there are real costs.
FWIW: Digicert currently sells VMC for a discounted price of 800USD, which they claim is not even economically viable. Without the discount expect VMCs to be ~1500USD.
No public database with all trademarks exists. And even if it does: who is going to maintain that public database? And how would you be able to trust said database?
How will you proof that you really are the legal owner of the logo in that database?
Again, the 'renew' click is not why this is expensive. It's just a cryptographic function that signs a bunch of data, Lets Encrypt has long proven that certificates can be created free-of-charge.
However, having a human verifying that everything checks out is the expensive part. Having that human work in an environment, following procedures that passes public audits is expensive.
None of the trademark offices is going to do your trademark validations for free. No-one is going to staff a call-centre for free.
You are right about the technical part of creating a certificate being trivially easy, but I believe you truly underestimate the costs of running a CA that is capable of delivering VMCs.
Maybe that some company can do it for less than the current prices, I don't know. Competition will show that eventually.
But if you truly think that you can do it for less than the current CAs, then start a CA yourself and start competing.
You need to be registered with the national patent and trademark office in one of six participating countries. These databases are financed and maintained almost entirely by the brand owners. You as a brand owner need to do your own research to make sure that your logo is original, then you pay to be entered into the database, then you need to continuously audit any new entries into these databases to make sure that they don't infringe on your existing trademark.
So you provide the CA with an trademark ID number that they can look up and they verify that you represent the company that owns the trademark. It's like 10 minutes on top of the existing EV process, but it's more than double the price.
At Mailhardener we built a BIMI validator tool [0], we also work with the AuthIndicators Working Group (the authors of the BIMI standard) to refine the standard.
Please be aware that the BIMI standard is far from finished. There are some serious documentation ambiguities that pose a risk to how well BIMI might function, and how email providers will adopt the technology.
Obviously, the CAs are now pushing hard for BIMI, since VMCs are a new source of recurring business for them.
In my personal opinion I think that BIMI is pushed to early, which could leave brands attempting to adopt BIMI disappointed. Obtaining a VMC is quite involved (and expensive) for small businesses, and the potential benefits are (currently) quite small, as not many email service providers support BIMI.
Don't get me wrong though, I think BIMI has a great goal of pushing the DMARC adoption rate (remember, you must implement strict DMARC for BIMI to work). I just don't see BIMI offering any reliable brand exposure for the coming years. Only time will tell I guess.
The IETF contains some really questionable propositions lately, not sure what to make of this, but it doesn't seem to solve a problem.
What does this provide in contrast to DKIM? Perhaps there are logistical problems with DKIM, but I wouldn't know. I wonder why it isn't more widely spread in the first place. The receiver can determine the requirement, which is the best solution imho.
For Google it seems to be more of a marketing issue than security. Furthermore provider could track domain requests. That isn't just suspicious, this is almost malicious. There is a reason mail clients don't display pictures outright.
BIMI is really cool. I played with it, setup the appropriate DNS records, created an SVG logo and ... well that's about where it stopped as registering a trademark is a bit of a step too far for getting all of this to work.
It does pass the checks though [1], but does not really work because of the missing certificate which you can for example get at DigiCert [2] once your trademark has been approved..
The domains I configured it for are: antwise.com and vimalin.com
so you can see at [1] how it looks like when it is supposed to work.
> Email is at the centre of most of life and business, and so we must ensure it is trustworthy and authentic.
While it‘s certainly true, to me it‘s missing the most important aspect for practical use here: Email must be deliverable.
I only started down the rabbit hole of email authentication long after we discovered important client and outreach mail never reached its intended sender.
Back then, most of this stuff was hidden in some layers of documentation in G Suite instead of in your face after setting up your custom domain (where it would belong imho).
If you‘re setting up your freshly bought startup domain, please do yourself a favor and follow this or a similar guide from A-Z.
To add to this, also remember to do this for anything else you might use to send email: servers/cloud services providers you use for transactional emails, any provider you might use for marketing emails, etc.
If you're looking for an Ansible playbook to setup a mail server with DKIM, DMARC and SPF check out Sovereign on github.
https://github.com/sovereign/sovereign
The project could use some help to get up to speed with Debian 11 "bullseye".
The bonus in there is a very valuable piece of advice. Thought I have good understanding of email authenticity measures, it never occurred to me that my domains that don't send out any emails should advertise the block all policy.
SPF attests that the mail was sent by an IP address that is authorized to send for a certain domain.
DKIM attests that various headers, usually including the From: address, are approved by the owner of the domain.
SPF prevents random people on the internet sending mail that purports to come from your domain. DKIM prevents people inserting forged headers into emails. These goals are orthogonal; neither is a stronger signal than the other.
SPF has many issues. One of which is that including external email providers will include large blocks of IP addresses to your SPF that you don't control. IP spoofing is also a thing. SPF also breaks horribly with forwarding email services (like catch-all inboxes). The SPF lookup limit of 10 additional lookups is way too low for most modern email setups.
SPF is just not very robust, and badly understood by most people (leading to many configuration mistakes). Hence that receivers cannot get much trust from the SPF outcome alone.
DKIM does send a "stronger signal" in the sense that it is more robust than SPF, and contains cryptographic proof. He who owns the private key is trusted by the domain (who published the public key) to send email on behalf of the domain. DKIM also does not suffer from the forwarding issue that SPF has.
As a result, a valid DKIM signature often does result in a higher trust rating by the receiver, than just an SPF pass.
DKIM + DMARC enforcement is the most reliable way to go. Obviously you still want to setup SPF as good as you can (until you start hitting size and lookup limits).
Disclaimer: I'm the founder of Mailhardener, an e-mail hardening platform.
IP spoofing is pretty hard to do with 2-way communication channels like TCP and a lot harder with TLS too. usually IP spoofing only works meaningfully with UDP and protocols that allow for transactions were you don't need replies from the target. to pull this of you would need to do BGP hijacking in order to receive return traffic which is a little above the efforts spammers will make.
Are you saying that my core point is wrong, that SPF and DKIM serve orthogonal purposes, and that it's incorrect that one is somehow "more robust" than the other?
I can't see what I said that you disagree with, except that somehow "DKIM and DMARC enforcement is the most reliable way to go". You've just said that you disagree, without providing much evidence.
You need DKIM if you want deliverability to (principally) GMail. You need SPF if you don't want your mailserver blacklisted. You don't need DMARC, really, unless you send so many emails that you need some kind of automated reporting infrastructure.
> SPF attests that the mail was sent by an IP address that is authorized to send for a certain domain.
Yes, but SPF ties the IP address only to the Sender address (MAIL FROM) - so utterly useless in stopping phishing emails for example, if attackers can use said domain in From address.
> DKIM attests that various headers, usually including the From: address, are approved by the owner of the domain.
As the From address (and implicitly domain) is the one people see (and trust) DKIM in a way should matter more.
Also, misconfigured SPF records happen from time to time (on big domains actually), where bad actors can then take advantage of them and abuse delivery, much harder for a DKIM private key to end up in the wrong hands.
ive seen DKIM signed mails that did not have their key in DNS fail verification without being marked as spam by google. and i have (not) received mails because of failing spf validation made delivery fail silently so i think you are quite right in that regard at least for gmails interpretation of those signals...
Gmail has a blackbox ML that takes SPF and DKIM into account, but with a certain scoring that might not matter that much compared to said IPs reputation in their system.
If we reach a point where the spaminess of an email can be correctly determined based on the reputation of the domain from which it was purported to be sent (after some cryptographic and policy checks), then is the problem of spam equivalent to the problem of bootstrapping trust for new domains?
I suppose technically there is the problem that a domain with a good reputation can be hacked or bought, but the costs of that happening should be orders of magnitude higher than the cost of buying a new domain.
Requiring new mail-serving domains to post a bond for good behaviour seems like it would similarly increase the costs of buying a good reputation, but the problem is coming up with a fair process to decide when a bond should be forfeited.
I'd be reluctant to give the ITU any control over the global email network, so instead perhaps there could be some system for countries to delegate the bond-forfeiting power to companies of their choosing, with each country having a vote in proportion to the cube root of their population size. All existing domains would be grandfathered in to not need bonds, so the potential damage from abuse of this system should be minimal.
Alternatively, just create electronic stamps* for email; cheap enough that it
doesn't enter the minds of those that send a few emails per day but expensive enough to ruin the profit margins of spammers.
* I'm thinking something similar to SSL: write email, create hash (one hash per recipient), submit signing request for hash, CA signs request and returns stamp (certificate), stamp is transmitted along with email. Additionally, allow people to give each other (or sites/newsletters they would like to receive) 'rubber stamps', allowing them to create (free) stamps for emails addressed to each other... And since no idea of mine is ever original I'm certain several smarter people have already written multiple RFC's on this and the idea never took off because of something I'm overlooking at the moment. Ah well.
> but expensive enough to ruin the profit margins of spammers.
And mailing list operators, and charities, and free software projects, and...
To be fair, your "rubber stamps" idea does mitigate this problem a lot, but I think the issue still remains that it's hard to get email providers to agree to drop emails from senders that haven't upgraded to this new system, as well as agreeing on a globally consistent set of CAs who are trusted to not act as a cartel.
It's even more difficult to get all existing pieces of email sending infrastructure to simultaneously upgrade to support users entering their rubber stamps, not to mention the security concerns of dealing with the funds needed to buy the non-rubber stamps. Perhaps the big email providers could force everyone to implement this as of a certain flag day, but I imagine the response to this weird new tax would be less than enthusiastic.
Slight off topic: how do sites know an email address from one of those “temporary email service providers” is a fake/short-term email address?
There’s a number of things I like to kick the tires on without giving my actual email. So I’ll use a temp email and more times than not it’s detected to be as such. How is that known?
I'm running one such service (inboxes.app), and it often boils down to five really obvious give-aways:
1. The server accepts emails to all emails addresses.
If I want to check if a service is a temporary email service or not, hit a few random addresses. If they're accepted and do not bounce that's definitley a little suspect.
2. known addresses
If I were going to prevent temp emails, I'd look out for temp-mail type domains. This is a game of cat and mouse though, but blocking the known ones is probably enough to wipe out 80% of these accounts.
3. Low open rate
Speaks for its self, especially with disposable emails.
4. random addresses
Usually emails are `first.last`, so getting f00b4rb4z@temporaryemailsite stands out
5. Your smtp server IPs are made available via your MX records
Offer (paid?) users a new domain to point their domains to with an IP which won't have been blacklisted by providers.
Ways of defending against boil down to: bounce addresses that haven't yet been created, allow users to use their own domains, offer random names rather than random strings as addresses. These are all areas I'm working on over time, but with all side projects it takes time :p
Side question if anyone happens to know and wants to tell... is there anything going on with GMAIL's spam filtering in the last year? I've been getting 1 to 2 daily misses with brain=dead-obvious spam messages hitting my inbox, for about a year now. I've been using gmail since the invite-only-beta and it was never, ever this bad before. I just had 4 spam messages in a row like "C0NgRAtuLationS1! yoU Won!" and the like and marking them as spam hasn't done anything positive in the last year.
Not a full answer but a partial from the other side of the fence: setting up my website and domain-name to send emails fully self-hosted - I finished setting up DKIM, DMARC and SFP on my brand-new, not-publicly-blacklisted IP address, and finally... Time to send some test emails out. Results:
[-] Outlook rejected my emails: vague reason given, but the fact is, Outlook rejects entire IP address allocations (for example, DigialOcean's entire VPS network), even individual DigitalOcean IPs which aren't publicly blacklisted. There is a proper process in-place for small upstarts like myself to get de-listed, but its not a given right that the process will favor the applicant.
[-] Gmail. Email received. No problem, no spam label. And I'm quite thankful to Google/Gmail for accepting my mail based on metrics other than IP addresses alone.
So while I can't answer why they're allowing obvious spam through their filter, I can say that at the present moment they are not as ruthless in blocking /16 networks (or whatever size) as some providers based on IP addresses alone (e.g., Outlook, E365).
Outlook is uniquely horrible. I’m this close to changing my signup form to proactively recommend third-party login whenever it detects someone entering an outlook/hotmail address.
Subnet level blocking is certainly a thing, but I’m setting up new domains routinely and Outlook works fine. Even with cheap / free domains.
Lack of verbose feedback / delivery errors makes debugging issues in your own really hard, so you might want to contact someone who knows a thing or two about emails for help.
> I’m setting up new domains routinely and Outlook works fine. Even with cheap / free domains.
You say domains
To ensure we're speaking to, not past, each other it will be essential to clarify one thing: what do you mean precisely by "setting up new domains", "cheap/ free domains"?
Domains, I'm not interested in chatting about those, but IP address space, then yes I'm all ears. If you can reliably setup new, fully self-hosted mail servers (i.e., not using SendGrid, MailGun etc), with port 25 open for both send and receive, on cheap, accessible VPS servers...
And...
Send email messages from those new servers directly to your Outlook's inbox.
Then you Good Sir, have the secret sauce.
*Caveat: AWS does have a lot of un-blacklisted IP address space, however, port 25 remains blocked for new customers without a "valid use-case" and an unspecified amount of payment history. Similar for Azure. So if your methodology is built around AWS/Azure then kudos, but for new-comers, it would be unrealistic from my experience to suggest the path forward will be without major hurdles and/or delays.
The lack of verbose feedback in the rejection reason from Outlook was, as I see it, a feature not a bug. A bit of friction to turn away the undetermined ones. I followed the trail of clues from the top search results of the error message and eventually got my new IP unblocked through the proper channel. So... I have made it from bottom tier (outright rejection of emails), to 2nd-bottom tier (directly to spam/junk folder). Next: build domain/ip reputation ever so slowly.
Yeah, I didn't want to get stuck in details, but I'm setting up a couple of new mailservers (freshly registered domains, self-hosted VPS at cheap cloud providers (DO, Hetzner, OVH, AWS). And they do get delivered directly into Outlook, Google, Protonmail and other major providers after two days tops (some still employ greylisting for fresh IPs/domains).
There's no secret sauce but there is a myriad of ways to shoot yourself in the foot when setting up a mail server from scratch. E.g. many standard setups may try delivering email over IPv6 if available which currently is a terrible idea but might be completely opaque to the user. Obviously, the results will also depend on what is it that you're sending as everyone uses point-based (or even ML) systems nowadays and headers + contents can earn you either negative or positive points.
Oh boy, thanks for the detail, love it. You've probably been more fortunate than you can appreciate for now. I mean, I've not yet had a DO IP which wasn't listed blocked by SNDS (Outlook private block-lists): https://sendersupport.olc.protection.outlook.com/snds/ipStat...
Would be interested to hear from others who've acquired new DO IPs that weren't blocked on that list though. Change my view of the whole scenario.
Yeah gmail's spam filter quality has gone way down. I'm not getting daily misses, but monthly definitely (on an email that doesn't receive much spam at all).
What's more, though, is that there's actually a lot of false-positives. Even emails with people I've talked to in the past. Probably at least one false positive per week.
I need to periodically check gmail spam because almost all legit emails from small business (order confirmations, support requests etc) with dedicated email server fall in there. I explicitly mark them as not Spam but it's still not working. All of those servers use TLS, DKIM, SPF etc.
And my spam folder contains almost none actual spam letters, they just stop arriving a few years ago dispite my email address is very public.
Oh man that's such a common problem. I wish they'd use something like SES just to get around this problem, but hey. Even warming up emails does a lot to help you avoid the filter.
PGP is not meant to be run on the server, and therefore cannot make decisions about accepting or rejecting email.
Additionally, PGP relies on a "web of trust" managed directly by the user. There is no way for a server to automatically discover the key from $YOURBANK to decide if the email is legit; you need the bank to give you their fingerprint through a side-channel. _pka DNS records help, but is that simpler than SPF?
> PGP is not meant to be run on the server, and therefore cannot make decisions about accepting or rejecting email.
Why not? To check a signature you're using a public key of the other party, so nothing prevents you from adding a milter that would check incoming mail against a list of public keys you trust, without compromising/sharing any secret key material with the server. (As long as you trust your server.)
Anyway, point of PGP is that you can do the checks end to end between MUAs, not just between MTAs. Relying on DNS to distribute key material is not that great anyway, so PGP with keys shared via a better channel, is more secure. (Less convenient too, perhaps, depending on your MUA.)
I would recommended to take these steps right in the beginning, when you setup email for a domain (or even when just registering a domain with no intent to use it for email).
It's much less stressing to work with these when you don't yet have anything important going on.
Pay-each-recipient is a great use for a cryptocurrency. The incentives align senders to receivers without requiring all of our correspondence to be read by intermediate machines outside of our control to block nuisance email.
but then it makes email so environmentally unfriendly, may as well just send dead trees.
edit: interesting, didn't know the idea of arbitrarily expensive computations as adding costs to combat abuse was first presented in the early 90s: https://en.wikipedia.org/wiki/Cynthia_Dwork
I remember reading somewhere in the 90's that Bill Gates proposed this as a potential solution for spam.
If I remember correctly his proposal was to have the sender solve a cryptographic puzzle, which takes a bit of CPU time. For normal every day use you wouldn't notice, but this subtle costs and throughput limit would make spamming no longer economically viable.
A mailing list will _always_ break SPF. When forwarding messages to mailing list subscribers, the mailing list server will not match the host in the "From" header field.
One of SPF's shortcomings is that is doesn't act upon the `From` header at all, which is the name/address the recipient most often is shown. Instead, it acts upon the `Return-Path` header, which a well configured mailing list will set to a domain it controls.
I can see that argument, but - it's kinda a philosophical question about "who the sender is". Is the the person who typed out the text? or is it the server which transcribed that text into N new emails?
The ML server will verify the original authors SPF. The N recipients will verify the ML servers SPF - the chain (which matches the series of MTA's involved) is still verified end to end.
> It'll break when used together with DMARC's alignment checks.
Yea, DMARC is a much bigger issue for mailing lists, but that's no reason to say "A mailing list will _always_ break SPF" - a well configured* ML has no issues with SPF at all.
* And, yes - the definition of "well configured" had to change when SPF was introduced, that's of course annoying, but there has been many many years for ML operators to make these changes.
> The ML server will verify the original authors SPF. The N recipients will verify the ML servers SPF - the chain (which matches the series of MTA's involved) is still verified end to end.
The recipients have no way to check that the mailing list server has checked SPF/DKIM/DMARC. Mailing lists very rarely drop messages because of a failing SPF/DKIM/DMARC check.
ARC tries to fix this, but requires recipients to trust the mailing list server. Just using plain DKIM is much better, recipients can just treat ML-forwarded emails just like direct emails.
It does, but it’s more involved to setup so depending on how many different sources your domain is using you may not be able to use DKIM everywhere.
Additionally, DKIM keys need to be rotated periodically just like SSL. Many services like Sendgrid or ProtonMail will handle this for you now by setting up multiple CNAME records so they can rotate the keys for you, but it only works with that sender.
SPF helps to address the gaps. I totally agree that strict DMARC policy plus DKIM should be enough though.
You're far from your original claim that "SPF always breaks mailing-list". You are now at "there is a way to configure DMARC that breaks mailing-lists".
If you run a mailing list, you probably need to look into ARC:
> Authenticated Received Chain (ARC) is an email authentication system designed to allow an intermediate mail server like a mailing list or forwarding service to sign an email's original authentication results. This allows a receiving service to validate an email when the email's SPF and DKIM records are rendered invalid by an intermediate server's processing.[1]
https://tools.ietf.org/id/draft-delany-nullmx-02.html