Setting Up a Mail Server That Doesn't End Up in Spam

"We set up Postfix and our emails are going to spam" is a sentence I hear often, and the fix is almost never in Postfix's configuration. Mail servers, Postfix, Exim, whatever you choose, are the easy part, they've been solved problems for decades. What actually determines whether an email lands in an inbox or a spam folder is a combination of DNS records that prove you're allowed to send mail for your domain, and a sending reputation that builds up (or doesn't) over time.

SPF: who's allowed to send as you

SPF (Sender Policy Framework) is a DNS TXT record listing which servers are allowed to send mail claiming to be from your domain:

example.com.  TXT  "v=spf1 ip4:203.0.113.10 -all"

This says: mail from example.com should only come from 203.0.113.10, and -all means anything else should be rejected outright by receiving servers. The most common SPF mistake isn't a missing record, it's an SPF record that's too permissive (~all everywhere, or includes services no longer in use) or, just as bad, a record so strict it breaks legitimate mail from a forwarding service nobody remembered to list.

DKIM: proving the message wasn't altered

DKIM (DomainKeys Identified Mail) signs outgoing messages with a private key, and publishes the matching public key in DNS so receiving servers can verify the signature wasn't tampered with in transit:

mail._domainkey.example.com.  TXT  "v=DKIM1; k=rsa; p=MIGfMA0GCSq..."

Most mail server software (OpenDKIM alongside Postfix, for example) generates this key pair for you, the work is publishing the public half correctly and making sure outgoing mail is actually getting signed, which is worth checking directly rather than assuming:

echo "test" | mail -s "DKIM test" [email protected]

then looking at the received message's headers for dkim=pass.

DMARC: telling receivers what to do if SPF or DKIM fail

DMARC ties SPF and DKIM together and tells receiving servers what to do with mail that fails both, and where to send reports about it:

_dmarc.example.com.  TXT  "v=DMARC1; p=quarantine; rua=mailto:[email protected]"

Starting with p=quarantine (send failures to spam rather than rejecting outright) rather than p=reject is the safer rollout, especially in the first weeks, since it gives you visibility into anything that's misconfigured without that mail simply vanishing. The rua reports are genuinely useful here, they show exactly which sources are sending mail as your domain and whether they're passing.

PTR records: the one DNS record you don't control... but your host does

A PTR record (reverse DNS) maps your server's IP address back to its hostname, and a lot of receiving mail servers check that this matches the hostname your server presents in its SMTP greeting. Unlike the other records, this one is set by your hosting provider, not in your own DNS zone, most VPS providers have a control panel option or support ticket for setting reverse DNS on an IP. If a server is sending mail as mail.example.com but its IP's PTR record resolves to something like vps-1234.providername.com, that mismatch alone is enough to get flagged by some filters.

The part that DNS records can't fix: reputation

Even with SPF, DKIM, DMARC, and PTR all correct, a brand-new IP address sending mail has no history, and some receiving servers treat unfamiliar senders more cautiously regardless of how correctly configured they are. This is where "warming up" matters: starting with low volume to addresses you're confident will engage with the email (open it, not mark it as spam), and increasing volume gradually rather than sending thousands of messages from a server that sent zero yesterday. A new mail server with perfect DNS and zero sending history still benefits from a slow start, reputation is built over days and weeks, not configured in a file.

Get the four records right, check that mail is actually being signed and aligning correctly using a tool like mail-tester.com, and give a new sending IP time to build a track record before relying on it for anything time-sensitive. That combination handles the vast majority of "why is our mail going to spam" cases, almost none of which turn out to be the mail server software itself.