Category Archives: Email

Trustworthy email: authentication using exim4, SPF, DKIM and DMARC

The email authentication technologies we’re about to implement are, according to the authentication authorities, all you need to be regarded as being from your domain when you send email, and someone else not being from your domain.  Effect: your emails can be considered trustworthy by email receivers who use these technologies. If they don’t use these technologies, they can’t tell.

At the very least, Google will be less likely to think your email is spam.

PTR record

A PTR record can be obtained from your host’s nameserver – it’s a reverse DNS record for your IP address. If the PTR record points at ec2-23-65-53-221.ap-southeast-2.compute.amazonaws.com rather than example.com (your domain), and you’re claiming to be sending mail from example.com, what’s the email recipient meant to think?

host 23.65.53.221

will tell you what the host for that IP is. Lodge a ticket with your hosting provider and get that PTR record changed to example.com. This might take about a day.

SPF record

Create a Sender Policy Framework record on your nameserver:

TXT @ "v=spf1 a mx -all"

This says “for my domain, I will only send email from IP addresses listed on the nameserver”.  Nameserver changes take time to propagate.

After your nameserver changes have propagated, you can go to https://dmarcian.com/spf-survey/ to check out if you got it right.

DKIM

DomainKeys Identified Mail is where things get more involved.  We’re doing this on a Debian Linux, like Ubuntu for exim4. We’re making our signing key 2048 bits, which is long enough to make life slightly unpleasant for us. Fortunately for you I’ve written a bash script that outputs the TXT record we need to create on the nameserver – because some nameservers (I’m looking at you, Gandi) can’t hold “long” strings – it’s broken into “small” strings:

sudo apt install openssl
cd /etc/exim4
sudo openssl genrsa -out dkim.private 2048
sudo openssl rsa -in dkim.private -out dkim.public -pubout -outform PEM
echo $(echo $(date -u +%Y%m%d && echo '.domainkey.example.com') | sed -e 's/[ ]//g' && echo $(echo ' TXT "v=DKIM1; p="' && echo $(grep 'PUBLIC KEY' -v dkim.public) | sed -e 's/[ ]//g' | fold -w200 | sed -e 's/\(.*\)/"\1"/g'))

which gives something like
20170419._domainkey TXT “v=DKIM1; p=” “MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvCNqU0Njd4YQ4e89T3FNc+uyOS2JwUqynGk7uwcSYHjIE2MGRuTxi56s4JgPKSnCVlBkJlUnXQHXFp2UGnLm8SADtjRMfWwpNxz6TmzXBpMnNZV1zvuoBBdcxh0Qg1TtCSACtWM6ehml0BmOHVA8Ippqj9iRlP2HMjuVMxZXewN9eJl”
“c6zsyOwQPvVKpJ+Rdvr+pPkDztAVTw7mNSeyy+TL6O/3L9sl7A19Yx8jLHKuGUh9LutVuv1VP16e7GwlnA3Zqn5C1jyY5Qvr2SEHZMcE3VzD7XKZtZWbpkGh+A5S15NrOH4k9tbVfNbjft6Y1jUJRTT+4DD0ZEVlr4zO+WQIDAQAB”

That all goes into one nameserver TXT record, spaces and all.  The world will join up the ” ” and get one big string. Note the bold number up there? That’s the selector. That a number needs to get larger with each new key.  Periodically you’re going to have to reissue your key because security.  You know what gets larger as time goes by?  The date.  Use the date.  If you screw up, use tomorrow’s date, etc.

Once you’ve got our public key out to the public via our public nameserver, we need to get exim to sign the payloads:

sudo nano conf.d/main/01_exim4-config_listmacrosdefs

After the line CONFDIR = /etc/exim4, add:

#DKIM loading
DKIM_CANON = relaxed
DKIM_DOMAIN = ${sender_address_domain}
DKIM_PRIVATE_KEY = CONFDIR/dkim.private
DKIM_SELECTOR = 20170419

and reload the mail server

sudo service exim4 restart

After an appropriate delay for nameserver propagation, use https://protodave.com/tools/dkim-key-checker/?selector=20170419&domain=example.com to check your work.
If that works out, mailto:check-auth@verifier.port25.com from example.com to ensure everything checks out:

echo -e "Test my DKIM plz\nMsg Body\n.\n\n" | mail -v check-auth@verifier.port25.com

DMARC

Domain-based Message Authentication, Reporting and Conformance is where the wheels can come off if you screwed anything up.  We’re going to set things up so that when you screw it up, computers scold you rather than putting your emails in the bin.

You will need to create two dmarc reporting accounts.  Servers will email you a (surprisingly detailed) report card on how you’re doing with your implementation. It’s best if these accounts are on the same domain, because technically you need to be or it’ll be ignored (Google will happily mail reports off-domain even if the other domain hasn’t said that’s okay).  Yours are dmarc_failures@example.com and dmarc_summary@example.com, according to the following nameserver entry:

_dmarc.example.com. 1800 IN TXT "v=DMARC1;p=none;pct=100;ruf=mailto:dmarc_failures@example.com;rua=mailto:dmarc_summary@example.com"

none is the consequence for screwing up. none is where we’ll start at, and see what the reporting records say to us.  After a while, you’ll be comfortable that everything is ticking along nicely, and you’ll up the consequent to quarantine (shove it in spam) or reject (burn it).

After your nameserver changes have propagated, you can go to https://dmarcian.com/dmarc-inspector/ to check out if you got it right.

As a human, to read the records sent to you, upload the files to https://dmarcian.com/dmarc-xml/

Making a captcha deamon for spamgourmet installations

For those of you following along at home, this is part of a cookbook style instruction set for getting spamgourmet going, but because of screwed up permission logic I can’t post this section there.

The captcha is for validating humanity when creating spamgourmet accounts. We’re going to limit what parts of the OS it can tromp over:

sudo useradd -c "captcha server for spamgourmet" -f -1 -M -r captcha
sudo /bin/mkdir -p /var/www-spamgourmet/captchasrv/
sudo chown -R captcha /usr/local/lib/spamgourmet/captchasrv/
sudo chown -R captcha /var/www-spamgourmet/captcha

Now we make our one-line shell script for running the daemon

sudo nano /etc/init.d/captcha.sh

#!/bin/sh
### BEGIN INIT INFO
# Provides:          captchasrv
# Required-Start:    $local_fs $network
# Required-Stop:     $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: captchasrv
# Description:       captcha daemon for spamgourmet
### END INIT INFO
sudo -u captcha perl /usr/local/lib/spamgourmet/captchasrv/captchasrv.pl &

Next we get it going

sudo chmod +x /etc/init.d/captcha.sh
sudo update-rc.d captcha.sh defaults

And check if that worked, there should be about four entries:

ps -aux | grep captc

Now the captcha server will start whenever the computer starts.

Install exim4 STARTTLS using a free LetsEncrypt certificate

Here we are on a Debian Linux, such as Ubuntu and we want to run a mail server. Exim4 is currently the most popular email server, but getting it up and working for free is a hassle – who wants to pay for a SSL certificate, on an ongoing basis? And then there’s the maintenance of the security of it – constant renewal, renouncing and re-installation of the certificates.

Wherever you see example.com, swap in your Fully Qualified Domain Name. That may be mail.example.com
It’s assumed you’re not logged in as root, but user ubuntu
Wherever you see 1.2.3.4, swap in your machine’s local IP address, from
ifconfig | grep "inet addr" | grep -v "127.0.0.1"

Security is all handled automatically by LetsEncrypt’s certbot. I’ll let you look that one up yourself. Run it up and get your certificate for example.com

Once you’ve got that handled, punch a hole in your firewall so that port 25 can get through from the outside world to your machine. Be aware: the outside world is filled full of botnets trying to hack into your machine.  After installing exim, keep an eye on the logs in /var/log/exim4/ for a while.

Let’s install exim4:
sudo apt-get install exim4
sudo dpkg-reconfigure exim4-config

  • pick “Internet site”
  • system mail name is example.com
  • IP address is 1.2.3.4 (the one returned by ifconfig, not the externally accessable one)
  • Other destinations: example.com
  • No relays
  • No smarthost
  • No Dial-on-Demand
  • mbox format (or whatever)
  • Split the files
  • ubuntu for postmaster mail

Check we’re now running a mail server:
sudo netstat -napt
should show
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 1.2.3.4:25 0.0.0.0:* LISTEN 25700/exim4

Now we have a mail server, the world needs to find it. Check your nameserver setting to ensure mail is destined this machine.  You probably want only one MX record.

Check the Internet can send mail to our server. After allowing for the appropriate propagation delay for your nameserver changes, use gmail or something to send an email to ubuntu@example.com – you should be able to read it by typing
mail

Now it’s time to enable MTA-MTA link encryption for secure transport of mail, by enabling STARTTLS on exim4 using our LetsEncrypt certificate
sudo nano /etc/exim4/conf.d/main/03_exim4-config_tlsoptions
Enable STARTTLS by adding/setting in the tlsoptions section:
MAIN_TLS_ENABLE = yes
MAIN_TLS_CERTKEY = no

before any of the IF shenanigans. Also add/replace pointers to the certificates:
tls_certificate = /etc/letsencrypt/live/example.com/fullchain.pem
tls_privatekey = /etc/letsencrypt/live/example.com/privkey.pem

The MAIN_TLS_CERTKEY = no entry fixes an exim4 log message
2017-04-16 09:13:24 TLS error on connection from your.home.ip.com (IcePlanet) [5.6.7.8] (cert/key setup: cert=/etc/exim4/exim.crt key=/etc/exim4/exim.key): Error while reading file.
You will see this when testing with swaks:
$ swaks -a -tls -q HELO -s example.com -au test -ap '<>'
=== Trying example.com:25...
=== Connected to example.com.
< - 220 your.vps.host.com ESMTP Exim 4.86_2 Ubuntu Sun, 16 Apr 2017 09:13:24 +0000 -> EHLO IcePlanet
< - 250-your.vps.host.com Hello your.home.ip.com [5.6.7.8]
STARTTLS
< ** 454 TLS currently unavailable *** STARTTLS attempted but failed -> QUIT
< - 221 your.vps.host.com closing connection
=== Connection closed with remote host.

Allow exim (which when running runs as user Debian-exim) to get to the certificates:

sudo groupadd privkey_users
sudo usermod -aG privkey_users Debian-exim
sudo sudo chmod g+rx /etc/letsencrypt/live/
sudo sudo chmod g+rx /etc/letsencrypt/archive/
sudo chown root:privkey_users /etc/letsencrypt/archive/
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/cert1.pem
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/chain1.pem
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/privkey1.pem
sudo chown root:privkey_users /etc/letsencrypt/archive/example.com/fullchain1.pem
sudo chown root:privkey_users /etc/letsencrypt/live/
sudo chown root:privkey_users /etc/letsencrypt/live/example.com/

Changing these permissions doesn’t affect apache2’s ability to get them.
The reason we’ve used a group here is to allow both exim and any other app (for example, a secondary service that wants to use 8080 to serve up a configuration page) to access the private keys; just add any other user that needs to use the private keys to the privkey_users group.

These permission changes prevent the following error message in your log file:
2008-06-03 08:27:35 TLS error on connection from me.at.home.com ([1.2.3.4]) [5.6.7.8] (cert/key setup: cert=/etc/ssl/certs/server.pem key=/etc/ssl/private/server.key): Error while reading file.

Restart the service and the TLS settings ought to be working
sudo service exim4 restart
Test STARTTLS is working from another machine
swaks -a -tls -q HELO -s example.com -au test -ap '<>'
There shouldn’t be any obvious complaining.

Done!

Comment spammers try to get clever

On my personal blog, on a post about the Melbourne public transport smartcard Myki, this comment popped into the moderation queue:

I found another very uefsul behaviour this morning.I don’t touch off when I get home at night. While, this seems weird how do they know I got off in Zone 1 (for which I have a pass) and didn’t travel into Zone 2 (for which I should be charged Money)? But, the fare manual says that if you have a MyKi Pass and you touch on in a zone for which that pass is valid (Parliament in my case) then there is no default fair.So, every morning when I touch on, it tells me deducting fare for previous trip . Which is $0.00. Cool.This morning I forgot to touch on some power issues on our line, got to chatting with the Station Hosts, just forgot. When I got to Parliament, I fully expected the gates to deny me egress and I’d have to do the silly thing where you act exasperated and they just wave you through the end gate even though you’re holding nothing but a wallet in your hand (MyKi works while in the wallet).BUT, it let me through. Seems that it’s happy enough that I started a trip (at Parliament) last night, spent 16 hours travelling and ended my trip back where I started. So it let me out basically a touch off of the trip home last night.Bizarre. But uefsul.

At first glance, it looked on-topic. But I was suspicious because the user link was to facebook.com/profile.php?id=XYZ (I’ve removed the ID) — and it came in the middle of a bunch of other (less-relevant) comments linking to similar URLs.

Googling around for key words in the comment, I found that it’s a copy of a comment from a completely different blog, with various misspellings inserted, and paragraph breaks removed. The original:

I found another very useful behaviour this morning.

I don’t touch off when I get home at night. While, this seems weird – how do they know I got off in Zone 1 (for which I have a pass) and didn’t travel into Zone 2 (for which I should be charged Money)? But, the fare manual says that if you have a MyKi Pass and you touch on in a zone for which that pass is valid (Parliament in my case) then there is no default fair.

So, every morning when I touch on, it tells me “deducting fare for previous trip”. Which is $0.00. Cool.

This morning I forgot to touch on – some power issues on our line, got to chatting with the Station Hosts, just forgot. When I got to Parliament, I fully expected the gates to deny me egress and I’d have to do the silly thing where you act exasperated and they just wave you through the end gate even though you’re holding nothing but a wallet in your hand (MyKi works while in the wallet).

BUT, it let me through. Seems that it’s happy enough that I started a trip (at Parliament) last night, spent 16 hours travelling and ended my trip back where I started. So it let me out – basically a touch off of the trip home last night.

Bizarre. But useful.

It seems the spammers are trying to get a bit more clever at sneaking their comment posts past moderators.

I still don’t know why, given WordPress has used NoFollow on their comment links for about ten years now.

Spam from Cotap

Subject line: “You’ve been added as a contact on Cotap”

Email text: “Cotap is a secure texting app for teams. View the [company name] directory. [link]

“Are you using WhatsApp, GroupMe, or iMessage to text your coworkers? See why Cotap is better for work.”

Yeah, no, this is spam, and it sounds like quite a few people are getting them.

Cotap was formed by former Yammer people… there’s speculation that when they left Yammer, they took a copy of the Yammer email list with them.

In any case, don’t fall for their trap. Don’t do business with spammers.

In fact, why on earth would you trust your business messaging to spammers?

Much spam from iCMG/KnowledgeHut/bmsend

At work I’m getting repeated spams from one mob which send surprisingly similar emails about conferences and training from various domains, which include:

enterprisearchitecturetrainingtoday.com
businessawards2013.com
BPMArchitects.com
newbpmtraining.net
myenterprisearchitecture.com
telcoarch.com
thebpmtraining.com
businessintelligencelive.com (added 2014-01-13)
enterprisearchitecturetrainer.com (added 2014-04-01)
mycloudcomputingtrainingnow.com (added 2014-04-07)
sixsigmaaccreditation.com — (added 2014-07-24 — also noted use of the brand name KnowledgeHut )
itilfoundationoceana.com — (added 2014-08-06 — still using KnowledgeHut name)
Sent by pmta90.dedicated.bmsend.com on behalf of BusinessAnalysisProgram.com (added 2014-08-18)
primeverastudy.com (added 2014-08-26)
mybusinesscasewriting.com (added 2014-09-16)
finance4nonfinace.com (added 2014-09-16)
ApacheHadoopbootcamp.com (added 2014-09-18)
pmtrainingPrograms.com (added 2014-09-18)
sixsigmaforbeginers.com (added 2014-09-26 — note typo)
msptrainingnow.com (added 2014-09-26)
mspaccreditationnow.com (added 2014-09-30)
prince2bootcamptoday.com (added 2014-09-30)
pmprojectnplanning.com (added 2014-10-03)
myprince2certification.com (added 2014-10-13)
mspbootcamptoday.com (added 2014-10-15)
ITILfoundationgulf.com (added 2014-10-22)
financialmodelinginexcel.com (added 2014-10-27)
scrumbootcampnow.com (added 2014-10-30)
knowledgehutglobal.com (added 2014-10-30)
businesssuccesstrainings.com (added 2014-11-10)
primeveratraining.com (added 2014-11-14)
sixsigmaintialsquare.com (added 2014-12-09 — note it appears to misspell “initial”)
projectmgmttoday.com (added 2014-12-19)
businesscasewritingcourseware.com (added 2014-12-19)
mspadvancecoursefinder.com (added 2014-12-19)
agilescruminitiatives.com (added 2015-01-12)
fpasnap.com (added 2015-01-13)
itilfoundationstudy.com (added 2015-01-22)
financialmodelingnexcel.com (added 2015-02-17)
scrumaccreditationcase.com (added 2015-02-17)
pgmpinitiatives.com (added 2015-03-05)
pmcertin.com (added 2015-02-17 — includes “itilsearch.com” in unsubscribe links)
sixsigmalearnings.com (added 2015-02-17 — other domains mentioned include bmsend.com, bmetrack.com, alliancestudy.com)
agileinitiatives.com (added 2015-03-11)
hadoopclassroomwork.com (added 2015-03-11)
pmprojectsolutionprogram.com (added 2015-03-16)
scrumtrainingtoday.com (added 2015-04-10)
global-health.com (added 2015-05-14 — using bmsend.com… Is this ICMG branching out into health services, or a different company making use of the same spam sending service? Looks like the latter. They even quote an Melbourne address: Level 2, 607 Bourke Street)
prepmtrain.com (added 2015-06-25 — also quotes pmatsearch.com in the dodgy unsubscribe link)
businesscasewritinginfo.com (added 2015-07-15)
cbapcourses.com (added 2015-07-15)
BusinessanalysisProfessionals.com (added 2015-07-15)
msproject2013tutor.com (added 2015-07-21)
businesscasewritingnewapps.com (added 2015-08-24)
agileandscrumintro.com (added 2015-12-02)
projectmanagementgain.com (added 2015-12-02)
itilpractice.com (added 2015-12-02)
msprojectmethods.com (added 2015-12-10)
Pythonread.com (added 2015-12-10)
bcwbusiness.com (added 2015-12-10)
businesscasewritingnow.com (added 2015-12-22)
learnbusinesscase.com (added 2015-12-22)
itilfreshtutor.com (added 2016-04-06)
coreproject.co (added 2016-04-06)
androiddevelopmentnow.com (added 2016-04-06) – emails include subject lines proclaiming “Learn Andorid”!
officepersonalsecretary.com (added 2016-04-06)
projectsmgmts.com (added 2016-04-28)
msprojectnewprofessional.com (added 2016-05-18)
sixsigmaapps.com (added 2016-05-18)
prince2newsession.com (added 2016-05-18)
Professionalsofbusinessanalysis.com (added 2016-06-21)
deftanalyze.com (added 2016-06-21)
skillcentrics.com (added 2016-06-21) – used as a Reply-To address
bigdatahadooptutors.com (added 2016-06-21)
iosdvptraining.com (added 2016-06-21)
angularjsconceptstraining.com (added 2016-06-21)
pm-careers.com (added 2016-06-22)
Latestprojectmanagement.com (added 2016-06-22)
scrumstudies.com (added 2016-06-28)
scrummasterscert.com (added 2016-06-29) – this appears to be sent via mailerinteractive.com
elationproject.com (added 2016-07-04)
itsm-skills.com (added 2016-08-23)
certs-skill.com (added 2016-08-23)

Apart from using many different domains, these guys also continually change the address within the domain, and Outlook doesn’t appear to be able to consign an entire domain to the blocked senders list.

Many include this footer:

You are receiving this e-mail because you happen to be either our client or were added to our comprehensive database on account of your contribution in the IT domain. However, should you no longer wish to receive any further mails from our side, please Click here Unsubscribe iCMG | Level 9, Avaya House, 123 Epping Road, North Ryde, NSW.| Phone +61 2 8005 0977

…but of course I’ve tried that and it doesn’t work… it probably only served to prove to them that mine was a live address.

Perhaps unsurprisingly, the street address quoted is a serviced office.

I have been putting these domains into the spam senders list in Exchange, but they still get through. I can only assume that the list in Exchange is a “soft” one.

Annoying.

I have, of course, passed on a spam message to the ACMA spam reporting people… but I don’t hold out much hope of any success there.

I’m adding additional domains as they come up — when I get the chance. It’s interesting to see that some of them include spelling errors; most are just semi-random buzzwords stuck together.

Top spams

The work email server spam filter does not simply reject everything suspicious – this would risk us losing legitimate emails, something made worse by some companies choosing to send invoices and remittances as PDF attachments with no accompanying text whatsoever, something the mail server considers dodgy.

So (until I work out a better, foolproof automated process, or take the time to properly tweak the spam settings on the server) I manually look through some of the doubtful messages to pluck out those that are not actually spam.

The most common types of spam messages caught seem to be…

5. Your credit card is blocked (enter all your details into our dodgy web site)…

4. Work from home and earn $$$…

3. Marry a gorgeous Russian girl…

2. Cheap replica watches (Rolex etc)…

1. Cheap medicine to help gentlemen with, err, size-related personal problems.

…though it appears cheap Canadian pharmacies are also gaining in popularity, despite this being for a .com.au address.

Horde access keys

Beware of Horde’s IMP webmail client and its access/shortcut keys.

One that’s caught me is that if new email composition is set to be in a separate window, and access keys are on, then Alt-F4 (which in Windows is normally the equivalent of Close) is pressed, instead of saving the email to Drafts, or cancelling the email, it sends it.

I’m a common user of Alt-F4, which means several times I’ve thought I was cancelling the email, but instead it’s sent it.

Another is Alt-D for Delete (the current message). On many browsers this predates Ctrl-L to go to the address window, and while I know I should learn Ctrl-L, I still commonly press Alt-D. If Horde is configured to not even put the message into the Trash, carelessly pressing Alt-D will zap the message forever more, no trace left.

To prevent these happening again, I’ve now turned off Access keys: Options / Global options / Display Options / Should access keys be defined for most links?

Amusing comment spam

Amusing comment spam left on my personal blog:

Spammers leave spam comments in the belief that they will gain better search engine rankings and traffic by building as many links to their websites as possible. Spammers often employ bots or other automated systems to look for mortgage blog and website and leave self serving promotional comments links..Spam is a numbers game so if spammers can send automated spam to large numbers of websites for very little money so even if they convert a small percentage of the sites they spam they can make a profit..Spammers will also leave links to their websites in an attempt to push link juice or Google Juice to their websites but most mortgage websites and blogs add a rel nofollow tag to prevent the passing of pagerank or link juice.

And this one, from a user apparently who signed him/herself “penis enlargement”.

It’s so hard to get backlinks these days, honestly i need a backlink by comments on your blog / forums or guestbook to make my website appear in search engine. I am getting desperate Now! I know you’ll laugh while reading this comment !!! Here is my website penis enlargement [url deleted] I know my comments do not relate to the topic, but PLEASE HELP ME!! APPROVING MY COMMENT!
Regards: PoormanBH2011

Yeah right. Like I’m going to approve that.

BTW, both were caught correctly by Akismet.

Email startup times

Gmail vs ThunderbirdA quick timing test on my main home workhorse computer, which isn’t the fastest in the world, but isn’t the slowest either. (Windows 7, Athlon 64 X2 dual core 4400+ 2300 Mhz, 3 Gb RAM, on a fast ADSL2+ net connection.)

Having started Windows and logged onto a clean desktop:

  • Start Chrome with GMail set as the home page: 8.5 seconds to ready
  • Start Thunderbird: 11.6 seconds to ready

No wonder people are heading into the cloud.

Subsequent timings (without a reboot, so some things may be cached, eg later in a session when you’ve closed your email and you want to go back in):

  • Chrome with GMail: 3.4 seconds
  • Thunderbird: 3.1 seconds

Interesting.