Blog

Reading Rolled Papyri

One of my favourite parts of my former role at the Bodleian Libraries was getting to work on exhibitions. Not just because it was varied and interesting work, but because it let me get up-close to remarkable artifacts that most people never even get the chance to see.

Miniature model of an exhibition space, constructed using painted blocks and laid-out on the floor of an exhibition space.
We also got to play dollhouse, laying out exhibitions in miniature.

A personal favourite of mine are the Herculaneum Papyri. These charred scrolls were part of a private library near Pompeii that was buried by the eruption of Mount Vesuvius in 79 CE. Rediscovered from 1752, these ~1,800 scrolls were distributed to academic institutions around the world, with the majority residing in Naples’ Biblioteca Nazionale Vittorio Emanuele III.

Under-construction exhibition including a highly-reflective suit worn by volcano field researchers.
The second time I was in an exhibition room with the Bodleian’s rolled-up Herculaneum Papyri was for an exhibition specifically about humanity’s relationship with volcanoes.

As you might expect of ancient scrolls that got buried, baked, and then left to rot, they’re pretty fragile. That didn’t stop Victorian era researchers trying a variety of techniques to gently unroll them and read what was inside.

Blackened fragments of an unrolled papyrus.
Unrolling the scrolls tends to go about as well as you’d anticipate. A few have been deciphered this way. Many others have been damaged or destroyed by unrolling efforts.

Like many others, what I love about the Herculaneum Papyri is the air of mystery. Each could be anything from a lost religious text to, I don’t know, somebody’s to-do list (“buy milk, arrange for annual service of chariot, don’t forget to renew volcano insurance…”).1

In recent years, we’ve tried “virtually unrolling” the scrolls using a variety of related technologies. And – slowly – we’re getting there.

X-ray tomography is amazing, but it’s hampered by the fact that the ink and paper have near-equivalent transparency to x-rays. Plus, all the other problems. But new techniques are helping to overcome them.

So imagine my delight when this week, for the first time ever, a complete word was extracted from one of the carbonised, still-rolled-up scrolls from Herculaneum. Something that would have seemed inconceivable to the historians who first discovered and catalogued the scrolls is now possible, thanks to their careful conservation over the years along with the steady advance of technology.

Computer-assisted photograph showing visible letters on a rolled scroll, with highlighting showing those that can be deciphered, forming a word.
The word appears to be “purple”: either πορφύ̣ρ̣ας̣ (a noun, similar to how we might say “pass the purple [pen]” or πορφυ̣ρ̣ᾶς̣: if we can decode more words around it then it which might become clear from the context.
Anyway, I thought that was exciting news so I wanted to share.

Footnotes

1 For more-serious academic speculation about the potential value of the scrolls, Richard Carrier’s got you covered.

× × × ×

Absence/Presents

I’m probably not going to get you a Christmas present. You probably shouldn’t get me one either.

Dan, wearing an "elf costume" Christmas jumper, looks into the camera while cuddling a French Bulldog. The pair are sitting on a beige sofa.
All I need for Christmas is… a woolly jumper and a dog, apparently. (And I only need the latter if the goose doesn’t get delivered.)

If you’re one of my kids and you’ve decided that maybe my blog isn’t just “boring grown-up stuff” and have come by, then you’re one of the exceptions. Lucky you.

Children get Christmas gifts from me. But if you’re an adult, all you’re likely to get from me is a hug, a glass of wine, and more food than you can possibly eat in a single sitting.

Top-down view of a dining table set with a Christmas-themed tablecloth. The meal has concluded and the seats have been vacated, but large amounts of food - most of a turkey, half a nutloaf, lots of mashed potato, several sprouts, stuffing balls, and chestnuts, some roast potatoes and parsnips, an entire boat full of gravy, and almost a dozen Yorkshire puddings - are still set out.
Turns out the real meaning of Christmas was eating yourself into indigestion all along.

I’ve come to the conclusion – much later than my mother and my sisters, who were clearly ahead of the curve – that Christmas presents are for kids.

Maybe, once, Christmas presents were for adults too, but by now the Internet has broken gift-giving to the extent it’s almost certainly preferable for me and the adults in my life if they just, y’know, order the thing they want than hoping that I’ll pick it out for them. Especially as so many of us are at a point where we already have a plethora of “stuff”, and don’t want to add to it unnecessarily at a time of year when, frankly, we’ve got better things to spend our time and money on.

Dan, wearing a Princess Twilight Sparkle / Frank Herbert's Dune crossover fan art t-shirt, sits on a grey sofa in front of a lit Christmas tree, holding a glass of wine. At the other end of the sofa JTA, a white man with a thick beard and glasses, reads to a (tired-looking) young boy. All three are surrounded by books.
I’ll still be participating fully in my household‘s “book exchange” Christmas Eve tradition, though, because it’s awesome.

Birthdays are still open season, because they aren’t hampered by the immediate expectation of reciprocity that Christmas carries. And I reserve the right to buy groups of (or containing) adults gifts at Christmas. But individual adults aren’t getting one this year, and they certainly shouldn’t feel like they need to get me anything either.1

I don’t know to what extent, if at all, Ruth and JTA will be following me in this idea, so if you’re somebody who might have expected a gift from or wanted to give a gift to one of them… you’re on your own; you work it out!

Here’s to a Merry Christmas full of presents for children, only!

Footnotes

1 If you’ve already bought me a gift for Christmas this year… firstly, that’s way too organised: you know it’s only October, right? And secondly: my birthday’s only a couple of weeks later…

× × ×

Weird A.I. Yankovic, a cursed deep dive into the world of voice cloning

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

In the parallel universe of last year’s Weird: The Al Yankovic Story, Dr. Demento encourages a young Al Yankovic (Daniel Radcliffe) to move away from song parodies and start writing original songs of his own. During an LSD trip, Al writes “Eat It,” a 100% original song that’s definitely not based on any other song, which quickly becomes “the biggest hit by anybody, ever.”

Later, Weird Al’s enraged to learn from his manager that former Jackson 5 frontman Michael Jackson turned the tables on him, changing the words of “Eat It” to make his own parody, “Beat It.”
Your browser does not support the video tag.

This got me thinking: what if every Weird Al song was the original, and every other artist was covering his songs instead? With recent advances in A.I. voice cloning, I realized that I could bring this monstrous alternate reality to life.

This was a terrible idea and I regret everything.

Everything that is wrong with, and everything that is right with, AI voice cloning, brought together in one place. Hearing simulations of artists like Michael Jackson, Madonna, and Kurt Cobain singing Weird Al’s versions of their songs is… strange and unsettling.

Some of them are pretty convincing, which is a useful and accessible reminder about how powerful these tools are becoming. An under-reported story from a few years back identified what might be the first recorded case of criminals using AI-based voice spoofing as part of a telephone scam, and since then the technology needed to enact such fraud has only become more widely-available. While this weirder-than-Weird-Al project is first and foremost funny, for many it foreshadows darker things.

Easy FoundryVTT Cloud Hosting

Foundry is a wonderful virtual tabletop tool well-suited to playing tabletop roleplaying games with your friends, no matter how far away they are. It compares very favourably to the market leader Roll20, once you get past some of the initial set-up challenges and a moderate learning curve.

Screenshot from FoundryVTT, showing a party of three adventurers crossing a rickety bridge through a blood-red swamp, facing off against a handful of fiends coming the other way. A popup item card for a "Dragon Slayer Longsword" is visible, along with two 20-sided dice.
The party of adventurers I’ve been DMing for since last summer use Foundry to simulate a tabletop (alongside a conventional video chat tool to let us see and hear one another).

You can run it on your own computer and let your friends “connect in” to it, so long as you’re able to reconfigure your router a little, but you’ll be limited by the speed of your home Internet connection and people won’t be able to drop in and e.g. tweak their character sheet except when you’ve specifically got the application running.

A generally better option is to host your Foundry server in the cloud. For most of its history, I’ve run mine on Fox, my NAS, but I’ve recently set one up on a more-conventional cloud virtual machine too. A couple of friends have asked me about how to set up their own, so here’s a quick guide:

Screenshot from Linode showing a server, "Foundry", running, with specs as described below.
I used Linode to spin up a server because I still had a stack of free credits following a recent project. The instructions will work on any cloud host where you can spin up a Debian 12 virtual machine, and can be adapted for other distributions of Linux.

You will need…

  • A Foundry license ($50 USD / £48 GBP, one-off payment1)
  • A domain name for which you control the DNS records; you’ll need to point a domain, like “danq.me” (or a subdomain of it, e.g. “vtt.danq.me”), at an IP address you’ll get later by creating an “A” record: your domain name registrar can probably help with this – I mostly use Gandi and, ignoring my frustration with recent changes to their email services, I think they’re great
  • An account with a cloud hosting provider: this example uses Linode but you can adapt for any of them
  • A basic level of comfort with the command-line

1. Spin up a server

Getting a virtual server is really easy nowadays.

Annotated screenshot showing a Linode provisioning form, with "Debian 12", the "London, UK", region, and "Dedicated 4GB" plan options selected.
Click, click, click, and you’ve got yourself a server.

You’ll need:

  • The operating system to be Debian 12 (or else you’ll need to adapt the instructions below)
  • The location to be somewhere convenient for your players: pick a server location that’s relatively-local to the majority of them to optimise for connection speeds
  • Approximately 2 CPUs and 4GB of RAM, per Foundry’s recommended server specifications
  • An absolute minimum of 1GB of storage space, I’d recommend plenty more: The Levellers’ campaign currently uses about 10GB for all of its various maps, art, videos, and game data, so give yourself some breathing room (space is pretty cheap) – I’ve gone with 80GB for this example, because that’s what comes as standard with the 2 CPU/4GB RAM server that Linode offer

Choose a root password when you set up your server. If you’re a confident SSH user, add your public key so you can log in easily (and then disable password authentication entirely!).

For laziness, this guide has you run Foundry as root on your new server. Ensure you understand the implications of this.2

2. Point your (sub)domain at it

DNS propogation can be pretty fast, but… sometimes it isn’t. So get this step underway before you need it.

Your newly-created server will have an IP address, and you’ll be told what it is. Put that IP address into an A-record for your domain.

Screenshot from Gandi showing adding an A record for vtt.danq.me -> 1.2.3.4.
The interface for adding a new DNS record in Gandi is pretty simple – record type, time to live, name, address – but it’s rarely more complicated that this with any registrar that provides DNS services.

3. Configure your server

In my examples, my domain name is vtt.danq.me and my server is at 1.2.3.4. Yours will be different!

Connect to your new server using SSH. Your host might even provide a web interface if you don’t have an SSH client installed: e.g. Linode’s “Launch LISH Console” button will do pretty-much exactly that for you. Log in as root using the password you chose when you set up the server (or your SSH private key, if that’s your preference). Then, run each of the commands below in order (the full script is available as a single file if you prefer).

3.1. Install prerequisites

You’ll need unzip (to decompress Foundry), nodejs (to run Foundry), ufw (a firewall, to prevent unexpected surprises), nginx (a webserver, to act as a reverse proxy to Foundry), certbot (to provide a free SSL certificate for Nginx), nvm (to install pm2) and pm2 (to keep Foundry running in the background). You can install them all like this:

apt update
apt upgrade
apt install -y unzip nodejs ufw nginx certbot nvm
npm install -g pm2

3.2. Enable firewall

By default, Foundry runs on port 30000. If we don’t configure it carefully, it can be accessed directly, which isn’t what we intend: we want connections to go through the webserver (over https, with http redirecting to https). So we configure our firewall to allow only these ports to be accessed. You’ll also want ssh enabled so we can remotely connect into the server, unless you’re exclusively using an emergency console like LISH for this purpose:

ufw allow ssh
ufw allow http
ufw allow https
ufw enable

3.3. Specify domain name

Putting the domain name we’re using into a variable for the remainder of the instructions saves us from typing it out again and again. Make sure you type your domain name (that you pointed to your server in step 2), not mine (vtt.danq.me):

DOMAIN=vtt.danq.me

3.4. Get an SSL certificate with automatic renewal

So long as the DNS change you made has propogated, this should Just Work. If it doesn’t, you might need to wait for a bit then try again.

certbot certonly --agree-tos --register-unsafely-without-email --rsa-key-size 4096 --webroot -w /var/www/html -d $DOMAIN

Don’t continue past this point until you’ve succeeded in getting the SSL certificate sorted.

The certificate will renew itself automatically, but you also need Nginx to restart itself whenever that happens. You can set that up like this:

printf "#!/bin/bash\nservice nginx restart\n" > /etc/letsencrypt/renewal-hooks/post/restart-nginx.sh
chmod +x /etc/letsencrypt/renewal-hooks/post/restart-nginx.sh

3.5. Configure Nginx to act as a reverse proxy for Foundry

You can, of course, manually write the Nginx configuration file: just remove the > /etc/nginx/sites-available/foundry from the end of the printf line to see the configuration it would write and then use/adapt to your satisfaction.

set +H
printf "server {\n listen 80;\n listen [::]:80;\n server_name $DOMAIN;\n\n # Redirect everything except /.well-known/* (used for ACME) to HTTPS\n root /var/www/html/;\n if (\$request_uri !~ \"^/.well-known/\") {\n return 301 https://\$host\$request_uri;\n }\n}\n\nserver {\n listen 443 ssl http2;\n listen [::]:443 ssl http2;\n server_name $DOMAIN;\n\n ssl_certificate /etc/letsencrypt/live/$DOMAIN/fullchain.pem;\n ssl_certificate_key /etc/letsencrypt/live/$DOMAIN/privkey.pem;\n\n client_max_body_size 300M;\n\n location / {\n # Set proxy headers\n proxy_set_header Host \$host;\n proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;\n proxy_set_header X-Forwarded-Proto \$scheme;\n\n # These are important to support WebSockets\n proxy_set_header Upgrade \$http_upgrade;\n proxy_set_header Connection \"Upgrade\";\n\n proxy_pass http://127.0.0.1:30000/;\n }\n}\n" > /etc/nginx/sites-available/foundry
ln -sf /etc/nginx/sites-available/foundry /etc/nginx/sites-enabled/foundry
service nginx restart

3.6. Install Foundry

3.6.1. Create a place for Foundry to live

mkdir {vtt,data}
cd vtt

3.6.2. Download and decompress it

Screenshot from FoundryVTT showing where to find the "Timed URL" download link button.
For this step, you’ll need to get a Timed URL from the Purchased Licenses page on your FoundryVTT account.

Substitute in your Timed URL in place of <url from website> (keep the quotation marks – " – though!):

wget -O foundryvtt.zip "<url from website>"
unzip foundryvtt.zip
rm foundryvtt.zip

3.6.3. Configure PM2 to run Foundry and keep it running

Now you’re finally ready to launch Foundry! We’ll use PM2 to get it to run automatically in the background and keep running:

pm2 start --name "Foundry" node -- resources/app/main.js --dataPath=/root/data

You can watch the logs for Foundry with PM2, too. It’s a good idea to take a quick peep at them to check it launched okay (press CTRL-C to exit):

pm2 logs 0

4. Start adventuring!

Screenshot showing FoundryVTT requesting a license key.
Point your web browser at your domain name (e.g. I might go to https://vtt.danq.me) and you should see Foundry’s first-load page, asking for your license key.

Provide your license key to get started, and then immediately change the default password: a new instance of Foundry has a blank default password, which means that anybody on Earth can administer your server: get that changed to something secure!

Now you’re running on Foundry!

Footnotes

1 Which currency you pay in, and therefore how much you pay, for a Foundry license depends on where in the world you are where your VPN endpoint says you are. You might like to plan accordingly.

2 Running Foundry as root is dangerous, and you should consider the risks for yourself. Adding a new user is relatively simple, but for a throwaway server used for a single game session and then destroyed, I wouldn’t bother. Specifically, the risk is that a vulnerability in Foundry, if exploited, could allow an attacker to reconfigure any part of your new server, e.g. to host content of their choice or to relay spam emails. Running as a non-root user means that an attacker who finds such a vulnerability can only trash your Foundry instance.

× × × × × ×

Out of the Trees

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

After “Monty Python’s Flying Circus” ended, Graham Chapman worked with an up-and-coming young writer named Douglas Adams on a new sketch comedy show for the BBC. It was called “Out of the Trees,” and it bombed. Only one episode was made, and that aired only once, on January 10, 1976.

Once the Beeb gave up on “Out of the Trees,” they did to it what they did to so many other programs of that era: they erased it.

Chapman had recorded the show on one of the very earliest home videotape formats… it took two years to build a compatible player.

It’s neither Chapman nor Adams best work, and you can see how it got canned after only a pilot episode. But it’s not terrible.

But the lesson here is one about the challenge of archiving non-print media. Anything that needs a device to “play” it, whether it’s as simple as a vinyl record or as complex as a videogame, is at greater risk of being lost forever. And the faster the pace of technology moves, the more stuff gets left behind as technology moves on. Is a digital dark age looming? Are we already in it, but that won’t be known until some future date?

Dan Q found GC2HDCZ Tom Burts Cache 4

This checkin to GC2HDCZ Tom Burts Cache 4 reflects a geocaching.com log entry. See more of Dan's cache logs.

Found while walking into High Wycombe to work after dropping my canine caching-companion (pictured) off at the nearby veterinary hospital for an operation. Didn’t need her help with this easy find, luckily! Shame about all the fly tipping littering this otherwise pleasant path. Greetings from Oxfordshire!

A champagne-coloured French Bulldog snuggles up against a human's leg, wearing blue jeans.

×

CapsulePress – Gemini / Spartan / Gopher to WordPress bridge

For a while now, this site has been partially mirrored via the Gemini1 and Gopher protocols.2 Earlier this year I presented hacky versions of the tools I’d used to acieve this (and made people feel nostalgic).

Now I’ve added support for Spartan3 too and, seeing as the implementations shared functionality, I’ve combined all three – Gemini, Spartan, and Gopher – into a single package: CapsulePress.

Diagram illustrating the behaviour of CapsulePress: a WordPress installation provides content, and CapsulePress makes that content available via gemini://, spartan://, and gopher:// URLs.

CapsulePress is a Gemini/Spartan/Gopher to WordPress bridge. It lets you use WordPress as a CMS for any or all of those three non-Web protocols in addition to the Web.

For example, that means that this post is available on all of:

Composite screenshot showing this blog post in, from top-left to bottom-right: (1) Firefox, via HTTPS, (2) Lagrange, via Gemini, (3) Lagrange, via Spartan, and (4) Lynx, via Gopher.

It’s also possible to write posts that selectively appear via different media: if I want to put something exclusively on my gemlog, I can, by assigning metadata that tells WordPress to suppress a post but still expose it to CapsulePress. Neat!

90s-style web banners in the style of Netscape ads, saying "Gemini now!", "Spartan now!", and "Gopher now!". Beneath then a wider banner ad promotes CapsulePress v0.1.
Using Gemini and friends in the 2020s make me feel like the dream of the Internet of the nineties and early-naughties is still alive. But with fewer banner ads.

I’ve open-sourced the whole thing under a super-permissive license, so if you want your own WordPress blog to “feed” your Gemlog… now you can. With a few caveats:

  • It’s hard to use. While not as hacky as the disparate piles of code it replaced, it’s still not the cleanest. To modify it you’ll need a basic comprehension of all three protocols, plus Ruby, SQL, and sysadmin skills.
  • It’s super opinionated. It’s very much geared towards my use case. It’s improved by the use of templates. but it’s still probably only suitable for this site for the time being, until you make changes.
  • It’s very-much unfinished. I’ve got a growing to-do list, which should be a good clue that it’s Not Finished. Maybe it never will but. But there’ll be changes yet to come.

Whether or not your WordPress blog makes the jump to Geminispace4, I hope you’ll came take a look at mine at one of the URLs linked above, and then continue to explore.

If you’re nostalgic for the interpersonal Internet – or just the idea of it, if you’re too young to remember it… you’ll find it there. (That Internet never actually went away, but it’s harder to find on today’s big Web than it is on lighter protocols.)

Footnotes

1 Also available via Gemini.

2 Also via Finger (but that’s another story).

3 Also available via Spartan.

4 Need a browser? I suggest Lagrange.

× × ×

Stopping WordPress Emoji ‘Images’ in Feeds

After sharing that Octopuns has started posting again after a 9½-year hiatus earlier today, I noticed something odd: where I’d written “I ❤️ FreshRSS“, the heart emoji was huge when viewed in my favourite feed reader.

Screenshot from a web-based RSS reader application, showing recent repost "Groundhog Day". The final line contains a link with the text "I ❤️ FreshRSS", but the red heart emoji seems to be enormous compared to the next adjacent to it.
Why yes, I do subscribe to my own RSS feed. What of it?

It turns out that by default, WordPress replaces emoji in its feeds (and when sending email) with images of those emoji, using the Tweemoji set, and with the alt-text set to the original emoji. These images are hosted at https://s.w.org/images/core/emoji/…-based URLs.

For example, this heart was served with the following HTML code (the number 2764 refers to the codepoint of the emoji):

<img src="https://s.w.org/images/core/emoji/14.0.0/72x72/2764.png"
     alt="❤"
   class="wp-smiley"
   style="height: 1em; max-height: 1em;"
/>

I can see why this functionality was added: what if the feed reader didn’t support Unicode or didn’t have a font capable of showing the appropriate emoji?

But I can also see reasons why it might not be desirable to everybody. For example:

  1. Downloading an image will always be slower than rendering an emoji.
  2. The code to include an image is always more-verbose than simply including an emoji.
  3. As seen above: a feed reader which imposes a minimum size on embedded images might well render one “wrong”.
  4. It’s marginally more-verbose for screen reader users to say “Image: heart emoji” than just “heart emoji”, I imagine.
  5. Serving an third-party image when a feed item is viewed has potential privacy implications that I try hard to avoid.
  6. Replacing emoji with images is probably unnecessary for modern feed readers anyway.

I opted to remove this functionality. I briefly considered overriding the emoji_url filter (which could be used to selfhost the emoji set) but I discovered that I could just un-hook the filters that were being added in the first place.

Here’s what I added to my theme’s functions.php:

remove_filter( 'the_content_feed', 'wp_staticize_emoji' );
remove_filter( 'comment_text_rss', 'wp_staticize_emoji' );

That’s all there is to it. Now, my feed reader shows my system’s emoji instead of a huge image:

Screenshot from a web-based RSS reader application, showing recent repost "Groundhog Day". The final line contains a link with the text "I ❤️ FreshRSS" shown correctly, with a red heart emoji at the appropriate font size.

I’m always grateful to discover that a piece of WordPress functionality, whether core or in an extension, makes proper use of hooks so that its functionality can be changed, extended, or disabled. One of the single best things about the WordPress open-source ecosystem is that you almost never have to edit somebody else’s code (and remember to re-edit it every time you install an update).

Want to hear about other ways I’ve improved WordPress’s feeds?

× ×

Groundhog Day

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Partial comic frame showing a groundhog popping its head out of a hole and shouting "RUN!".

After a break of nine and a half years, webcomic Octopuns is back. I have two thoughts:

  1. That’s awesome. I love Octopuns and I’m glad it’s back. If you want a quick taster – a quick slice, if you will – of its kind of humour, I suggest starting with Pizza.
  2. How did I know that Octopuns was back? My RSS reader told me. RSS remains a magical way to keep an eye on what’s happening on the Internet: it’s like a subscription service that delivers you exactly what you want, as soon as it’s available.

I’ve been pleasantly surprised before when my feed reader has identified a creator that’s come back from the dead. I ❤️ FreshRSS.

×

Crowdsourced Burger Photography

A fast food giant faces a lawsuit because their burgers don’t look like they do in the marketing. It’s not the first time.

Marketing photograph showing Burger King's "Whopper" burger.
Did you ever see a Whopper™️ that looked like this? Me neither.

If I ran a fast food franchise affected by this kind of legal action, do you know what I would do? I’d try to turn it back around into marketing exercise with a bit of crowdsourcing!

Think about it: get your customers to take photos and send them to you. For every franchisee that uses a photo you take, you get a voucher for a free meal (redeemable at any outlet, of course). And where it appears on the digital signage menus they all seem to have nowadays, your photo will have your name on it too.

Most submissions will be… unsuitable, of course. You’ll need a team of people vetting submissions. But for every 50 people who send a blurry picture of an unappetising bit of sludge-meat in a bun; for every 10 people who actually try hard but get too much background in or you can see the logo on their clothing or whatever; for every 5 people that deliberately send something offensive… you might get one genuinely good candid burger picture. Those pics get pushed out to franchisees to use. Sorted.

Now if anybody complains that you fake your photos you can explain that every one of your food pictures was taken by a real-life customer, and their name or handle is on the bottom of each one. Sure, you get to vet them, but they’re still all verifiably genuine pictures of your food.

And you probably only have to do this gimmick for a year and then everybody will forget. Crowdsourcing as a marketing opportunity: that’s what I’d be doing if I were crowned Burger King.

×

Debian + DKIM for Dummies

Earlier this year, for reasons of privacy/love of selfhosting, I moved the DanQ.me mailing list from Mailchimp to Listmonk (there’s a blog post about how I set it up), relaying outbound messages via an SMTP server provided by my domain registrar, Gandi.

Subscribe for email updates, or discover other ways to subscribe:

Email no more than
I assume that you knew that you can get an email, no more than once per day or once per week (your choice!) of what I get up to online, right? Email not your jam: there are plenty of other options too!

Unfortunately, Gandi recently announced that they’ll no longer be providing email services for free, so rather than fork out €4/month for an email address I won’t even read, I decided to inhouse it.

And because I learned a few things while doing so, I wrote this blog post so that next time I have to configure Postfix + DKIM, I’ll know where to find a guide. If it helps you in the meantime, that’s just a bonus.

Photograph of a French Bulldog on a wooden floor playing tug-of-war using a multicoloured plaited rope (the human holding the other end of the rope is behind the camera).
If the first rule of computing is “never roll your own crypto” (based on Schneier’s Law), the second rule might be “don’t run your own mailserver”. I don’t have a good picture to illustrate that, so here’s a photo of my dog playing tug-of-war.

Postfix

Running your own mailserver is a pain. I used to do it for all of my email, but – like many other nerds – when spam reached its peak and deliverability became an issue, I gave up and oursourced it1.

Screenshot of a Weekly Digest email from DanQ.me, showing in Mozilla Thunderbird.
Fun fact: when I’m at my desktop, I use a classic desktop email application for my personal email, like it’s the 90s or something2.
Luckily, I don’t need it to do much. I just need a mail transfer agent with an (unauthenticated, but local-only) SMTP endpoint: something that Listmonk can dump emails into, which will then reach out to the mailservers representing each of the recipients and relay them on. A default install of Postfix does all that out-of-the-box, so I ran sudo apt install postfix, accepted all the default options, and put the details into Listmonk.
Screenshot showing Listmonk's SMTP configuration screen. The host "192.168.2.12" and port "25" have been entered, TLS has been set to "STARTTLS", Skip TLS verification is enabled, and Auth Protocol is set to "None".
Listmonk makes adding an SMTP server very easy, and even includes a quick “test connection” link with which you can try out your settings.

Next, I tweaked my DNS configuration to add an SPF record, and tested it. This ought to have been enough to achieve approximate parity with what Gandi had been providing me with. Not bad.

$ dig +short -t TXT danq.link
"v=spf1 a mx a:fox.q-t-a.uk ip4:83.151.206.115 ~all"
You really can’t be doing without an SPF record as a minimum these days.

I sent a test email to a Gmail account, where I noticed two problems:

Screenshot from GMail showing a message with a red slashed padlock icon, which when clicked advises that "mail.danq.link did not encrypt this message".
It turns out that since the last time I ran a mailserver “for real”, the use of TLS for inter-server communication has become… basically mandatory. You don’t strictly have to do it, but if you don’t, some big email providers will put scary security warnings on your messages. This is a good thing.

The first problem was that Postfix on Debian isn’t configured by-default to use opportunistic TLS when talking to other mailservers. That’s a bit weird, but I’m sure there’s a good reason for it. The solution was to add smtp_tls_security_level = may to my /etc/postfix/main.cf.

The second problem was that without a valid DKIM signature on them, about half of my test emails were going straight to the spam folder. Again, it seems that since the last time I seriously ran a mailserver 20 years ago, this has become something that isn’t strictly required… but your emails aren’t going to get through if you don’t.

I’ve put it off this long, but I think it’s finally time for me to learn some practical DKIM.

Understanding DKIM

What’s DKIM, then?

Diagram illustrating the flow of email from sender to recipient. On the way it's signed by the sender's mailserver's private key, which publishes the public key via DNS. Further along, the recipient's mailserver retreives the public key and uses it to verify the signature.
I’ve already got an elementary understanding of how DKIM works, which I’ll summarise below.
  1. A server that wants to send email from a domain generates a cryptographic keypair.
  2. The public part of the key is published using DNS. The private part is kept securely on the server.
  3. When the server relays mail on behalf of a user, it uses the private key to sign the message body and a stated subset of the headers3, and attaches the signature as an email header.
  4. When a receiving server (or, I suppose, a client) receives mail, it can check the signature by acquiring the public key via DNS and validating the signature.

In this way, a recipient can be sure that an email received from a domain was sent with the authorisation of the owner of that domain. Properly-implemented, this is a strong mitigation against email spoofing.

OpenDKIM

To set up my new server to sign outgoing mail, I installed OpenDKIM and its keypair generator using sudo apt install opendkim opendkim-tools. It’s configuration file at /etc/opendkim.conf needed the following lines added to it:

# set up a socket for Postfix to connect to:
Socket inet:12301@localhost

# set up a file to specify which IPs/hosts can send through us without authentication and get their messages signed:
ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts           refile:/etc/opendkim/TrustedHosts

# set up a file to specify which selector/domain are used to each incoming email address:
SigningTable            refile:/etc/opendkim/SigningTable

# set up a file to specify which signing key to use for each selector/domain:
KeyTable                refile:/etc/opendkim/KeyTable

Into /etc/opendkim/TrustedHosts I put a list of local IPs/domains that would have their emails signed by this server. Mine looks like this (in this example I’m using example.com as my domain name, and default as the selector for it: the selector can be anything you like, it only matters if you’ve got multiple mailservers signing mail for the same domain). Note that 192.168.0.0/16 is the internal subnet on which my sending VM will run.

127.0.0.0.1
::1
192.168.0.0/16
*.example.com
example.com

/etc/opendkim/SigningTable maps email addresses (I’m using a wildcard) to the subdomain whose TXT record will hold the public key for the signature. This also goes on to inform the KeyTable which private key to use:

*@example.com default._domainkey.example.com

And then /etc/opendkim/KeyTable says where to find the private key for that:

default._domainkey.example.com example.com:default:/etc/opendkim/keys/example.com/default.private

Next, we need to create and secure that keypair. In /etc/opendkim/keys/example.com/, we run:

sudo opendkim-genkey -s default -d example.com
sudo chown opendkim:opendkim default.private

At last, we can configure Postfix to filter all mail through OpenDKIM by adding to our /etc/postfix/main.cf file:

milter_protocol = 2
milter_default_action = accept
smtpd_milters = inet:localhost:12301
non_smtpd_milters = inet:localhost:12301

DNS

The public key needs publishing via DNS. Conveniently, when you create a keypair using its tools, OpenDKIM provides a sample (in BIND-style) for you to copy-paste from or adapt: look in /etc/opendkim/keys/example.com/default.txt!

Screenshot from Gandi's Simple DNS management tools, showing danq.link with an SPF record as descibed earlier and a new TXT record on default._domainkey as just described.
Gandi’s DNS “Simple View” is great for one-off and quick operations, but I really appreciate that they have a BIND-style syntax “Advanced View” for when I’m making bigger and more-complex DNS configuration changes.

Once we’ve restarted both services (sudo service postfix restart; sudo service opendkim restart), we can test it!

Screenshot from GMail showing "DKIM: 'PASS' with domain danq.link".
Once the major email providers – who have the worst spam problem to deal with – say that your email signature looks good, you’re good.

So I learned something new today.

If you, too, love to spend your Saturday mornings learning something new, have a look at those subscription options to decide how you’d like to hear about whatever I get up to next.

Footnotes

1 I still outsource my personal email, and I sing the praises of the excellent folks behind ProtonMail.

2 My desktop email client also doubles as my newsreader, because, yes, of course you can still find me on USENET. Which, by the way, is undergoing a mini-revival

3 Why doesn’t DKIM sign all the headers in an email? Because intermediary servers and email clients will probably add their own headers, thereby invalidating the signature! DKIM gets used to sign the From: header, for obvious reasons, and ought to be used for other headers whose tampering could be significant such as the Date: and Subject:, but it’s really up to the signing server to choose a subset.

× × × × × × ×

Woodward Draw

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Screenshot showing a completed game of Woodward Draw; the final word was "goat".

  • Explore the set of 4 letter words
  • Either change one letter of the previous word
  • Or rearrange all the letters of the previous word
  • Find all 105 picture words!

Daniel Linssen (via itch.io)

Woodward Draw by Daniel Linssen is the kind of game that my inner Scrabble player both loves and hates. I’ve been playing on and off for the last three days to complete it, and it’s been great. While not perfectly polished1 and with a few rough edges2, it’s still a great example of what one developer can do with a little time.

It deserves a hat tip of respect, but I hope you’ll give it more than that by going and playing it (it’s free, and you can play online or download a copy3). I should probably check out their other games!

Footnotes

1 At one point the background colour, in order to match a picture word, changed to almost the same colour as the text of the three words to find!

2 The tutorial-like beginning is a bit confusing until you realise that you have to play the turn you’re told to, to begin with, for example.

3 Downloadable version is Windows only.

Dan Q found GC54E29 WAG 16 – Elvendon Valley

This checkin to GC54E29 WAG 16 - Elvendon Valley reflects a geocaching.com log entry. See more of Dan's cache logs.

The poor little geopup’s only got tiny legs, and the 8km we’ve walked so-far has got her pretty tired-out, so this’ll be the last cache of the series before we go and find ourselves some lunch and go home. It’s been a very enjoyable series so far, and I fully intend to return to complete it (and perhaps find some of those earlier caches that I failed to spot).

For this final cache of the morning (well, afternoon: barely!), I found the likely spot straightaway and picked up something that looked out of place. Nope, no sign of the cache though; that’s strange. It took a few seconds to realise that yes, the cache was hidden behind the thing I’d picked up… it was just also covered with leaf little and detritus. Soon had it retrieved in the end, though.

A huge number of butterflies flocked in the field to our right: it was quite impressive. I’ve snapped a picture showing just one, so that I can later look up what kind of butterfly it is!

Close-up photograph showing a butterfly atop a purple flower in a grassy meadow.

×