Articles – Page 5

Vmail via FreshRSS

It’s time for… Dan Shares Yet Another FreshRSS XPath Scraping Recipe!

Vmail

I’m a huge fan of the XPath scraping feature of FreshRSS, my favourite feed reader (and one of the most important applications in my digital ecosystem). I’ve previously demonstrated how to use the feature to subscribe to Forward, reruns of The Far Side, and new The Far Side content, despite none of those sites having “official” feeds.

Vmail is cool. It’s vole.wtf’s (of ARCC etc. fame) community newsletter, and it’s as batshit crazy as you’d expect if you were to get the kinds of people who enjoy that site and asked them all to chip in on a newsletter.

Totes bonkers.

But email’s not how I like to consume this kind of media. So obviously, I scraped it.

Recipe

Want to subscribe to Vmail using your own copy of FreshRSS? Here’s the settings you’re looking for –

Type of feed source: HTML + XPath (Web scraping)
XPath for finding news items: //table/tbody/tr
It’s just a table with each row being a newsletter; simple!
XPath for item title: descendant::a
XPath for item content: .
XPath for item link (URL): descendant::a/@href
XPath for item date: descendant::td[1]
Custom date/time format: d M *y
The dates are in a format that’s like 01 May ’24 – two-digit days with leading zeros, three-letter months, and a two-digit year preceded by a curly quote, separated by spaces. That curl quote screws up PHP’s date parser, so we have to give it a hint.
XPath for unique item ID: descendant::th
Optional, but each issue’s got its own unique ID already anyway; we might as well use it!
Article CSS selector on original website: #vmail
Optional, but recommended: this option lets you read the entire content of each newsletter without leaving FreshRSS.

So yeah, FreshRSS continues to be amazing. And lately it’s helped me keep on top of the amazing/crazy of vole.wtf too.

A Pressure Cooker for Tea

Duration 5:15

Podcast Version

This post is also available as a podcast. Listen here, download for later, or subscribe wherever you consume podcasts.

I’m not a tea-drinker¹. But while making a cuppa for Ruth this morning, a thought occurred to me and I can’t for a moment believe that I’m the first person to think of it:

What about a pressure-cooker, but for tea?²

Hear me out.

It’s been stressed how important it is that the water used to brew the tea is 100℃, or close to it possible. That’s the boiling point of water at sea level, so you can’t really boil your kettle hotter than that or else the water runs away to pursue a new life as a cloud.

That temperature is needed to extract the flavours, apparently³. And that’s why you can’t get a good cup of tea at high altitudes, I’m told: by the time you’re 3000 metres above sea level, water boils at around 90℃ and most British people wilt at their inability to make a decent cuppa⁴.

It’s a question of pressure, right? Increase the pressure, and you increase the boiling point, allowing water to reach a higher temperature before it stops being a liquid and starts being a gas. Sooo… let’s invent something!

I’m thinking a container about the size of a medium-sized Thermos flask or a large keep-cup – you need thick walls to hold pressure, obviously – with a safety valve and a heating element, like a tiny version of a modern pressure cooker. The top half acts as the lid, and contains a compartment into which you put your teabag or loose leaves (optionally in an infuser). After being configured from the front panel, the water gets heated to a specified temperature – which can be above the ambient boiling point of water owing to the pressurisation – at which point the tea is released from the upper half. The temperature is maintained for a specified amount of time and then the user is notified so they can release the pressure, open the top, lift out the inner cup, remove the teabag, and enjoy their beverage.

This isn’t just about filling the niche market of “dissatisfied high-altitude tea drinkers”. Such a device would also be suitable for other folks who want a controlled tea experience. You could have it run on a timer and make you tea at a particular time, like a teasmade. You can set the temperature lower for a controlled brew of e.g. green tea at 70℃. But there’s one other question that a device like this might have the capacity to answer:

What is the ideal temperature for making black tea?

We’re told that it’s 100℃, but that’s probably an assumption based on the fact that that’s as hot as your kettle can get water to go, on account of physics. But if tea is bad when it’s brewed at 90℃ and good when it’s brewed at 100℃… maybe it’s even better when it’s brewed at 110℃!

A modern pressure cooker can easily maintain a liquid water temperature of 120℃, enabling excellent extraction of flavour into water (this is why a pressure cooker makes such excellent stock).

A mug of tea held by the handle. — It’s possible that the perfect cup of tea *hasn’t been invented yet*, owing to limitations in the boiling point of water.

I’m not the person to answer this question, because, as I said: I’m not a tea drinker. But surely somebody’s tried this⁵? It shouldn’t be too hard to retrofit a pressure cooker lid with a sealed compartment that releases, even if it’s just on a timer, to deposit some tea into some superheated water?

Because maybe, just maybe, superheated water makes better tea. And if so, there’s a possible market for my proposed device.

Footnotes

¹ I probably ought to be careful confessing to that or they’ll strip my British citizenship.

² Don’t worry, I know better than to suggest air-frying a cup of ta. What kind of nutter would do that?

³ Again, please not that I’m not a tea-drinker so I’m not really qualified to comment on the flavour of tea at all, let alone tea that’s been brewed at too-low a temperature.

⁴ Some high-altitude tea drinkers swear by switching from black tea to green tea, white tea, or oolong, which apparently release their aromatics at lower temperatures. But it feels like science, not compromise, ought to be the solution to this problem.

⁵ I can’t find the person who’s already tried this, if they exist, but maybe they’re out there somewhere?

Building a Secret Cabinet

We’ve recently had the attics of our house converted, and I moved my bedroom up to one of the newly-constructed rooms.

To make the space my own, I did a little light carpentry up there: starting with a necessary reshaping of the doors, then moving on to shelving and eventually… a secret cabinet!

I’d love to tell you about how I built it: but first, a disclaimer! I am a software engineer, and with good reason. Letting me near a soldering iron is ill-advised. Letting me use a table saw is tempting fate.

Letting me teach you anything about how you should use a soldering iron or a table saw is, frankly, asking for trouble.

A spirit level on an unfinished shelf, under a window and beneath an uncarpeted floor. — Knowing that I’d been short on shelf space in my old bedroom, I started work on fitting shelves for my new bedroom before the carpet had even arrived.

Building a secret cabinet wasn’t part of my plan, but came about naturally after I got started. I’d bought a stack of pine planks and – making use of Ruth’s table saw – cut them to squarely fit beneath each of the two dormer windows¹. While sanding and oiling the wood I realised that I had quite a selection of similarly-sized offcuts and found myself wondering if I could find a use for them.

Dan drinks a 0% alcohol beer in front of three upright planks of wood. — The hardest part of sanding and oiling wood on the hottest day of the year is all the beer breaks you have to take. Such a drag.

I figured I had enough lumber to insert a small cabinet into one of the bookshelves, and that got me thinking… what about if it were a secret cabinet, disguised as books unless you knew where to look. Or to go one step further: what if it had some kind of electronic locking mechanism that could be triggered from somewhere else in the room².

Magpie decal 'perched' on a light switch. — There are other ways in which I’ve made my new room distinctly-“mine” – like the pair of magpies – but probably the secret cabinet is the most-distinctive.

Not wanting to destroy a stack of real books, which is the traditional way to get a collection of book spines for the purpose of decorating a “fake bookshelf” panel³, I looked online and discovered the company that made the fake book spines used at the shop of my former employer. They looked ideal: carefully shaped and painted panels with either an old-school or contemporary look.

Buuut, they don’t seem to be well-equipped for short runs and are doubtless pricey, so I looked elsewhere and found the eBay presence of Beatty Lockey Antiques in Loewstof. They’d acquired a stack of them second-hand from the set of Netflix’s The School for Good and Evil.⁴

(By the way: at time of writing they’ve still got a few panels left, if you want to make your own…)

I absolutely must sing the praises of Brad at Beatty Lockey Antiques who, after the first delivery of fake book fronts was partially-damaged in transit, was super quick about helping me find the closest-available equivalent (I’d already measured-up based on the one I’d thought I was getting) and sent a replacement.

The cabinet is just a few bits of wood glued together and reinforced with L-shaped corner braces, with a trio of thin strips – made from leftover architrave board – attached using small brass hinges. The fake book fronts are stuck to the strips using double-sided mounting tape left over from installing a bathroom mirror. A simple magnetic clasp holds the door shut when pushed closed⁵, and the hinges are inclined to “want” the door to stand half-open, which means it only needs a gentle push away from the magnetic catch to swing it open.

I mounted a Raspberry Pi Zero W into a rear corner inside the cabinet⁶, and wired it up via a relay to what was sold to me as a “large push-pull solenoid”, then began experimenting with the position in which I’d need to mount it to allow it to “kick” open the door, against the force of the magnetic clasp⁷.

This was, amazingly, the hardest part of the whole project! Putting the solenoid too close to the door didn’t work: it couldn’t “push” it from a standing start. Too far away, and the natural give of the door took the strain without pushing it open. Just the right distance, and the latch had picked up enough momentum that its weight “kicked” the door away from the magnet and followed-through to ensure that it kept moving.

A second solenoid, mounted inside the top of the cabinet, slides into the “loop” part of a large bolt fitting, allowing the cabinet to be electronically “locked”.

A Raspberry Pi Zero, relay, and solenoid assembly on the bottom outside edge of the inside of the cabinet. — I seriously must’ve spent about an hour getting the position of that little “kicker” in the bottom right *just right*.

Next up came the software. I started with a very simple Python program⁸ that would run a webserver and, on particular requests, open the lock solenoid and push with the “kicker” solenoid.

#!/usr/bin/python
#
# a basic sample implementation of a web interface for a secret cabinet
#
# setup:
#   sudo apt install -y python3-flask
#   wget https://github.com/sbcshop/Zero-Relay/blob/master/pizero_2relay.py
#
# running:
#   sudo flask --app web run --host=0.0.0.0 --port 80

from flask import Flask, redirect, url_for
import pizero_2relay as pizero
from time import sleep

# set up pizero_2relay with the two relays attached to this Pi Zero:
r1 = pizero.relay("R1") # The "kicker" relay
r2 = pizero.relay("R2") # The "locking bolt" relay

app = Flask(__name__)

# GET / - nothing here
@app.route("/")
def index():
  return "Nothing to see here."

# GET /relay - show a page with "open" and "lock" links
@app.route("/relay")
def relay():
  return "<html><head><meta name='viewport' content='width=device-width, initial-scale=1'></head><body><ul><li><a href='/relay/open'>Open</a></li><li><a href='/relay/lock'>Lock</a></li></ul>"

# GET /relay/open - open the secret cabinet then return to /relay
# This ought to be a POST request in your implementation, and you probably
# want to add some security e.g. a 
@app.route("/relay/open")
def open():
  # Retract the lock:
  r2.off()
  sleep(0.5)
  # Fire the kicker twice:
  r1.on()
  sleep(0.25)
  r1.off()
  sleep(0.25)
  r1.on()
  sleep(0.25)
  r1.off()
  # Redirect back:
  return redirect(url_for('relay'))

@app.route("/relay/lock")
def lock():
  # Engage the lock:
  r2.on()
  return redirect(url_for('relay'))

Don’t use this code as-is on any kind of open network, obviously. Follow the comments for some tips on what you’ll need to change.

Once I had something I could trigger from a web browser or with curl, I could start experimenting with trigger mechanisms. I had a few ideas (and prototyped a couple of them), including:

A mercury tilt switch behind a different book, so you pull it to release the cabinet in the style of a classic movie secret door.
A microphone that listens for a specific pattern of knocks on a nearby surface.

I had far too much fun playing about with crappy prototypes.

An RFID reader mounted underneath another surface, and a tag on the underside of an ornament: moving the ornament to the “right” place on the surface triggers the cabinet (elsewhere in the room).
The current design, shown in the video above, where a code⁹ is transmitted to the cabinet for verification.

I think I’m happy with what I’ve got going on with it now. And it’s been a good opportunity to improve my carpentry, electronics, and Python.

Footnotes

¹ The two dormer windows, wouldn’t you guarantee it, were significantly different widths despite each housing a window of the same width. Such are the quirks of extending a building that the previous occupier had previously half-heartedly tried to extend, I guess.

² Why yes, I am a big fan of escape rooms. Why do you ask?

³ For one thing, I live with JTA, and I’m confident that he’d somehow be able to hear the silent screams of whatever trashy novels I opted to sacrifice for the good of the project.

⁴ As a bonus, my 10-year-old is a big fan of the book series that inspired the film (and a more-muted fan of the film itself) and she was ever-so excited at my project using real-life parts of the set of the movie… that she’s asked me to make a similar secret cabinet for her, when we get around to redecorating her room later in the year!

⁵ If I did it again, I might consider using a low-powered electromagnetic lock to hold the door shut. In this design, I used a permanent magnet and a pair of latch solenoids: one to operate a bolt, the second to “kick” the door open against the pull of the magnet, and… it feels a little clumsier than a magnetic lock might’ve.

⁶ That double-sided mounting tape really came in handy for this project!

⁷ Props to vlogger Technology Connections, one of whose excellent videos on the functionality of 1970s pinball tables – maybe this one? – taught me what a latch solenoid was in the first place, last year, which probably saved me from the embarrassment of trying to do this kind of thing with, I don’t know, a stepper motor or something.

⁸ I’m not a big fan of Python normally, but the people who made my relays had some up with a convenience library for them that was written in it, so I figured it would do.

⁹ Obviously the code isn’t A-B; I changed it temporarily for the video.

Ladybird Browser

I’ve been playing with the (pre-Alpha version of) Ladybird, and it fills me with such joy and excitement.

Browser diversity

Back in 2018, while other Web developers were celebrating, I expressed my dismay at the news that Microsoft Edge was on the cusp of switching from using Microsoft’s own browser engine EdgeHTML to using Blink. Blink is the engine that powers almost all other mainstream browsers; all but Firefox, which continues to stand atop Gecko.

The developers who celebrated this loss of rendering engine diversity were, I suppose, happy to have one fewer browser in which they must necessarily test their work. I guess these are the same developers who don’t test the sites they develop for accessibility (does your site work if you can’t see the images? what about with a keyboard but without a pointing device? how about if you’re colourblind?), or consider what might happen if a part of their site fails (what if the third-party CDN that hosts your JavaScript libraries goes down or is blocked by the user’s security software or their ISP?).

But I was sad, because – as I observed after Andre Alves Garzia succinctly spelled it out – browser engines are an endangered species. Building a new browser that supports the myriad complexities of the modern Web is such a huge endeavour that it’s unlikely to occur from scratch: from this point on, all “new” browsers are likely to be based upon an existing browser engine.

Engine diversity is important. Last time we had a lull in engine diversity, the Web got stuck, stagnating in the shadow of Internet Explorer 6’s dominance and under the thumb of Microsoft’s interests. I don’t want those days to come back; that’s a big part of why Firefox is my primary web browser.

A Ladybird book browser

Spoof cover for "The Ladybird Book of The Browser" — I actually still own a copy of the book from which I adapted this cover!

Ladybird is a genuine new browser engine. Y’know, that thing I said that we might never see happen again! So how’ve they made it happen?

It helps that it’s not quite starting from scratch. It’s starting point is the HTML viewer component from SerenityOS. And… it’s pretty good. It’s DOM processing’s solid, it seems to support enough JavaScript and CSS that the modern Web is usable, even if it’s not beautiful 100% of the time.

They’re not even expecting to make an Alpha release until next year! Right now if you want to use it at all, you’re going to need to compile the code for yourself and fight with a plethora of bugs, but it works and that, all by itself, is really exciting.

They’ve got four full-time engineers, funded off donations, with three more expected to join, plus a stack of volunteer contributors on Github. I’ve raised my first issue against the repo; sadly my C++ probably isn’t strong enough to be able to help more-directly, even if I somehow did have enough free time, which I don’t. But I’ll be watching-from-afar this wonderful, ambitious, and ideologically-sound initiative.

#100DaysToOffload

Woop! This is my 100th post of the year (stats), even using my more-conservative/pedant-friendly “don’t count checkins/reposts/etc. rule. If you’re not a pedant, I achieved #100DaysToOffload when I found a geocache alongside Regents Canal while changing trains to go to Amsterdam where I played games with my new work team, looked at windows and learned about how they’ve been taxed, and got nerdsniped by a bus depot. In any case: whether you’re a pedant or not you’ve got to agree I’ve achieved it now!

Science Weekend

This weekend was full of science.

Research

String of large electromagnets used for steering an electron beam. — If you wanna bend a stream of electrons travelling at nearly the speed of light, you’re gonna need a lot of big magnets.

This started on Saturday with a trip to the Harwell Campus, whose first open day in eight years provided a rare opportunity for us to get up close with cutting edge science (plus some very kid-friendly and accessible displays) as well as visit the synchrotron at Diamond Light Source.

Dan with a child in front of beamlines coming out of the Diamond Light Synchrotron ring. — It’s hard to convey the scale of the thing; turns out you need a big ol’ ring if you want to spin electrons fast enough to generate a meaningful amount of magnetobremsstrahlung radiation.

The whole thing’s highly-recommended if you’re able to get to one of their open days in the future, give it a look. I was particularly pleased to see how enthused about science it made the kids, and what clever questions they asked.

For example: the 7-year-old spent a long time cracking a variety of ciphers in the computing tent (and even spotted a flaw in one of the challenge questions that the exhibitors then had to hand-correct on all their handouts!); the 10-year-old enjoyed quizzing a researcher who’d been using x-ray crystallography of proteins.

Medicine

And then on Sunday I finally got a long-overdue visit to my nearest spirometry specialist for a suite of experiments to try to work out what exactly is wrong with my lungs, which continue to be a minor medical mystery.

Dan holds a piece of medical apparatus to his mouth. — “Once you’ve got your breath back, let’s fill you with drugs and do those experiments again…”

It was… surprisingly knackering. Though perhaps that’s mostly because once I was full of drugs I felt briefly superpowered and went running around the grounds of the wonderfully-named Brill Hill Windmill with the dog until was panting in pretty much the way that I might normally have been, absent an unusually-high dose of medication.

For amusement purposes alone, I’d be more-likely to recommend the first day’s science activities than the second, but I can’t deny that it’s cool to collect a load of data about your own body and how it works in a monitorable, replicable way. And maybe, just maybe, start to get to the bottom of why my breathing’s getting so much worse these last few years!

A Stupid Joke About Elephants

Duration 2:48

Podcast Version

This post is also available as a podcast. Listen here, download for later, or subscribe wherever you consume podcasts.

You’ve probably been taught that you can tell the difference between African Elephants and Indian Elephants by looking at their head and ears. The larger African Elephants have a rounded cranium and big ears (with a shape somewhat like the continent of Africa itself!), whereas the smaller Indian Elephants have a two-lobed skull and diminutive ears that tuck tidily alongside their heads.

But suppose you don’t manage to get a glimpse at the front end of the elephant as it passes you. What hope is there of identifying the species? Well: you can look at its back!

African Elephants, it turns out, have a concave back, whereas Indian elephants have a convex back (a bit like a hump)!

I was having difficulty sleeping one night during the UK‘s current heatwave, so naturally I opted to practice my newfound ability to distinguish elephant species by their spines. Indian, Indian, African, Indian, African, African… etc.

And then I came across this one:

African Elephant backs are concave. Indian Elephant backs are convex. But what does it mean when you see a flat elephant’s back?

It turns out…

…

…that’s a grey area.

Special Roads

Duration 7:19

Podcast Version

This post is also available as a podcast. Listen here, download for later, or subscribe wherever you consume podcasts.

Sometimes I’ve seen signs on dual carriageways and motorways that seem to specify a speed limit that’s the same as the national speed limit (i.e. 60 or 70 mph for most vehicles, depending on the type of road), which seem a bit… pointless? Today I learned why they’re there, and figured I’d share with you!

To get there, we need a history lesson.

As early as the 1930s, it was becoming clear that Britain might one day need a network of high-speed, motor-vehicle-only roads: motorways. The first experimental part of this network would be the Preston By-pass¹.

Construction wouldn’t actually begin until the 1950s, and it wasn’t just the Second World War that got in the way: there was a legislative challenge too.

When the Preston By-pass was first conceived, there was no legal recognition for roads that restricted the types of traffic that were permitted to drive on them. If a public highway were built, it would have to allow pedestrians, cyclists, and equestrians, which would doubtless undermine the point of the exercise! Before it could be built, the government needed to pass the Special Roads Act 1949, which enabled the designation of public roads as “special roads”, to which entry could be limited to certain classes of vehicles².

Monochrome photograph showing a sign at the entrance to the Preston By-pass, reading: 'Motorway. NO L-drivers, mopeds, motorcycles under 50cc., invalid-carriages, pedal-cycles, pedestrians, animals'. — The original motorways had to spell out the regulations at their junctions.

If you don’t check your sources carefully when you research the history of special roads, you might be taken in by articles that state that special roads are “now known as motorways”, which isn’t quite true. All motorways are special roads, by definition, but not all special roads are motorways.

There’s maybe a dozen or more non-motorway special roads, based on research by Pathetic Motorways (whose site was amazingly informative on this entire subject). They tend to be used in places where something is like a motorway, but can’t quite be a motorway. In Manchester, a couple of the A57(M)’s sliproads have pedestrian crossings and so have to be designated special roads rather than motorways, for example³.

Now we know what special roads are, that we might find them all over the place, and that they can superficially look like motorways, let’s talk about speed limits.

The Road Traffic Act 1934 introduced the concept of a 30mph “national speed limit” in built-up areas, which is still in force today. But outside of urban areas there was no speed limit. Perhaps there didn’t need to be, while cars were still relatively slow, but automobiles became increasingly powerful. The fastest speed ever legally achieved on a British motorway came in 1964 during a test by AC Cars, when driver Jack Sears reached 185mph.

In the late 1960s an experiment was run in setting a speed limit on motorways of 70mph. Then the experiment was extended. Then the regulation was made permanent.

There’ve been changes since then, e.g. to prohibit HGVs from going faster than 60mph, but fundamentally this is where Britain’s national speed limit on motorways comes from.

You’ve probably spotted the quirk already. When “special roads” were created, they didn’t have a speed limit. Some “special roads” were categorised as “motorways”, and “motorways” later had a speed limit imposed. But there are still a few non-motorway “special roads”!

Putting a national speed limit sign on a special road would be meaningless, because these roads have no centrally-legislated speed limit. So they need a speed limit sign, even if that sign, confusingly, might specify a speed limit that matches what you’d have expected on such a road⁴. That’s the (usual) reason why you sometimes see these surprising signs.

As to why this kind of road are much more-common in Scotland and Wales than they are anywhere else in the UK: that’s a much deeper-dive that I’ll leave as an exercise for the reader.

Footnotes

¹ The Preston By-pass lives on, broadly speaking, as the M6 junctions 29 through 32.

² There’s little to stop a local authority using the powers of the Special Roads Act and its successors to declare a special road accessible to some strange and exotic permutation of vehicle classes if they really wanted: e.g. a road could be designated for cyclists and horses but forbidden to motor vehicles and pedestrians, for example! (I’m moderately confident this has never happened.)

³ There’s a statutory instrument that makes those Mancunian sliproads possible, if you’re having trouble getting to sleep on a night and need some incredibly dry reading.

⁴ An interesting side-effect of these roads might be that speed restrictions based on the class of your vehicle and the type of road, e.g. 60mph for lorries on motorways, might not be enforceable on special roads. If you wanna try driving your lorry at 70mph on a motorway-like special road with “70” signs, though, you should do your own research first; don’t rely on some idiot from the Internet. I Am Not A Lawyer etc. etc.

ARCC

In the late ’70s, a shadowy group of British technologists concluded that nuclear war was inevitable and secretly started work on a cutting-edge system designed to help rebuild society. And thanks to Matt Round-and-friends at vole.wtf (who I might have mentioned before), the system they created – ARCC – can now be emulated in your browser.

3D rendering of an ARCC system, by HappyToast.

I’ve been playing with it on-and-off all year, and I’ve (finally) managed to finish exploring pretty-much everything the platform currently has to offer, which makes it pretty damn good value for money for the £6.52 I paid for my ticket (the price started at £2.56 and increases by 2p for every ticket sold). But you can get it cheaper than I did if you score 25+ on one of the emulated games.

Most of what I just told you is true. Everything… except the premise. There never was a secretive cabal of engineers who made this whackballs computer system. What vole.wtf emulates is an imaginary system, and playing with that system is like stepping into a bizarre alternate timeline or a weird world. Over several separate days of visits you’ll explore more and more of a beautifully-realised fiction that draws from retrocomputing, Cold War fearmongering, early multi-user networks with dumb terminal interfaces, and aesthetics that straddle the tripoint between VHS, Teletext, and BBS systems. Oh yeah, and it’s also a lot like being in a cult.

Needless to say, therefore, it presses all the right buttons for me.

If you enjoy any of those things, maybe you’d like this too. I can’t begin to explain the amount of work that’s gone into it. If you’re looking for anything more-specific in a recommendation, suffice to say: this is a piece of art worth seeing.

Somewhat-Effective Spam Filters

I’ve tried a variety of unusual strategies to combat email spam over the years.

Here are some of them (each rated in terms the geekiness of its implementation and its efficacy), in case you’d like to try any yourself. They’re all still in use in some form or another:

Spam filters

Geekiness: 1/10
Efficacy: 5/10

Your email provider or your email software probably provides some spam filters, and they’re probably pretty good. I use Proton‘s and, when I’m at my desk, Thunderbird‘s. Double-bagging your spam filter only slightly reduces the amount of spam that gets through, but increases your false-positive rate and some non-spam gets mis-filed.

A particular problem is people who email me for help after changing their name on FreeDeedPoll.org.uk, probably because they’re not only “new” unsolicited contacts to me but because by definition many of them have strange and unusual names (which is why they’re emailing me for help in the first place).

Frankly, spam filters are probably enough for many people. Spam filtering is in general much better today than it was a decade or two ago. But skim the other suggestions in case they’re of interest to you.

Unique email addresses

Geekiness: 3/10
Efficacy: 8/10

If you give a different email address to every service you deal with, then if one of them misuses it (starts spamming you, sells your data, gets hacked, whatever), you can just block that one address. All the addresses come to the same inbox, for your convenience. Using a catch-all means that you can come up with addresses on-the-fly: you can even fill a paper form with a unique email address associated with the company whose form it is.

On many email providers, including the ever-popular GMail, you can do this using plus-sign notation. But if you want to take your unique addresses to the next level and you have your own domain name (which you should), then you can simply redirect all email addresses on that domain to the same inbox. If Bob’s Building Supplies wants your email address, give them bobs@yourname.com, which works even if Bob’s website erroneously doesn’t accept email addresses with plus signs in them.

This method actually works for catching people misusing your details. On one occasion, I helped a band identify that their mailing list had been hacked. On another, I caught a dodgy entrepreneur who used the email address I gave to one of his businesses without my consent to send marketing information of a different one of his businesses. As a bonus, you can set up your filtering/tagging/whatever based on the incoming address, rather than the sender, for the most accurate finding, prioritisation, and blocking.

Also, it makes it easy to have multiple accounts with any of those services that try to use the uniqueness of email addresses to prevent you from doing so. That’s great if, like me, you want to be in each of three different Facebook groups but don’t want to give Facebook any information (not even that you exist at the intersection of those groups).

Signed unique email addresses

Geekiness: 10/10
Efficacy: 2/10

Unique email addresses introduce two new issues: (1) if an attacker discovers that your Dreamwidth account has the email address dreamwidth@yourname.com, they can probably guess your LinkedIn email, and (2) attackers will shotgun “likely” addresses at your domain anyway, e.g. admin@yourname.com, management@yourname.com, etc., which can mean that when something gets through you get a dozen copies of it before your spam filter sits up and takes notice.

What if you could assign unique email addresses to companies but append a signature to each that verified that it was legitimate? I came up with a way to do this and implemented it as a spam filter, and made a mobile-friendly webapp to help generate the necessary signatures. Here’s what it looked like:

The domain directs all emails at that domain to the same inbox.
If the email address is on a pre-established list of valid addresses, that’s fine.
Otherwise, the email address must match the form of:
- A string (the company name), followed by
- A hyphen, followed by
- A hash generated using the mechanism described below, then
- The @-sign and domain name as usual

The hashing algorithm is as follows: concatenate a secret password that only you know with a colon then the “company name” string, run it through SHA1, and truncate to the first eight characters. So if my password were swordfish1 and I were generating a password for Facebook, I’d go:

SHA1 ( swordfish1 : facebook) [ 0 ... 8 ] = 977046ce
Therefore, the email address is facebook-977046ce@myname.com
If any character of that email address is modified, it becomes invalid, preventing an attacker from deriving your other email addresses from a single point (and making it hard to derive them given multiple points)

I implemented the code, but it soon became apparent that this was overkill and I was targeting the wrong behaviours. It was a fun exercise, but ultimately pointless. This is the one method on this page that I don’t still use.

Honeypots

Geekiness: 8/10
Efficacy: ?/10

A honeypot is a “trap” email address. Anybody who emails it get aggressively marked as a spammer to help ensure that any other messages they send – even to valid email addresses – also get marked as spam.

I litter honeypots all over the place (you might find hidden email addresses on my web pages, along with text telling humans not to use them), but my biggest source of honeypots is formerly-valid unique addresses, or “guessed” catch-all addresses, which already attract spam or are otherwise compromised!

I couldn’t tell you how effective it is without looking at my spam filter’s logs, and since the most-effective of my filters is now outsourced to Proton, I don’t have easy access to that. But it certainly feels very satisfying on the occasions that I get to add a new address to the honeypot list.

Instant throwaways

Geekiness: 5/10
Efficacy: 6/10

OpenTrashmail is an excellent throwaway email server that you can deploy in seconds with Docker, point some MX records at, and be all set! A throwaway email server gives you an infinite number of unique email addresses, like other solutions described above, but with the benefit that you never have to see what gets sent to them.

If you offer me a coupon in exchange for my email address, it’s a throwaway email address I’ll give you. I’ll make one up on the spot with one of my (several) trashmail domains at the end of it, like justgivemethedamncoupon@danstrashmailserver.com. I can just type that email address into OpenTrashmail to see what you sent me, but then I’ll never check it again so you can spam it to your heart’s content.

As a bonus, OpenTrashmail provides RSS feeds of inboxes, so I can subscribe to any email-based service using my feed reader, and then unsubscribe just as easily (without even having to tell the owner).

Summary

With the exception of whatever filters your provider or software comes with, most of these options aren’t suitable for regular folks. But you’re only a domain name (assuming you don’t have one already) away from being able to give unique email addresses to everybody you deal with, and that’s genuinely a game-changer all by itself and well worth considering, in my opinion.

Woke Kids

The other weekend, I joined in with the parade at Witney Pride, accompanied by our 10-year-old who’d expressed an interest in coming too.

It was her first Pride but she clearly got the idea, turning up with a wonderful hand-coloured poster she’d made which, in rainbow colours, encouraged the reader to “be kind”.

A Pride parade marches down a high street: Dan and his eldest can be seen in the very background. — You’ve seen pictures of Pride parades before, possibly even ones with me in them.

You know what: our eldest is so woke it makes me embarrassed on behalf of my past self at her age. Or even at twice her age, when I still didn’t have the level of social and societal awareness and care about queer issues that she does already.

A tweeny girl and a 40-something man with rainbows painted on their faces wave flags in a Pride parade. The child has coloured-in a poster saying "be kind". — I’d equipped her with a whistle (on a rainbow lanyard) and instructions that in the event of protests from religious nuts she shouldn’t engage with them (because that’s what they want) but instead just to help ensure that our parade was *louder* than them! I needn’t have worried though: Witney ain’t Oxford or London and our march seemed to see nothing but joy and support from the folks we passed.

When we got to the parade’s destination, the kid found a stall selling a variety of badges, and selected for herself a “she/her/hers” pronoun pin.

“It’s not like anybody’s likely to look at me and assume that my pronouns are anything other than that,” she explained, “But I want it to be normal to talk about, and I want to show solidarity for genderqueer people.”

That’s a level of allyship that it took me until I was much, much older to attain. So proud!

Framing Device

Doors

As our house rennovations/attic conversions come to a close, I found myself up in what will soon become my en suite, fitting a mirror, towel rail, and other accessories.

Wanting to minimise how much my power tool usage disturbed the rest of the house, I went to close the door separating my new bedroom from my rest of my house, only to find that it didn’t properly fit its frame and instead jammed part-way-closed.

“Oh,” I said, as the door clearly failed to shut, “Damn.”

Somehow we’d never tested that this door closed properly before we paid the final instalment to the fitters. And while I’m sure they’d have come back to repair the problem if I asked, I figured that it’d be faster and more-satisfying to fix it for myself.

Homes

As a result of an extension – constructed long before we moved in – the house in Preston in which spent much of my childhood had not just a front and a back door but what we called the “side door”, which connected the kitchen to the driveway.

Unfortunately the door that was installed as the “side door” was really designed for interior use and it suffered for every winter it faced the biting wet North wind.

My father’s DIY skills could be rated as somewhere between mediocre and catastrophic, but his desire to not spend money “frivolously” was strong, and so he never repaired nor replaced the troublesome door. Over the course of each year the wood would invariably absorb more and more water and swell until it became stiff and hard to open and close.

The solution: every time my grandfather would visit us, each Christmas, my dad would have his dad take down the door, plane an eighth of an inch or so off the bottom, and re-hang it.

Sometimes, as a child, I’d help him do so.

Planes

The first thing to do when repairing a badly-fitting door is work out exactly where it’s sticking. I borrowed a wax crayon from the kids’ art supplies, coloured the edge of the door, and opened and closed it a few times (as far as possible) to spot where the marks had smudged.

Fortunately my new bedroom door was only sticking along the top edge, so I could get by without unmounting it so long as I could brace it in place. I lugged a heavy fence post rammer from the garage and used it to brace the door in place, then climbed a stepladder to comfortably reach the top.

A small box plane perched atop a sloping door. — I figured I’d only need to remove a few millimetres, so I didn’t mind doing it from atop a stepladder. Hey: here’s a fun thing – when I think about planing a door with my grandfather, I think in inches; when I think about doing it myself, I think in metric!

Loss

After my paternal grandfather died, there was nobody left who would attend to the side door of our house. Each year, it became a little stiffer, until one day it wouldn’t open at all.

Surely this would be the point at which he’d pry open his wallet and pay for it to be replaced?

A middle-aged man carrying walking poles on an urban riverbank drags a car tyre that's chained to his waist. — I’m not sure there’s a more apt metaphor for my dad’s ability to be stubborn than this photo of him dragging a tyre around Gateshead as a training activity for an Arctic expedition.

Nope. Instead, he inexpertly screwed a skirting board to it and declared that it was now no-longer a door, but a wall.

I suppose from a functionalist perspective he was correct, but it still takes a special level of boldness to simply say “That door? It’s a wall now.”

Sand

Of all the important tasks a carpenter (or in this case, DIY-er) must undertake, hand sanding must surely be the least-satisfying.

But reaching the end of the process, the feel of a freshly-planed, carefully-sanded piece of wood is fantastic. This surface represented chaos, and now it represents order. Order that you yourself have brought about.

Often, you’ll be the only one to know. When my grandfather would plane and sand the bottom edge of our house’s side door, he’d give it a treatment of oil (in a doomed-to-fail attempt to keep the moisture out) and then hang it again. Nobody can see its underside once it’s hung, and so his handiwork was invisible to anybody who hadn’t spent the last couple of months swearing at the stiffness of the door.

A paintbrush applies white paint to the top of a door. — Swish, swish. *Now* I’m glad I sanded.

Even though the top of my door is visible – particularly visible, given its sloping face – nobody sees the result of the sanding because it’s hidden beneath a layer of paint.

A few brush strokes provide the final touch to a spot of DIY… that in provided a framing device for me to share a moment of nostalgia with you.

Sweep away the wood shavings. Keep the memories.

Multi-Phase Maps in FoundryVTT

FoundryVTT is a fantastic Web-based environment for tabletop roleplaying adventures¹ and something I particularly enjoy is the freedom for virtually-unlimited scripting. Following a demonstration to a fellow DM at work last week I promised to throw together a quick tutorial into scripting simple multi-phase maps using Foundry.²

Why multi-phase maps?

Animated battlemap which organically grows into a leafy flower over six stages. — For this demonstration, I’ll be using AtraxianBear’s Growing Flower Dungeon.

You might use a multi-phase map to:

Allow the development and expansion of a siege camp outside the fortress where the heroes are holed-up.³
Rotate through day and night cycles or different times of day, perhaps with different things to interact with in each.⁴
Gradually flood a sewer with rising water… increasing the range of the monster that dwells within.⁵
Re-arrange parts of the dungeon when the characters flip certain switches, opening new paths… and closing others.

I’ll use the map above to create a simple linear flow, powered by a macro in the hotbar. Obviously, more-complex scenarios are available, and combining this approach with a plugin like Monk’s Active Tile Triggers can even be used to make the map appear to dynamically change in response to the movement or actions of player characters!

Setting the scene

Create a scene, using the final state of the map as the background. Then, in reverse-order, add the previous states as tiles above it.

Not shown, but highly-recommended: lock each tile when you’re done placing it, so that you don’t accidentally interact with it when you mean to e.g. drag-select multiple actors.

Make a note of the X-position that your tiles are in when they’re where they supposed to be: we’ll “move” the tiles off to the side when they’re hidden, to prevent their ghostly half-hidden forms getting in your way as game master. We’ll also use this X-position to detect which tiles have already been moved/hidden.

Also make note of each tile’s ID, so your script can reference them. It’s easiest to do this as you go along. When you’re ready to write your macro, reverse the list, because we’ll be hiding each tile in the opposite order from the order you placed them.

Writing the script

Next, create a new script macro, e.g. by clicking an empty slot in the macro bar. When you activate this script, the map will move forward one phase (or, if it’s at the end, it’ll reset).

I find Foundry’s built-in script editor a little… small? So I write my scripts in my favourite text editor and then copy-paste.

Here’s the code you’ll need – the 👈 emoji identifies the places you’ll need to modify the code, specifically:

const revealed_tiles_default_x = 250 should refer to the X-position of your tiles when they’re in the correct position.
const revealed_tiles_modified_x = 2825 should refer to the X-position they’ll appear at “off to the right” of your scene. To determine this, just move one tile right until it’s sufficiently out of the way of the battlemap and then check what it’s X-position is! Or just take the default X-position, add the width of your map in pixels, and then add a tiny bit more.
const revealed_tiles = [ ... ] is a list of the tile IDs of each tile what will be hidden, in turn. In my example there are five of them (the sixth and final image being the scene background).

const revealed_tiles_default_x = 250;   // 👈 X-position of tiles when displayed
const revealed_tiles_modified_x = 2825; // 👈 X-position of tiles when not displayed
const revealed_tiles = [
  '2xG7S8Yqk4x1eAdr',                   // 👈 list of tile IDs in order that they should be hidden
  'SjNQDBImHvrjAHWX',                   //     (top to bottom)
  'tuYg4FvLgIla1l21',
  'auX4sj64PWmkAteR',
  'yAL4YP0I4Cv4Sevt',
].map(t=>canvas.tiles.get(t));

/*************************************************************************************************/

// Get the topmost tile that is still visible:
const next_revealed_tile_to_move = revealed_tiles.find(t=>
  t.position.x == revealed_tiles_default_x
);

// If there are NO still-visible tiles, we must need to reset the map:
if( ! next_revealed_tile_to_move ) {
  // To reset the map, we go through each tile and put it back where it belongs -
  for(tile of revealed_tiles){
    canvas.scene.updateEmbeddedDocuments("Tile", [ {
      _id: tile.id,
      x: revealed_tiles_default_x,
      hidden: false
    } ]);
  }
} else {
  // Otherwise, hide the topmost visible tile (and move it off to the side to help the GM) -
  canvas.scene.updateEmbeddedDocuments("Tile", [ {
    _id: next_revealed_tile_to_move.id,
    x: revealed_tiles_modified_x,
    hidden: true
  } ]);
}

I hope that the rest of the code is moderately self-explanatory for anybody with a little JavaScript experience, but if you’re just following this kind of simple, linear case then you don’t need to modify it anyway. But to summarise, what it does is:

Finds the first listed tile that isn’t yet hidden (by comparing its X-position to the pre-set X-position).
If there aren’t any such tiles, we must have hidden them all already, so perform a reset: to do this – iterate through each tile and set its X-position to the pre-set X-position, and un-hide it.
Otherwise, move the first not-hidden tile to the alternative X-position and hide it.

I hope you have fun with scripting your own multi-phase maps. Just don’t get so caught-up in your awesome scenes that you fail to give the players any agency!

Footnotes

¹ Also, it’s on sale at 20% off this week to celebrate its fourth anniversary. Just sayin’.

² I can neither confirm nor deny that a multi-phase map might be in the near future of The Levellers‘ adventure…

³ AtraxianBear has a great series of maps inspired by the 1683 siege of Vienna by the Ottomans that could be a great starting point for a “gradually advancing siege” map.

⁴ If you’re using Dungeon Alchemist as part of your mapmaking process you can just export orthographic or perspective outputs with different times of day and your party’s regular inn can be appropriately lit for any time of day, even if the party decides to just “wait at this table until nightfall”.

⁵ Balatro made a stunning map with rising water as a key feature: there’s a preview available.

Don’t Ask, Don’t Teach

Politics and pundits

The UK’s Conservative government, having realised that their mandate is worthless, seems to be in a panicked rush to try to get the voters to ignore any of the real issues. Instead, they say, we should be focussed on things like ludicrously-expensive and ineffective ways to handle asylum seekers and making life as hard as possible for their second-favourite scapegoat: trans and queer people.

The latest move in that second category seems likely to be a plan to, among other things, discourage teachers from talking about gender identity in schools, with children of any age. From the article I linked:

The BBC has not seen the new guidelines but a government source said they included plans to ban any children being taught about gender identity.

If asked, teachers will have to be clear gender ideology is contested.

Needless to say, such guidance is not likely to be well-received by teachers:

Pepe Di’Iasio, headteacher at a school in Rotherham, told Today that he believes pupils are being used “as a political football”.

Teachers “want well informed and evidence-based decisions”, he said, and not “politicised” guidance.

Cringey political poster reading "Is this Labour's idea of a comprehensive education? Take the politics out of education, vote Conservative", alongside three books: Young gay & proud, Police: Out of School!, and The playbook for kids about sex. — I can only assume that the Tories still have a stack of this genuine 1987 billboard poster (ugh) in stock, and are hoping to save money by reusing them.

People and pupils

This shit isn’t harmless. Regardless of how strongly these kinds of regulations are enforced, they can have a devastating chilling effect in schools.

I speak from experience.

A group of teenagers stand around awkwardly. — I don’t know if this is the “most-90s” photo I own of myself, but it’s gotta be close. Taken at the afterparty from a school production of *South Pacific*, so probably at least a *little* disproportionately-queer gathering.

Most of my school years were under the shadow of Section 28. Like I predict for the new Conservative proposals, Section 28 superficially didn’t appear to have a major impact: nobody was ever successfully prosecuted under it, for example. But examining its effects in that way completely overlooks the effect it had on how teachers felt they had to work.

For example…

In around 1994, I witnessed a teacher turn a blind eye to homophobic bullying of a pupil by their peer, during a sex education class. Simultaneously, the teacher coolly dismissed the slurs of the bully, saying that we weren’t “talking about that in this class” and that the boy should “save his chatter for the playground”. I didn’t know about the regulations at the time: only in hindsight could I see that this might have been a result of Section 28. All I got to see at the time was a child who felt that his homophobic harassment of his classmate had the tacit endorsement of the teachers, so long as it didn’t take place in the classroom.

A gay friend, who will have been present but not involved in the above event, struggled with self-identity and relationships throughout his teenage years, only “coming out” as an adult. I’m confident that he could have found a happier, healthier life had he felt supported – or at the very least not-unwelcome – at school. I firmly believe that the long-running third-degree side-effects of Section 28 effectively robbed him of a decade of self-actualisation about his identity.

The long tail of those 1980s rules were felt long-after they were repealed. And for a while, it felt like things were getting better. But increasingly it feels like we’re moving backwards.

A pride rainbow painted down the back of a white person's first, held in the air. — As a country and as a society, we can do better than this.

With general elections coming up later this year, it’ll soon be time to start quizzing your candidates on the issues that matter to you. Even (perhaps especially) if your favourite isn’t the one who wins, it can be easiest to get a politicians’ ear when they and their teams are canvassing for your vote; so be sure to ask pointed questions about the things you care about.

I hope that you’ll agree that not telling teachers to conceal from teenagers the diversity of human identity and experience is something worth caring about.

Update: Only a couple of hours after I posted this, the awesome folks (whom I’ve mentioned before) at the Vagina Museum tooted a thread about the long tail of Section 28. It’s well-worth a read.

Blog to 5K

This is my 5,000th post on this blog.

Okay, we’re gonna need a whole lot of caveats on the “this is 5,000” claim:

Engage pedantry mode

First, there’s a Ship of Theseus consideration. By “this blog”, I’m referring to what I feel is a continuation (with short breaks) of my personal diary-style writing online from the original “Avatar Diary” on castle.onza.net in the 1990s via “Dan’s Pages” on avangel.com in the 2000s through the relaunch on scatmania.org in 2003 through migrating to danq.me in 2012. If you feel that a change of domain precludes continuation, you might disagree with me. Although you’d be a fool to do so: clearly a blog can change its domain and still be the same blog, right? Back in 2018 I celebrated the 20th anniversary of my first blog post by revisiting how my blog had looked, felt, and changed over the decades, if you’re looking for further reading.

Similarly, one might ask if retroactively republishing something that originally went out via a different medium “counts”².

In late 1999 I ran “Cool Thing of the Day (to do at the University of Wales, Aberystwyth)” as a way of staying connected to my friends back in Preston as we all went our separate ways to study. Initially sent out by email, I later maintained a web page with a log of the entries I’d sent out, but the address wasn’t publicly-circulated. I consider this to be a continuation of the Avatar Diary before it and the predecessor to Dan’s Pages on avangel.com after it, but a pedant might argue that because the content wasn’t born as a blog post, perhaps it’s invalid.

Pedants might also bring up the issue of contemporaneity. In 2004 a server fault resulted in the loss of a significant number of 149 blog posts, of which only 85 have been fully-recovered. Some were resurrected from backups as late as 2012, and some didn’t recover their lost images until later still – this one had some content recovered as late as 2017! If you consider the absence of a pre-2004 post until 2012 a sequence-breaker, that’s an issue. It’s theoretically possible, of course, that other old posts might be recovered and injected, and this post might before the 5,001st, 5,002nd, or later post, in terms of chronological post-date. Who knows!

Then there’s the posts injected retroactively. I’ve written software that, since 2018, has ensured that my geocaching logs get syndicated via my blog when I publish them to one of the other logging sites I use, and I retroactively imported all of my previous logs. These never appeared on my blog when they were written: should they count? What about more egregious examples of necroposting, like this post dated long before I ever touched a keyboard? I’m counting them all.

I’m also counting other kinds of less-public content too. Did you know that I sometimes make posts that don’t appear on my front page, and you have to subscribe e.g. by RSS to get them? They have web addresses – although search engines are discouraged from indexing them – and people find them with or without subscribing. Maybe you should subscribe if you haven’t already?

Note that I’m not counting my comments on my own blog, even though many of them are very long, like this 2,700-word exploration of a jigsaw puzzle geocache, or this 1,000-word analogy for cookie theft via cross-site scripting. I’d like to think that for any post that you’d prefer to rule out, given the issues already described, you’d find a comment that could justifiably have been a post in its own right.[/footnote]

Back to celebration mode

Generating a chart...
If this message doesn't go away, the JavaScript that makes this magic work probably isn't doing its job right: please tell Dan so he can fix it.

I’ve only recently started actively keeping stats on my blogging activity, without which I probably wouldn’t even have noticed that my “5K” milestone was coming up!

Let’s take a look at some of those previous milestone posts:

In post 1,000 I announced that I was ready for 2005’s NaNoWriMo. I had a big ol’ argument in the comments with Statto about the value of the exercise. It’s possible that I ultimately wrote more words arguing with him than I did on my writing project that month.
Post 2,000, in 2012, saw me attend the coroner’s inquest into my father’s death. Kate, one of his partners, had appeared as a “surprise witness”, seemingly mostly because she wanted one last “I told you so” about the condition of my dad’s walking boots.
I reposted a link to a Perry Bible Fellowship comic for post 3,000, in 2017. For webcomics that update irregularly and infrequently, like PBF and Bird & Moon,³ I’m glad that I’m an avid feed user so I get to hear about these things as-they-happen, but I appreciated that others might not, hence the repost.
I was in and around San Francisco in August 2019, and post 4,000 was me finding a geocache. The same geocaching expedition saw me discover (and vlog about) a favourite geocache.

It takes a pretty special geocache for me to make a video about it (unlike my geohashing expeditions, for which videos aren’t uncommon). The only other one I can think of was one of my own…

I absolutely count this as the 5,000th post on this blog.

Dan with a champagne flute, fireworks in the background, points to a screen showing this blog post. — Here’s to the next 5,000!

Footnotes

¹ Don’t go look at them. Just don’t. I was a teenager.

² Via a bit of POSSE and a bit of PESOS I do a lot of crossposting (the diagram in that post is a little out-of-date now, though).

³ Bird & Moon, of course, doesn’t have a subscription feed that I’m aware of, but FreshRSS‘s “killer feature” of XPath scraping makes the same kind of thing possible.

Let Your Players Lead The Way

I’ve been GMing/DMing/facilitating¹ roleplaying games for nearby 30 years, but I only recently began to feel like I was getting to be good at it.

The secret skill that was hardest for me to learn? A willingness to surrender control to the players.

I’m a big fan of the Karma/Drama/Fortune (K/D/F) model for understanding resolution. My relationship with K/D/F is a story for another blog post, but I’ll use it as as a framework here.

Karma, Drama, Fortune

I could write a lot about the way I interpret the K/D/F model, but for today here’s a quick primer:

The K/D/F model describes the relationship between three forces: Karma (player choices), Drama (story needs) and Fortune (luck, e.g. dice rolls). For example,

When the lich king comes to the region to provide a villainous plot hook, that’s Drama. Nobody had to do anything and no dice were rolled. The story demanded a “big bad” and so – within the limitations of the setting – one turned up.
When his lucky critical hit kills an ally of the adventurers, that’s Fortune. That battle could have gone a different way, but the dice were on the villain’s side and he was able to harm the players. When we don’t know which way something will go, and it matters, we hit the dice.
When one of the heroes comes up with a clever way to use a magical artefact from a previous quest to defeat him, that’s Karma. It was a clever plan, and the players were rewarded for their smart choices by being able to vanquish the evil thing.
And elsewhere on their quest they probably saw many other resolutions. Each of those may have leaned more-heavily on one or another of the three pillars, or balanced between them equally.

The balance point varies by group and can change over time, but crucially it doesn’t neglect any one of the three aspects.

Disbalancing drama

For most of my many years of gamemastering, I saw my role as being the sole provider the “drama” part of the K/D/F model. The story comes from me, the choices and dice rolls come from the players, right?

Nope, I was wrong. That approach creates an inevitable trend, whether large or small, towards railroading: “forcing” players down a particular path.

A gamemaster with an inflexible and excessively concrete idea of the direction that a story must go will find that they become unable to see the narrative through any other lens. In extreme examples, the players are deprotagonised and the adventure just becomes a series of set pieces, connected by the gamemaster’s idea of how things should play out. I’ve seen this happen. I’ve even caused it to happen, sometimes.²

Flowchart for the quest "Expedition to the Lonely Mountain: an adventure for 14 first-level characters, one of them a halfling and the rest of them dwarves." Shows a linear, non-branching flowchat: birthday party; accept quest; captured by trolls; rescued by wizard; meet elves; get weapons; captured by goblins; rescued by wizard; wolves; rescued by eagles; giant spiders; captured by wood elves; escape by river; lake town; climb mountain; meet dragon; trick dragon, learn weakness, steal treasure; dragon attack. — What if Bilbo and his party escaped from the wood elves by land, heading directly East to Erebor instead of via Esgaroth? What if he failed to determine Smaug’s weakness, or chose not to steal from him? What if the dwarves successfully fought off the goblins and didn’t need rescue? The difference between an adventure *story* and an adventure *game* should be that in a game, nobody – not even the author – can be certain ahead of time of the answers to all the questions.

A catalogue of failures

I’ve railroaded players to some degree or another on an embarrassing number of occasions.

In the spirit of learning from my mistakes, here are three examples of me being a Bad GM.

Quantum Ogre

Scenario: In a short-lived high fantasy GURPS campaign, I wanted the party to meet a band of gypsies and have their fortune told, in order to foreshadow other parts of the story yet to come.

What I did: I pulled a quantum ogre (magician’s choice) on them: whether they travelled by road, or water, or hacked their way through the forest, they were always going to meet the gypsies: their choice of route didn’t really matter.

Why that was wrong: I’d elevated the value of the encounter I’d planned higher than the importance of player agency. The more effort it took to write something, the more I felt the need to ensure it happened!

Two things I could’ve done: Reassessed the importance of the encounter. Found other ways to foreshadow the plot that didn’t undermine player choices, and been more-flexible about my set pieces.

Fudging

Scenario: In a Spirit of the Century one-shot an antagonist needed to kidnap a NPC from aboard an oceanbound ship. To my surprise – with some very lucky rolls – the players foiled the plot!

What I did: I used a fudge – an exploit based on the fact that in most games the gamemaster controls both the plot and the hidden variables of the game mechanics – to facilitate the antagonist kidnapping a different NPC, and adapted the story to this new reality.

Why that was wrong: It made the players feel like their choices didn’t matter. I justified it to myself by it being a one-shot, but that undermines the lesson: I could’ve done better.

Two things I could’ve done: Used the failed attack as a precursor to a later renewed offensive by a villain who’s now got a personal interest in seeing the party fail. Moved towards a different story, perhaps to a different element of the antagonist’s plan.

Ex Machina

Scenario: In a long-running Warhammer Fantasy Roleplay (1st edition!) campaign, a series of bad choices and terrible luck left the party trapped and unable to survive the onslaught of a literal army of bloodthirsty orcs.

What I did: I whipped out a ~~deus~~ spiritus ex machina, having a friendly ghost NPC basically solve for them a useful puzzle they’d been struggling with, allowing them to escape alive (albeit with the quest truly failed).

Why that was wrong: It deprotagonised the adventurers, making them unimportant in their own stories. At the time, I felt that by saving the party I was “saving” the game, but instead I was undermining its value.

Two things I could’ve done: TPK: sometimes it’s the right thing to allow everybody to die! Pivot the plot to facilitate their capture (e.g. the arch-nemesis can’t solve the puzzle either and wants to coerce them into helping), leading to new challenges and interesting moral choices.

Those examples are perhaps extreme, but I’m pretty sure I’ve set up my fair share of lesser sins too. Like chokepoints that strongly encourage a particular direction: do that enough and you train your players to wait until they identify the chokepoint before they take action! Or being less invested in players’ plans if those plans deviate from what I anticipated, and having a convenient in-party NPC prompting players with what they ought to do next. Ugh.³

The good news is, of course, that we’ve all always got the opportunity for growth and self-improvement.

Scan from the D&D 5e adventure book "Descent into Avernus", showing a linear progression from the Dungeon of the Dead Three to The Low Lantern to Vanthampur Villa. In each case, player motivation is supposed, e.g. "The characters confront another of Duke Vanthampur's sons", "the adventurers attack her villa", etc. — In my defence, many *professionally published* adventures are a series of scenes connected by the assumption that the author knows exactly how the players will proceed from each. These don’t teach gamemasters how to handle any deviation: no wonder we don’t learn not to railroad!

The self-improvement path

I’ve gotten better at this in general over the years, but when I took over from Simon at DMing for The Levellers in July, I decided that I was going to try to push myself harder than ever to avoid railroading. Simon was always especially good at promoting player freedom and autonomy, and I wanted to use this inspiration as a vehicle to improve my own gamemastering.

What does that look like within the framework of an established campaign?

Photo with narrow depth of field, showing a tabletop roleplaying game. A grey plastic minature of a dwarf with a battle axe faces off against a large red minature of a dragon. Beside them, two glassy red six-sided dice and a similarly-coloured twenty-sided die can be seen. — Some days, a critical hit is just enough. Other days, you should’ve just stayed at the inn and got drunk. This third-party photo is copyrighted with all rights reserved; not under my usual license.

Well: I ensure there are clues (usually three of them!) to point the players in the “right” direction. And I’ll be on hand to give “nudges” if they’re truly stuck for what to do next, typically by providing a “recap” of the things they’ve previously identified as hooks that are worth following-up (including both the primary plotline and any other avenues they’ve openly discussed investigating).

But that’s the limit to how I allow Drama to control the direction of the story. Almost everything else lies in the hands of Karma and Fortune.

Three-frame captioned screengrab from Community Season 5 Episode 10 (Advanced Advanced Dungeons & Dragons). Abed is sat amongst his dungeon mastering supplies. "South? You could go South," he says. He pulled out an thick binder and slams it onto the table. "I've generated some details about the surrounding area," he adds, flipping it open to show it full of notes. Beneath, it's captioned "eye contact intensifies". — Let all gamemasters strive to be as prepared as Abed.

Needless to say, opening up the possibility space for my players makes gamemastering harder⁴! But… not by as much as I expected. Extra prep-work was necessary, especially at the outset, in order to make sure that the world I was inheriting/building upon was believable and internally-consistent (while ensuring that if a player decided to “just keep walking East” they wouldn’t fall off the edge of the world). But mostly, the work did itself.

Because here’s the thing I learned: so long as you’re willing to take what your players come up with and run with it, they’ll help make the story more compelling. Possibly without even realising it.⁵

"Roll Safe" image meme showing a black man with his finger to his temple, indicating he's had a bright idea. The image is captioned "Players can't derail your plot... if your plot isn't on rails". — This tip brought to you from the Department of Splitting The Party Before They Think To Do It Themselves.

The Levellers are a pretty special group. No matter what the situation, they can always be relied upon to come up with a plan that wasn’t anywhere on their DM‘s radar. When they needed to cross a chasm over their choice of one of two bridges, each guarded by a different variety of enemy, I anticipated a few of the obvious options on each (fighting, magic, persuasion and intimidation, bribery…) but a moment later they were talking about having their druid wildshape into something easy-to-carry while everybody else did a group-spider climb expedition down the chasm edge and along the underside of a bridge. That’s thinking outside the box!

But the real magic has come when the party, through their explorations, have unlocked entirely new elements of the story.

Player-driven content

In our campaign, virtually all of the inhabitants of a city have inadvertently sold their immortal souls to a Archduchess of Hell by allowing, over generations, their declaration of loyalty to their city to become twisted away from their gods and towards their mortal leader, who sold them on in exchange for a sweet afterlife deal. The knights of the city were especially-impacted, as the oath they swore upon promised their unending loyalty in this life as well. When the fiendish pact was made, these knights were immediately possessed by evil forces, transforming into horrendous creatures (who served to harass the party for some time).

Pile of old papers with handwritten notes, including ingredients for a high fantasy "dream machine". — Every party’s got a character who pores over the textual detail of every prop⁶, right? Or is it just every party that includes JTA as a player?

But there’s a hole in this plot⁷. As-written, at least one knight avoided fiendish possession and lived to tell the tale! The player characters noticed this and latched on, so I ran with it. Why might the survivor knights be different from those who became part of the armies of darkness? Was there something different about their swearing-in ceremony? Maybe the reasons are different for different survivors?

I didn’t have answers to these questions to begin with, but the players were moving towards investigating, so I provided some. This also opened up an entire new possible “soft” quest hook related to the reason for the discrepancy. So just like that, a plothole is discovered and investigated by a player, and that results in further opportunities for adventure.

As it happens, the party didn’t even go down that route at all and instead pushed-on in their existing primary direction, but the option remains. All thanks to player curiosity, there’s a possible small quest that’s never been written down or published, and is unique to our group and the party’s interests. And that’s awesome.

In Conclusion

I’m not the best GM in the world. I’m not even the best GM I know. But I’m getting better all the time; learning lessons like how to release the reins a little bit and see where my players can take our adventures.

And for those lessons, I’m grateful to those same players.

Footnotes

¹ I’m using the terms GM, DM, and facilitator interchangeably, and damned if I’m writing them all out every single time.

² A gamemaster giving all of the narrative power to any one of the three elements of K/D/F breaks the game, but in different ways. 100% karma and what you’ve got is a storytelling game, not a roleplaying game: which is fine if that’s what everybody at the table thinks they’re playing: otherwise not. 100% drama gives you a recital, not a jam session: the gamemaster might as well just be writing a book. 100% fortune leads to unrealistic chaos: with no rules to the world (either from the plot or from the consequences of actions) you’re just imagining all possible outcomes in your universe and picking one at random. There’s a balance, and where it sits might vary from group to group, but 100% commitment to a single element almost always breaks things.

³ A the “lesser sins” I mention show, the edges of what construes railroading and what’s merely “a linear quest” is a grey area, and where the line should be drawn varies from group to group. When I’m running a roleplaying session for my primary-school-aged kids, for example, I’m much more-tolerant of giving heavy-handed nudges at a high-level to help them stay focussed on what their next major objective was… but I try harder than ever to encourage diverse and flexible problem-solving ideas within individual scenes, where childish imagination can really make for memorable moments. One time, a tabaxi warrior, on fire, was falling down the outside of a tower… but his player insisted that he could shout a warning through the windows he passed before landing in flawless catlike fashion (albeit mildly singed). My adult players would be rolling athletics checks to avoid injury, but my kids? They can get away with adding details like that by fiat. Different audience, see?

⁴ A recent session took place after a hiatus, and I wasn’t confident that – with the benefit of a few months’ thinking-time – the party would continue with the plan they were executing before the break. And they didn’t! I’d tried to prep for a few other eventualities in the anticipation of what they might do and… I guessed wrong. So, for the first time in recorded history, our session ended early. Is that the end of the world? Nope.

⁵ Want a really radical approach to player-driven plot development? Take a look at this video by Zee Bashew, which I’m totally borrowing from next time I start running a new campaign.

⁶ You know what I miss? Feelies. That’s probably why I try to provide so many “props”, whether physical or digital, in my adventures.

⁷ The plothole isn’t even my fault, for once: it’s functionally broken as-delivered in the source book, although that matters little because we’ve gone so-far outside the original source material now we’re on a whole different adventure, possibly to reconvene later on.