BBC News… without the crap

Did I mention recently that I love RSS? That it brings me great joy? That I start and finish almost every day in my feed reader? Probably.

I used to have a single minor niggle with the BBC News RSS feed: that it included sports news, which I didn’t care about. So I wrote a script that downloaded it, stripped sports news, and re-exported the feed for me to subscribe to. Magic.

RSS reader showing duplicate copies of the news story "Barbie 2? 'We'd love to,' says Warner Bros boss", and an entry from BBC Sounds.
Lately my BBC News feed has caused me some annoyance and frustration.

But lately – presumably as a result of technical changes at the Beeb’s side – this feed has found two fresh ways to annoy me:

  1. The feed now re-publishes a story if it gets re-promoted to the front pagebut with a different <guid> (it appears to get a #0 after it when first published, a #1 the second time, and so on). In a typical day the feed reader might scoop up new stories about once an hour, any by the time I get to reading them the same exact story might appear in my reader multiple times. Ugh.
  2. They’ve started adding iPlayer and BBC Sounds content to the BBC News feed. I don’t follow BBC News in my feed reader because I want to watch or listen to things. If you do, that’s fine, but I don’t, and I’d rather filter this content out.

Luckily, I already have a recipe for improving this feed, thanks to my prior work. Let’s look at my newly-revised script (also available on GitHub):

#!/usr/bin/env ruby
require 'bundler/inline'

# # Sample crontab:
# # At 41 minutes past each hour, run the script and log the results
# */20 * * * * ~/bbc-news-rss-filter-sport-out.rb > ~/bbc-news-rss-filter-sport-out.log 2>>&1

# Dependencies:
# * open-uri - load remote URL content easily
# * nokogiri - parse/filter XML
gemfile do
  source 'https://rubygems.org'
  gem 'nokogiri'
end
require 'open-uri'

# Regular expression describing the GUIDs to reject from the resulting RSS feed
# We want to drop everything from the "sport" section of the website, also any iPlayer/Sounds links
REJECT_GUIDS_MATCHING = /^https:\/\/www\.bbc\.co\.uk\/(sport|iplayer|sounds)\//

# Load and filter the original RSS
rss = Nokogiri::XML(open('https://feeds.bbci.co.uk/news/rss.xml?edition=uk'))
rss.css('item').select{|item| item.css('guid').text =~ REJECT_GUIDS_MATCHING }.each(&:unlink)

# Strip the anchors off the <guid>s: BBC News "republishes" stories by using guids with #0, #1, #2 etc, which results in duplicates in feed readers
rss.css('guid').each{|g|g.content=g.content.gsub(/#.*$/,'')}

File.open( '/www/bbc-news-no-sport.xml', 'w' ){ |f| f.puts(rss.to_s) }
It’s amazing what you can do with Nokogiri and a half dozen lines of Ruby.

That revised script removes from the feed anything whose <guid> suggests it’s sports news or from BBC Sounds or iPlayer, and also strips any “anchor” part of the <guid> before re-exporting the feed. Much better. (Strictly speaking, this can result in a technically-invalid feed by introducing duplicates, but your feed reader oughta be smart enough to compensate for and ignore that: mine certainly is!)

You’re free to take and adapt the script to your own needs, or – if you don’t mind being tied to my opinions about what should be in BBC News’ RSS feed – just subscribe to my copy at: https://fox.q-t-a.uk/bbc-news-no-sport.xml

RSS reader showing duplicate copies of the news story "Barbie 2? 'We'd love to,' says Warner Bros boss", and an entry from BBC Sounds.×

Open Turds

I’ve open-sourced a lot of pretty shit code.

So whenever somebody says “I’m not open-sourcing this because the code is shit”, I think: wow, it must be spectacularly bad.

And that only makes me want to see it more.

[Bloganuary] Fun Five

This post is part of my attempt at Bloganuary 2024. Today’s prompt is:

List five things you do for fun.

This feels disappointingly like the prompt from day 2, but I’m gonna pivot it by letting my answer from three weeks ago only cover one of the five points:

  1. Code
  2. Magic
  3. Piano
  4. Play
  5. Learn

Let’s take a look at each of those, briefly.

Code

Code is poetry. Code is fun. Code is a many-splendoured thing.

This is not what real coding looks like. This is what real coding looks like.

When I’m not coding for work or coding as a volunteer, I’m often caught coding for fun. Sometimes I write WordPress-ey things. Sometimes I write other random things. I tend to open-source almost everything I write, most of it via my GitHub account.

Magic

Now I don’t work in the city centre nor have easy access to other magicians, I don’t perform as much magic as I used to. But I still try to keep my hand in and occasionally try new things; I enjoy practicing sleights when I’m doing work-related things that don’t require my hands (meetings, code reviews, waiting for the damn unit tests to run…), a tip I learned from fellow magician Andy.

My favourite go-to trick with an untampered deck of cards is my variant of the Ambitious Classic; here’s a bit from the middle of the trick from the last time I performed it in a video meeting.

You’ll usually find a few decks of cards on my desk at any given time, mostly Bikes.1

Piano

I started teaching myself piano during the Covid lockdowns as a distraction from not being able to go anywhere (apparently I’m not the only one), and as an effort to do more of what I’m bad at.2 Since then, I’ve folded about ten minutes of piano-playing3, give or take, into my routine virtually every day.

This is what piano playing looks like. But perhaps only barely.

I fully expect that I’ll never be as accomplished at it as, say, the average 8-year-old on YouTube, but that’s not what it’s about. If I take a break from programming, or meetings, or childcare, or anything, I can feel that playing music exercises a totally different part of my mind. I’d heard musicians talk about such an experience before, but I’d assumed that it was hyperbole… but from my perspective, they’re right: practicing an instrument genuinely does feel like using a part of your brain than you use for anything else, which I love!

Play

I wrote a whole other Bloganuary post on the ways in which I integrate “play” into my life, so I’ll point you at that rather than rehash anything.

A lot of my RPG-gaming takes place online, via virtual tabletops, and is perhaps the most obvious “playtime” play activities I routinely engage in.

At the weekend I dusted off Vox Populi, my favourite mod for Civilization V, my favourite4 entry in the Civilization series, which in turn is one of my favourite video game series5. I don’t get as much time for videogaming as I might like, but that’s probably for the best because a couple of hours disappeared on Sunday evening before I even blinked! It’s addictive stuff.

Learn

As I mentioned back on day 3 of bloganuary, I’m a lifelong learner. But even when I’m not learning in an academic setting, I’m doubtless learning something. I tend to alternate between fiction and non-fiction books on my bedside table. I often get lost on deep-dives through the depths of the Web after a Wikipedia article makes me ask “wait, really?” And just sometimes, I set out to learn some kind of new skill.

It’s not always wacky and off-the-wall things like basic blacksmithing that I learn. Sometimes it’s normal, practical activities like baking bread or… umm… Argentine tango?

In short: with such a variety of fun things lined-up, I rarely get the opportunity to be bored6!

Footnotes

1 I like the feel of Bicycle cards and the way they fan. Plus: the white border – which is actually a security measure on playing cards designed to make bottom-dealing more-obvious and thus make it harder for people to cheat at e.g. poker – can actually be turned to work for the magician when doing certain sleights, including one seen in the mini-video above…

2 I’m not strictly bad at it, it’s just that I had essential no music tuition or instrument experience whatsoever – I didn’t even have a recorder at primary school! – and so I was starting at square zero.

3 Occasionally I’ll learn a bit of a piece of music, but mostly I’m trying to improve my ability to improvise because that scratches an itch in a part of my brain in a way that I find most-interesting!

4 Games in the series I’ve extensively played include: Civilization, CivNet, Civilization II (also Test of Time), Alpha Centauri (a game so good I paid for it three times, despite having previously pirated it), Civilization III, Civilization IV, Civilization V, Beyond Earth (such a disappointment compared to SMAC) and Civilization VI, plus all their expansions except for the very latest one for VI. Also spinoffs/clones FreeCiv, C-Evo, and both Call to Power games. Oh, and at least two of the board games. And that’s just the ones I’ve played enough to talk in detail about: I’m not including things like Revolution which I played an hour of and hated so much I shan’t touch it again, nor either version of Colonization which I’m treating separately…

5 Way back in 2007 I identified Civilization as the top of the top 10 videogames that stole my life, and frankly that’s still true.

6 At least, not since the kids grew out of Paw Patrol so I don’t have to sit with them and watch it any more!

CapsulePress – Gemini / Spartan / Gopher to WordPress bridge

For a while now, this site has been partially mirrored via the Gemini1 and Gopher protocols.2 Earlier this year I presented hacky versions of the tools I’d used to acieve this (and made people feel nostalgic).

Now I’ve added support for Spartan3 too and, seeing as the implementations shared functionality, I’ve combined all three – Gemini, Spartan, and Gopher – into a single package: CapsulePress.

Diagram illustrating the behaviour of CapsulePress: a WordPress installation provides content, and CapsulePress makes that content available via gemini://, spartan://, and gopher:// URLs.

CapsulePress is a Gemini/Spartan/Gopher to WordPress bridge. It lets you use WordPress as a CMS for any or all of those three non-Web protocols in addition to the Web.

For example, that means that this post is available on all of:

Composite screenshot showing this blog post in, from top-left to bottom-right: (1) Firefox, via HTTPS, (2) Lagrange, via Gemini, (3) Lagrange, via Spartan, and (4) Lynx, via Gopher.

It’s also possible to write posts that selectively appear via different media: if I want to put something exclusively on my gemlog, I can, by assigning metadata that tells WordPress to suppress a post but still expose it to CapsulePress. Neat!

90s-style web banners in the style of Netscape ads, saying "Gemini now!", "Spartan now!", and "Gopher now!". Beneath then a wider banner ad promotes CapsulePress v0.1.
Using Gemini and friends in the 2020s make me feel like the dream of the Internet of the nineties and early-naughties is still alive. But with fewer banner ads.

I’ve open-sourced the whole thing under a super-permissive license, so if you want your own WordPress blog to “feed” your Gemlog… now you can. With a few caveats:

  • It’s hard to use. While not as hacky as the disparate piles of code it replaced, it’s still not the cleanest. To modify it you’ll need a basic comprehension of all three protocols, plus Ruby, SQL, and sysadmin skills.
  • It’s super opinionated. It’s very much geared towards my use case. It’s improved by the use of templates. but it’s still probably only suitable for this site for the time being, until you make changes.
  • It’s very-much unfinished. I’ve got a growing to-do list, which should be a good clue that it’s Not Finished. Maybe it never will but. But there’ll be changes yet to come.

Whether or not your WordPress blog makes the jump to Geminispace4, I hope you’ll came take a look at mine at one of the URLs linked above, and then continue to explore.

If you’re nostalgic for the interpersonal Internet – or just the idea of it, if you’re too young to remember it… you’ll find it there. (That Internet never actually went away, but it’s harder to find on today’s big Web than it is on lighter protocols.)

Footnotes

1 Also available via Gemini.

2 Also via Finger (but that’s another story).

3 Also available via Spartan.

4 Need a browser? I suggest Lagrange.

90s-style web banners in the style of Netscape ads, saying "Gemini now!", "Spartan now!", and "Gopher now!". Beneath then a wider banner ad promotes CapsulePress v0.1.× Diagram illustrating the behaviour of CapsulePress: a WordPress installation provides content, and CapsulePress makes that content available via gemini://, spartan://, and gopher:// URLs.× Composite screenshot showing this blog post in, from top-left to bottom-right: (1) Firefox, via HTTPS, (2) Lagrange, via Gemini, (3) Lagrange, via Spartan, and (4) Lynx, via Gopher.×

Werewolves and Wanderer

This blog post is also available as a video. Would you prefer to watch/listen to me tell you about the video game that had the biggest impact on my life?

Of all of the videogames I’ve ever played, perhaps the one that’s had the biggest impact on my life1 was: Werewolves and (the) Wanderer.2

This simple text-based adventure was originally written by Tim Hartnell for use in his 1983 book Creating Adventure Games on your Computer. At the time, it was common for computing books and magazines to come with printed copies of program source code which you’d need to re-type on your own computer, printing being significantly many orders of magnitude cheaper than computer media.3

Front cover of The Amazing Amstrad Omnibus, by Martin Fairbanks, with its bright yellow text on a red background.
Werewolves and Wanderer was adapted for the Amstrad CPC4 by Martin Fairbanks and published in The Amazing Amstrad Omnibus (1985), which is where I first discovered it.
When I first came across the source code to Werewolves, I’d already begun my journey into computer programming. This started alongside my mother and later – when her quantity of free time was not able to keep up with my level of enthusiasm – by myself.

I’d been working my way through the operating manual for our microcomputer, trying to understand it all.5

Scan of a ring-bound page from a technical manual. The page describes the use of the "INPUT" command, saying "This command is used to let the computer know that it is expecting something to be typed in, for example, the answer to a question". The page goes on to provide a code example of a program which requests the user's age and then says "you look younger than [age] years old.", substituting in their age. The page then explains how it was the use of a variable that allowed this transaction to occur.
The ring-bound 445-page A4 doorstep of a book quickly became adorned with my pencilled-in notes, the way a microcomputer manual ought to be. It’s strange to recall that there was a time that beginner programmers still needed to be reminded to press [ENTER] at the end of each line.
And even though I’d typed-in dozens of programs before, both larger and smaller, it was Werewolves that finally helped so many key concepts “click” for me.

In particular, I found myself comparing Werewolves to my first attempt at a text-based adventure. Using what little I’d grokked of programming so far, I’d put together a series of passages (blocks of PRINT statements6) with choices (INPUT statements) that sent the player elsewhere in the story (using, of course, the long-considered-harmful GOTO statement), Choose-Your-Own-Adventure style.

Werewolves was… better.

Photograph of Dan in his mid-teens, with shoulder-length bleached-blonde hair and wearing a t-shirt with a picture of a snarling wolf, sits in front of a running PC (with its beige case open) on which an external modem is precariously balanced.
By the time I was the model of a teenage hacker, I’d been writing software for years. Most of it terrible.

Werewolves and Wanderer was my first lesson in how to structure a program.

Let’s take a look at a couple of segments of code that help illustrate what I mean (here’s the full code, if you’re interested):

10 REM WEREWOLVES AND WANDERER

20 GOSUB 2600:REM INTIALISE
30 GOSUB 160
40 IF RO<>11 THEN 30

50 PEN 1:SOUND 5,100:PRINT:PRINT "YOU'VE DONE IT!!!":GOSUB 3520:SOUND 5,80:PRINT "THAT WAS THE EXIT FROM THE CASTLE!":SOUND 5,200
60 GOSUB 3520
70 PRINT:PRINT "YOU HAVE SUCCEEDED, ";N$;"!":SOUND 5,100
80 PRINT:PRINT "YOU MANAGED TO GET OUT OF THE CASTLE"
90 GOSUB 3520
100 PRINT:PRINT "WELL DONE!"
110 GOSUB 3520:SOUND 5,80
120 PRINT:PRINT "YOUR SCORE IS";
130 PRINT 3*TALLY+5*STRENGTH+2*WEALTH+FOOD+30*MK:FOR J=1 TO 10:SOUND 5,RND*100+10:NEXT J
140 PRINT:PRINT:PRINT:END

...

2600 REM INTIALISE
2610 MODE 1:BORDER 1:INK 0,1:INK 1,24:INK 2,26:INK 3,18:PAPER 0:PEN 2 
2620 RANDOMIZE TIME
2630 WEALTH=75:FOOD=0
2640 STRENGTH=100
2650 TALLY=0
2660 MK=0:REM NO. OF MONSTERS KILLED

...

3510 REM DELAY LOOP
3520 FOR T=1 TO 900:NEXT T
3530 RETURN
Locomotive BASIC had mandatory line numbering. The spacing and gaps (...) have been added for readability/your convenience.

What’s interesting about the code above? Well…

  • The code for “what to do when you win the game” is very near the top. “Winning” is the default state. The rest of the adventure exists to obstruct that. In a language with enforced line numbering and no screen editor7, it makes sense to put fixed-length code at the top… saving space for the adventure to grow below.
  • Two subroutines are called (the GOSUB statements):
    • The first sets up the game state: initialising the screen (2610), the RNG (2620), and player characteristics (26302660). This also makes it easy to call it again (e.g. if the player is given the option to “start over”). This subroutine goes on to set up the adventure map (more on that later).
    • The second starts on line 160: this is the “main game” logic. After it runs, each time, line 40 checks IF RO<>11 THEN 30. This tests whether the player’s location (RO) is room 11: if so, they’ve exited the castle and won the adventure. Otherwise, flow returns to line 30 and the “main game” subroutine happens again. This broken-out loop improving the readability and maintainability of the code.8
  • A common subroutine is the “delay loop” (line 3520). It just counts to 900! On a known (slow) processor of fixed speed, this is a simpler way to put a delay in than relying on a real-time clock.

The game setup gets more interesting still when it comes to setting up the adventure map. Here’s how it looks:

2680 REM SET UP CASTLE
2690 DIM A(19,7):CHECKSUM=0
2700 FOR B=1 TO 19
2710   FOR C=1 TO 7
2720     READ A(B,C):CHECKSUM=CHECKSUM+A(B,C)
2730   NEXT C:NEXT B
2740 IF CHECKSUM<>355 THEN PRINT "ERROR IN ROOM DATA":END

...

2840 REM ALLOT TREASURE
2850 FOR J=1 TO 7
2860   M=INT(RND*19)+1
2870   IF M=6 OR M=11 OR A(M,7)<>0 THEN 2860
2880   A(M,7)=INT(RND*100)+100
2890 NEXT J

2910 REM ALLOT MONSTERS
2920 FOR J=1 TO 6
2930   M=INT(RND*18)+1
2940   IF M=6 OR M=11 OR A(M,7)<>0 THEN 2930
2950   A(M,7)=-J
2960 NEXT J
2970 A(4,7)=100+INT(RND*100)
2980 A(16,7)=100+INT(RND*100)

...

3310 DATA   0,  2,  0,  0,  0,  0,  0
3320 DATA   1,  3,  3,  0,  0,  0,  0
3330 DATA   2,  0,  5,  2,  0,  0,  0
3340 DATA   0,  5,  0,  0,  0,  0,  0
3350 DATA   4,  0,  0,  3, 15, 13,  0
3360 DATA   0,  0,  1,  0,  0,  0,  0
3370 DATA   0,  8,  0,  0,  0,  0,  0
3380 DATA   7, 10,  0,  0,  0,  0,  0
3390 DATA   0, 19,  0,  8,  0,  8,  0
3400 DATA   8,  0, 11,  0,  0,  0,  0
3410 DATA   0,  0, 10,  0,  0,  0,  0
3420 DATA   0,  0,  0, 13,  0,  0,  0
3430 DATA   0,  0, 12,  0,  5,  0,  0
3440 DATA   0, 15, 17,  0,  0,  0,  0
3450 DATA  14,  0,  0,  0,  0,  5,  0
3460 DATA  17,  0, 19,  0,  0,  0,  0
3470 DATA  18, 16,  0, 14,  0,  0,  0
3480 DATA   0, 17,  0,  0,  0,  0,  0
3490 DATA   9,  0, 16,  0,  0,  0,  0
Again, I’ve tweaked this code to improve readability, including adding indention on the loops, “modern-style”, and spacing to make the DATA statements form a “table”.

What’s this code doing?

  • Line 2690 defines an array (DIM) with two dimensions9 (19 by 7). This will store room data, an approach that allows code to be shared between all rooms: much cleaner than my first attempt at an adventure with each room having its own INPUT handler.
  • The two-level loop on lines 2700 through 2730 populates the room data from the DATA blocks. Nowadays you’d probably put that data in a separate file (probably JSON!). Each “row” represents a room, 1 to 19. Each “column” represents the room you end up at if you travel in a given direction: North, South, East, West, Up, or Down. The seventh column – always zero – represents whether a monster (negative number) or treasure (positive number) is found in that room. This column perhaps needn’t have been included: I imagine it’s a holdover from some previous version in which the locations of some or all of the treasures or monsters were hard-coded.
  • The loop beginning on line 2850 selects seven rooms and adds a random amount of treasure to each. The loop beginning on line 2920 places each of six monsters (numbered -1 through -6) in randomly-selected rooms. In both cases, the start and finish rooms, and any room with a treasure or monster, is ineligible. When my 8-year-old self finally deciphered what was going on I was awestruck at this simple approach to making the game dynamic.
  • Rooms 4 and 16 always receive treasure (lines 29702980), replacing any treasure or monster already there: the Private Meeting Room (always worth a diversion!) and the Treasury, respectively.
  • Curiously, room 9 (the lift) defines three exits, even though it’s impossible to take an action in this location: the player teleports to room 10 on arrival! Again, I assume this is vestigal code from an earlier implementation.
  • The “checksum” that’s tested on line 2740 is cute, and a younger me appreciated deciphering it. I’m not convinced it’s necessary (it sums all of the values in the DATA statements and expects 355 to limit tampering) though, or even useful: it certainly makes it harder to modify the rooms, which may undermine the code’s value as a teaching aid!
Map showing the layout of the castle in video game "Werewolves and the Wanderer". Entering from outside the castle, to the West, the player must progress through the ground floor, up the stairwell in the Inner Hallway, into the Lift, and then East to the exit, but there are several opportunities to diverge from this path and e.g. explore the dungeons or various dead ends on the ground or first floors.
By the time I was 10, I knew this map so well that I could draw it perfectly from memory. I almost managed the same today, aged 42. That memory’s buried deep!

Something you might notice is missing is the room descriptions. Arrays in this language are strictly typed: this array can only contain integers and not strings. But there are other reasons: line length limitations would have required trimming some of the longer descriptions. Also, many rooms have dynamic content, usually based on random numbers, which would be challenging to implement in this way.

As a child, I did once try to refactor the code so that an eighth column of data specified the line number to which control should pass to display the room description. That’s a bit of a no-no from a “mixing data and logic” perspective, but a cool example of metaprogramming before I even knew it! This didn’t work, though: it turns out you can’t pass a variable to a Locomotive BASIC GOTO or GOSUB. Boo!10

An experimental program being run that attempts to GOSUB a variable, failing with a syntax error on the relevant line.
In hindsight, I could have tested the functionality before I refactored with a very simple program, but I was only around 10 or 11 and still had lots to learn!

Werewolves and Wanderer has many faults11. But I’m clearly not the only developer whose early skills were honed and improved by this game, or who hold a special place in their heart for it. Just while writing this post, I discovered:

A decade or so later, I’d be taking my first steps as a professional software engineer. A couple more decades later, I’m still doing it.

And perhaps that adventure -the one that’s occupied my entire adult life – was facilitated by this text-based one from the 1980s.

Footnotes

1 The game that had the biggest impact on my life, it might surprise you to hear, is not among the “top ten videogames that stole my life” that I wrote about almost exactly 16 years ago nor the follow-up list I published in its incomplete form three years later. Turns out that time and impact are not interchangable. Who knew?

2 The game is variously known as Werewolves and Wanderer, Werewolves and Wanderers, or Werewolves and the Wanderer. Or, on any system I’ve been on, WERE.BAS, WEREWOLF.BAS, or WEREWOLV.BAS, thanks to the CPC’s eight-point-three filename limit.

3 Additionally, it was thought that having to undertake the (painstakingly tiresome) process of manually re-entering the source code for a program might help teach you a little about the code and how it worked, although this depended very much on how readable the code and its comments were. Tragically, the more comprehensible some code is, the more long-winded the re-entry process.

4 The CPC’s got a fascinating history in its own right, but you can read that any time.

5 One of my favourite features of home microcomputers was that seconds after you turned them on, you could start programming. Your prompt was an interface to a programming language. That magic had begun to fade by the time DOS came to dominate (sure, you can program using batch files, but they’re neither as elegant nor sophisticated as any BASIC dialect) and was completely lost by the era of booting directly into graphical operating systems. One of my favourite features about the Web is that it gives you some of that magic back again: thanks to the debugger in a modern browser, you can “tinker” with other people’s code once more, right from the same tool you load up every time. (Unfortunately, mobile devices – which have fast become the dominant way for people to use the Internet – have reversed this trend again. Try to View Source on your mobile – if you don’t already know how, it’s not an easy job!)

6 In particular, one frustration I remember from my first text-based adventure was that I’d been unable to work around Locomotive BASIC’s lack of string escape sequences – not that I yet knew what such a thing would be called – in order to put quote marks inside a quoted string!

7 “Screen editors” is what we initially called what you’d nowadays call a “text editor”: an application that lets you see a page of text at the same time, move your cursor about the place, and insert text wherever you feel like. It may also provide features like copy/paste and optional overtyping. Screen editors require more resources (and aren’t suitable for use on a teleprinter) compared to line editors, which preceeded them. Line editors only let you view and edit a single line at a time, which is how most of my first 6 years of programming was done.

8 In a modern programming language, you might use while true or similar for a main game loop, but this requires pushing the “outside” position to the stack… and early BASIC dialects often had strict (and small, by modern standards) limits on stack height that would have made this a risk compared to simply calling a subroutine from one line and then jumping back to that line on the next.

9 A neat feature of Locomotive BASIC over many contemporary and older BASIC dialects was its support for multidimensional arrays. A common feature in modern programming languages, this language feature used to be pretty rare, and programmers had to do bits of division and modulus arithmetic to work around the limitation… which, I can promise you, becomes painful the first time you have to deal with an array of three or more dimensions!

10 In reality, this was rather unnecessary, because the ON x GOSUB command can – and does, in this program – accept multiple jump points and selects the one referenced by the variable x.

11 Aside from those mentioned already, other clear faults include: impenetrable controls unless you’ve been given instuctions (although that was the way at the time); the shopkeeper will penalise you for trying to spend money you don’t have, except on food, presumably as a result of programmer laziness; you can lose your flaming torch, but you can’t buy spares in advance (you can pay for more, and you lose the money, but you don’t get a spare); some of the line spacing is sometimes a little wonky; combat’s a bit of a drag; lack of feedback to acknowledge the command you enterted and that it was successful; WHAT’S WITH ALL THE CAPITALS; some rooms don’t adequately describe their exits; the map is a bit linear; etc.

Front cover of The Amazing Amstrad Omnibus, by Martin Fairbanks, with its bright yellow text on a red background.× Scan of a ring-bound page from a technical manual. The page describes the use of the "INPUT" command, saying "This command is used to let the computer know that it is expecting something to be typed in, for example, the answer to a question". The page goes on to provide a code example of a program which requests the user's age and then says "you look younger than [age] years old.", substituting in their age. The page then explains how it was the use of a variable that allowed this transaction to occur.× Photograph of Dan in his mid-teens, with shoulder-length bleached-blonde hair and wearing a t-shirt with a picture of a snarling wolf, sits in front of a running PC (with its beige case open) on which an external modem is precariously balanced.× Map showing the layout of the castle in video game "Werewolves and the Wanderer". Entering from outside the castle, to the West, the player must progress through the ground floor, up the stairwell in the Inner Hallway, into the Lift, and then East to the exit, but there are several opportunities to diverge from this path and e.g. explore the dungeons or various dead ends on the ground or first floors.× An experimental program being run that attempts to GOSUB a variable, failing with a syntax error on the relevant line.×

Solving Jigidi… Again

(Just want the instructions? Scroll down.)

A year and a half ago I came up with a technique for intercepting the “shuffle” operation on jigsaw website Jigidi, allowing players to force the pieces to appear in a consecutive “stack” for ludicrously easy solving. I did this partially because I was annoyed that a collection of geocaches near me used Jigidi puzzles as a barrier to their coordinates1… but also because I enjoy hacking my way around artificially-imposed constraints on the Web (see, for example, my efforts last week to circumvent region-blocking on radio.garden).

My solver didn’t work for long: code changes at Jigidi’s end first made it harder, then made it impossible, to use the approach I suggested. That’s fine by me – I’d already got what I wanted – but the comments thread on that post suggests that there’s a lot of people who wish it still worked!2 And so I ignored the pleas of people who wanted me to re-develop a “Jigidi solver”. Until recently, when I once again needed to solve a jigsaw puzzle in order to find a geocache’s coordinates.

Making A Jigidi Helper

Rather than interfere with the code provided by Jigidi, I decided to take a more-abstract approach: swapping out the jigsaw’s image for one that would be easier.

This approach benefits from (a) having multiple mechanisms of application: query interception, DNS hijacking, etc., meaning that if one stops working then another one can be easily rolled-out, and (b) not relying so-heavily on the structure of Jigidi’s code (and therefore not being likely to “break” as a result of future upgrades to Jigidi’s platform).

Watch a video demonstrating the approach:

It’s not as powerful as my previous technique – more a “helper” than a “solver” – but it’s good enough to shave at least half the time off that I’d otherwise spend solving a Jigidi jigsaw, which means I get to spend more time out in the rain looking for lost tupperware. (If only geocaching were even the weirdest of my hobbies…)

How To Use The Jigidi Helper

To do this yourself and simplify your efforts to solve those annoying “all one colour” or otherwise super-frustrating jigsaw puzzles, here’s what you do:

  1. Visit a Jigidi jigsaw. Do not be logged-in to a Jigidi account.
  2. Copy my JavaScript code into your clipboard.
  3. Open your browser’s debug tools (usually F12). In the Console tab, paste it and press enter. You can close your debug tools again (F12) if you like.
  4. Press Jigidi’s “restart” button, next to the timer. The jigsaw will restart, but the picture will be replaced with one that’s easier-to-solve than most, as described below.
  5. Once you solve the jigsaw, the image will revert to normal (turn your screen around and show off your success to a friend!).

What makes it easier to solve?

The replacement image has the following characteristics that make it easier to solve than it might otherwise be:

  • Every piece has written on it the row and column it belongs in.
  • Every “column” is striped in a different colour.
  • Striped “bands” run along entire rows and columns.

To solve the jigsaw, start by grouping colours together, then start combining those that belong in the same column (based on the second digit on the piece). Join whole or partial columns together as you go.

I’ve been using this technique or related ones for over six months now and no code changes on Jigidi’s side have impacted upon it at all, so it’s probably got better longevity than the previous approach. I’m not entirely happy with it, and you might not be either, so feel free to fork my code and improve it: the legiblity of the numbers is sometimes suboptimal, and the colour banding repeats on larger jigsaws which I’d rather avoid. There’s probably also potential to improve colour-recognition by making the colour bands span the gaps between rows or columns of pieces, too, but more experiments are needed and, frankly, I’m not the right person for the job. For the second time, I’m going to abandon a tool that streamlines Jigidi solving because I’ve already gotten what I needed out of it, and I’ll leave it up to you if you want to come up with an improvement and share it with the community.

Footnotes

1 As I’ve mentioned before, and still nobody believes me: I’m not a fan of jigsaws! If you enjoy them, that’s great: grab a bucket of popcorn and a jigsaw and go wild… but don’t feel compelled to share either with me.

2 The comments also include asuper-helpful person called Rich who’s been manually solving people’s puzzles for them, and somebody called Perdita who “could be my grandmother” (except: no) with whom I enjoyed a conversation on- and off-line about the ethics of my technique. It’s one of the most-popular comment threads my blog has ever seen.

Spring ’83 Came And Went

Just in time for Robin Sloan to give up on Spring ’83, earlier this month I finally got aroud to launching STS-6 (named for the first mission of the Space Shuttle Challenger in Spring 1983), my experimental Spring ’83 server. It’s been a busy year; I had other things to do. But you might have guessed that something like this had been under my belt when I open-sourced a keygenerator for the protocol the other day.

If you’ve not played with Spring ’83, this post isn’t going to make much sense to you. Sorry.

Introducing STS-6

Screenshots showing STS-6, listing the most-recent blog posts on DanQ.me, in two different display styles.
My output looks distinctly different in The Kingswood Palimpsest then in The Oakland Follower-Sentinel (two key reference Spring ’83 clients), and that’s fine and expected.

My server is, as far as I can tell, very different from any others in a few key ways:

  • It does not allow third-party publishing at all. Some might argue that this undermines the aim of the exercise, but I disagree. My IndieWeb inclinations lead me to favour “self-hosted” content, shared from its owners’ domain. Also: the specification clearly states that a server must implement a denylist… I guess my denylist simply includes all keys that are not specifically permitted.
  • It’s geared towards dynamic content. My primary board self-publishes whenever I produce a new blog post, listing the most recent blog posts published. I have another half-implemented which shows a summary of the most-recent post, and another which would would simply use a WordPress page as its basis – yes, this was content management, but published over Spring ’83.
  • It provides helpers to streamline content production. It supports internal references to other boards you control using the format {{board:123}}which are automatically converted to addresses referencing the public key of the “current” keypair for that board. This separates the concept of a board and its content template from that board’s keypairs, making it easier to link to a board. To put it another way, STS-6 links are self-healing on the server-side (for local boards).
  • It helps automate content-fitting. Spring ’83 strictly requires a maximum board size of 2,217 bytes. STS-6 can be configured to fit a flexible amount of dynamic content within a template area while respecting that limit. For my posts list board, the number of posts shown is moderated by the size of the resulting board: STS-6 adds more and more links to the board until it’s too big, and then removes one!
  • It provides “hands-off” key management features. You can pregenerate a list of keys with different validity periods and the server will automatically cycle through them as necessary, implementing and retroactively-modifying <link rel="next"> connections to keep them current.

I’m sure that there are those who would see this as automating something that was beautiful because it was handcrafted; I don’t know whether or not I agree, but had Spring ’83 taken off in a bigger way, it would always only have been a matter of time before somebody tried my approach.

From a design perspective, I enjoyed optimising an SVG image of my header so it could meaningfully fit into the board. It’s pretty, and it’s tolerably lightweight.

If you want to see my server in action, patch this into your favourite Spring ’83 client: https://s83.danq.dev/10c3ff2e8336307b0ac7673b34737b242b80e8aa63ce4ccba182469ea83e0623

A dead end?

Without Robin’s active participation, I feel that Spring ’83 is probably coming to a dead end. It’s been a lot of fun to play with and I’d love to see what ideas the experience of it goes on to inspire next, but in its current form it’s one of those things that’s an interesting toy, but not something that’ll make serious waves.

In his last lab essay Robin already identified many of the key issues with the system (too complicated, no interpersonal-mentions, the challenge of keys-as-identifiers, etc.) and while they’re all solvable without breaking the underlying mechanisms (mentions might be handled by Webmention, perhaps, etc.), I understand the urge to take what was learned from this experiment and use it to help inform the decisions of the next one. Just as John Postel’s Quote of the Day protocol doesn’t see much use any more (although maybe if my finger server could support QotD?) but went on to inspire the direction of many subsequent “call-and-response” protocols, including HTTP, it’s okay if Spring ’83 disappears into obscurity, so long as we can learn what it did well and build upon that.

Meanwhile: if you’re looking for a hot new “like the web but lighter” protocol, you should probably check out Gemini. (Incidentally, you can find me at gemini://danq.me, but that’s something I’ll write about another day…)

Screenshots showing STS-6, listing the most-recent blog posts on DanQ.me, in two different display styles.×

Debugging Mysteries

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Just wanted to share with you something I found ages ago but only just got around to mentioning – Julia Evans‘ (of Wizard Zines fame) Debugging Mysteries:

There are five mysteries here right now:

The Case of the Slow Websites
The Case of the Connection Timeout
The Case of the DNS Update that Didn’t Work
The Case of the 50ms Request
The Case of the Failed Docker Connection

Go back to wizardzines.com

GitHub repository: jvns/twine-stories
Blog post about this project: Notes on building debugging puzzles

Each mystery is a Twine-powered “choose your own adventure” game in which you must diagnose the kind of issue that a software developer might, for real. I think these are potentially excellent tools for beginner programmers, not just because they provide some information about the topic of each, but because they encourage cultivating a mindset of the kind of thinking that’s required to get to the bottom of gnarly problems.

Better yet, she’s open-sourced the entire thing (and I was excited to see she wrote in Twee, which is very dear to my heart).

Beating Children at Mastermind

This blog post is also available as a video. Would you prefer to watch/listen to me tell you about how I’ve implemented a tool to help me beat the kids when we play Mastermind?

I swear that I used to be good at Mastermind when I was a kid. But now, when it’s my turn to break the code that one of our kids has chosen, I fail more often than I succeed. That’s no good!

Black, white, brown, blue, green, orange and yellow Mastermind pegs in a disordered heap.
If you didn’t have me pegged as a board gamer… where the hell have you been?

Mastermind and me

Maybe it’s because I’m distracted; multitasking doesn’t help problem-solving. Or it’s because we’re “Super” Mastermind, which differs from the one I had as a child in that eight (not six) peg colours are available and secret codes are permitted to have duplicate peg colours. These changes increase the possible permutations from 360 to 4,096, but the number of guesses allowed only goes up from 8 to 10. That’s hard.

A plastic Mastermind board in brown and green; it has twelve spots for guessing and shows six coloured pegs. The game has been won on the sixth guess.
The set I had as a kid was like this, I think. Photo courtesy ZeroOne; CC-BY-SA license.

Or maybe it’s just that I’ve gotten lazy and I’m now more-likely to try to “solve” a puzzle using a computer to try to crack a code using my brain alone. See for example my efforts to determine the hardest hangman words and make an adverserial hangman game, to generate solvable puzzles for my lock puzzle game, to cheat at online jigsaws, or to balance my D&D-themed Wordle clone.

Hey, that’s an idea. Let’s crack the code… by writing some code!

Screenshot showing Mastermind game from WebGamesOnline.com. Seven guesses have been made, each using only one colour for each of the four pegs, and no guesses are corect; only red pegs have never been guessed.
This online edition plays a lot like the version our kids play, although the peg colours are different. Next guess should be an easy solve!

Representing a search space

The search space for Super Mastermind isn’t enormous, and it lends itself to some highly-efficient computerised storage.

There are 8 different colours of peg. We can express these colours as a number between 0 and 7, in three bits of binary, like this:

Decimal Binary Colour
0 000 Red
1 001 Orange
2 010 Yellow
3 011 Green
4 100 Blue
5 101 Pink
6 110 Purple
7 111 White

There are four pegs in a row, so we can express any given combination of coloured pegs as a 12-bit binary number. E.g. 100 110 111 010 would represent the permutation blue (100), purple (110), white (111), yellow (010). The total search space, therefore, is the range of numbers from 000000000000 through 111111111111… that is: decimal 0 through 4,095:

Decimal Binary Colours
0 000000000000 Red, red, red, red
1 000000000001 Red, red, red, orange
2 000000000010 Red, red, red, yellow
…………
4092 111111111100 White, white, white, blue
4093 111111111101 White, white, white, pink
4094 111111111110 White, white, white, purple
4095 111111111111 White, white, white, white

Whenever we make a guess, we get feedback in the form of two variables: each peg that is in the right place is a bull; each that represents a peg in the secret code but isn’t in the right place is a cow (the names come from Mastermind’s precursor, Bulls & Cows). Four bulls would be an immediate win (lucky!), any other combination of bulls and cows is still valuable information. Even a zero-score guess is valuable- potentially very valuable! – because it tells the player that none of the pegs they’ve guessed appear in the secret code.

A plastic Mastermind board in blue and yellow with ten guess spaces and eight pegs. The sixth guess is unscored but looks likely to be the valid solution.
If one of Wordle‘s parents was Scrabble, then this was the other. Just ask its Auntie Twitter.

Solving with Javascript

The latest versions of Javascript support binary literals and bitwise operations, so we can encode and decode between arrays of four coloured pegs (numbers 0-7) and the number 0-4,095 representing the guess as shown below. Decoding uses an AND bitmask to filter to the requisite digits then divides by the order of magnitude. Encoding is just a reduce function that bitshift-concatenates the numbers together.

116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
/**
 * Decode a candidate into four peg values by using binary bitwise operations.
 */
function decodeCandidate(candidate){
  return [
    (candidate & 0b111000000000) / 0b001000000000,
    (candidate & 0b000111000000) / 0b000001000000,
    (candidate & 0b000000111000) / 0b000000001000,
    (candidate & 0b000000000111) / 0b000000000001
  ];
}

/**
 * Given an array of four integers (0-7) to represent the pegs, in order, returns a single-number
 * candidate representation.
 */
function encodeCandidate(pegs) {
  return pegs.reduce((a, b)=>(a << 3) + b);
}

With this, we can simply:

  1. Produce a list of candidate solutions (an array containing numbers 0 through 4,095).
  2. Choose one candidate, use it as a guess, and ask the code-maker how it scores.
  3. Eliminate from the candidate solutions list all solutions that would not score the same number of bulls and cows for the guess that was made.
  4. Repeat from step #2 until you win.

Step 3’s the most important one there. Given a function getScore( solution, guess ) which returns an array of [ bulls, cows ] a given guess would score if faced with a specific solution, that code would look like this (I’m convined there must be a more-performant way to eliminate candidates from the list with XOR bitmasks, but I haven’t worked out what it is yet):

164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
/**
 * Given a guess (array of four integers from 0-7 to represent the pegs, in order) and the number
 * of bulls (number of pegs in the guess that are in the right place) and cows (number of pegs in the
 * guess that are correct but in the wrong place), eliminates from the candidates array all guesses
 * invalidated by this result. Return true if successful, false otherwise.
 */
function eliminateCandidates(guess, bulls, cows){
  const newCandidatesList = data.candidates.filter(candidate=>{
    const score = getScore(candidate, guess);
    return (score[0] == bulls) && (score[1] == cows);
  });
  if(newCandidatesList.length == 0) {
    alert('That response would reduce the candidate list to zero.');
    return false;
  }
  data.candidates = newCandidatesList;
  chooseNextGuess();
  return true;
}

I continued in this fashion to write a full solution (source code). It uses ReefJS for component rendering and state management, and you can try it for yourself right in your web browser. If you play against the online version I mentioned you’ll need to transpose the colours in your head: the physical version I play with the kids has pink and purple pegs, but the online one replaces these with brown and black.

Testing the solution

Let’s try it out against the online version:

As expected, my code works well-enough to win the game every time I’ve tried, both against computerised and in-person opponents. So – unless you’ve been actively thinking about the specifics of the algorithm I’ve employed – it might surprise you to discover that… my solution is very-much a suboptimal one!

A young boy sits cross-legged on the floor, grinning excitedly at a Mastermind board (from the code-maker's side).
My code has only failed to win a single game… and that turned out to because my opponent, playing overexcitedly, cheated in the third turn. To be fair, my code didn’t lose either, though: it identified that a mistake must have been made and we declared the round void when we identified the problem.

My solution is suboptimal

A couple of games in, the suboptimality of my solution became pretty visible. Sure, it still won every game, but it was a blunt instrument, and anybody who’s seriously thought about games like this can tell you why. You know how when you play e.g. Wordle (but not in “hard mode”) you sometimes want to type in a word that can’t possibly be the solution because it’s the best way to rule in (or out) certain key letters? This kind of strategic search space bisection reduces the mean number of guesses you need to solve the puzzle, and the same’s true in Mastermind. But because my solver will only propose guesses from the list of candidate solutions, it can’t make this kind of improvement.

Animation showing how three clues alone are sufficient to derive a unique answer from the search space of the original "break into us" lock puzzle.
My blog post about Break Into Us used a series of visual metaphors to show search space dissection, including this one. If you missed it, it might be worth reading.

Search space bisection is also used in my adverserial hangman game, but in this case the aim is to split the search space in such a way that no matter what guess a player makes, they always find themselves in the larger remaining portion of the search space, to maximise the number of guesses they have to make. Y’know, because it’s evil.

Screenshot showing a single guess row from Online Mastermind, with the guess Red, Red, Green, Green.
A great first guess, assuming you’re playing against a random code and your rules permit the code to have repeated colours, is a “1122” pattern.

There are mathematically-derived heuristics to optimise Mastermind strategy. The first of these came from none other than Donald Knuth (legend of computer science, mathematics, and pipe organs) back in 1977. His solution, published at probably the height of the game’s popularity in the amazingly-named Journal of Recreational Mathematics, guarantees a solution to the six-colour version of the game within five guesses. Ville [2013] solved an optimal solution for a seven-colour variant, but demonstrated how rapidly the tree of possible moves grows and the need for early pruning – even with powerful modern computers – to conserve memory. It’s a very enjoyable and readable paper.

But for my purposes, it’s unnecessary. My solver routinely wins within six, maybe seven guesses, and by nonchalantly glancing at my phone in-between my guesses I can now reliably guess our children’s codes quickly and easily. In the end, that’s what this was all about.

Black, white, brown, blue, green, orange and yellow Mastermind pegs in a disordered heap.× A plastic Mastermind board in brown and green; it has twelve spots for guessing and shows six coloured pegs. The game has been won on the sixth guess.× Screenshot showing Mastermind game from WebGamesOnline.com. Seven guesses have been made, each using only one colour for each of the four pegs, and no guesses are corect; only red pegs have never been guessed.× A plastic Mastermind board in blue and yellow with ten guess spaces and eight pegs. The sixth guess is unscored but looks likely to be the valid solution.× A young boy sits cross-legged on the floor, grinning excitedly at a Mastermind board (from the code-maker's side).× Screenshot showing a single guess row from Online Mastermind, with the guess Red, Red, Green, Green.×

Bash+Batch In One File

Today I wanted to write a script that I could execute from both *nix (using Bash or a similar shell) and on Windows (from a command prompt, i.e. a batch file). I found Max Norin’s solution which works, but has a few limitations, e.g. when run it outputs either the word “off” when run in *nix or the word “goto” when run on Windows. There’s got to be a sneaky solution, right?

Here’s my improved version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
@goto(){
  # Linux code here
  uname -o
}

@goto $@
exit

:(){
@echo off
rem Windows script here
echo %OS%

Mine exploits the fact that batch files can prefix commands with @ to suppress outputting them as they execute. So @goto can be a valid function name in bash/zsh etc. but is interpreted as a literal goto command in a Windows Command Prompt. This allows me to move the echo off command – which only has meaning to Windows – into the Windows section of the script and suppress it with @.

The file above can be saved as e.g. myfile.cmd and will execute in a Windows Command Prompt (or in MS-DOS) or your favourite *nix OS. Works in MacOS/BSD too, although obviously any more-sophisticated script is going to have to start working around the differences between GNU and non-GNU versions of core utilities, which is always a bit of a pain! Won’t work in sh because you can’t define functions like that.

But the short of it is you can run this on a stock *nix OS and get:

$ ./myfile.cmd
GNU/Linux

And you can run it on Windows and get:

> .\myfile.cmd
Windows_NT

You can’t put a shebang at the top because Windows hates it, but there might be a solution using PowerShell scripts (which use hashes for comments: that’s worth thinking about!). In any case: if the interpreter strictly matters you’ll probably want to shell to it on line 3 with e.g. bash -c.

Why would you want such a thing? I’m not sure. But there is is, if you do.

Pronouns in Three Rings

The Old Way

Prior to 2018, Three Rings had a relatively simple approach to how it would use pronouns when referring to volunteers.

If the volunteer’s gender was specified as a “masculine” gender (which particular options are available depends on the volunteer’s organisation, but might include “male”, “man”, “cis man”, and “trans man”), the system would use traditional masculine pronouns like “he”, “his”, “him” etc.

If the gender was specified as a “feminine” gender (e.g .”female”, “woman”, “cis women”, “trans woman”) the system would use traditional feminine pronouns like “she”, “hers”, “her” etc.

For any other answer, no specified answer, or an organisation that doesn’t track gender, we’d use singular “they” pronouns. Simple!

Three Rings' "Gender" drop-down, with "Female" selected.This selection was reflected throughout the system. Three Rings might say:

  • They have done 7 shifts by themselves.
  • She verified her email address was hers.
  • Would you like to sign him up to this shift?

Unfortunately, this approach didn’t reflect the diversity of personal pronouns nor how they’re applied. It didn’t support volunteer whose gender and pronouns are not conventionally-connected (“I am a woman and I use ‘them/they’ pronouns”), nor did it respect volunteers whose pronouns are not in one of these three sets (“I use ze/zir pronouns”)… a position it took me an embarrassingly long time to fully comprehend.

So we took a new approach:

The New Way

Three Rings' "Pronouns" drop-down, with "ze/zir/zir/zirs/zemself" selected.

From 2018 we allowed organisations to add a “Pronouns” property, allowing volunteers to select from 13 different pronoun sets. If they did so, we’d use it; failing that we’d continue to assume based on gender if it was available, or else use the singular “they”.

Flowchart showing how the pronouns used by Three Rings for a volunteer are contingent on what properties their org records, whether the volunteer has specified pronouns, failing that whether they've specified a gender, falling back to "singular they" pronouns.
The process has some further complexities to cover the fact that we say “they are” but “he is“, but this broadly covers it.

Let’s take a quick linguistics break

Three Rings‘ pronoun field always shows five personal pronouns, separated by slashes, because you can’t necessarily derive one from another. That’s one for each of five types:

  1. the subject, used when the person you’re talking about is primary argument to a verb (“he called”),
  2. object, for when the person you’re talking about is the secondary argument to a transitive verb (“he called her“),
  3. dependent possessive, for talking about a noun that belongs to a person (“this is their shift”),
  4. independent possessive, for talking about something that belongs to a person potentially would an explicit noun (“this is theirs“), and the
  5. reflexive (and intensive), two types which are generally the same in English, used mostly in Three Rings when a person is both the subject and indeirect of a  verb (“she signed herself up to a shift”).

Let’s see what those look like – here are the 13 pronoun sets supported by Three Rings at the time of writing:

Subject Object Possessive Reflexive/intensive
Dependent Independent
he him his himself
she her hers herself
they them their theirs themselves
e em eir eirs emself
ey eirself
hou hee hy hine hyself
hu hum hus humself
ne nem nir nirs nemself
per pers perself
thon thons thonself
ve ver vis verself
xe xem xyr xyrs xemself
ze zir zirs zemself

That’s all data-driven rather than hard-coded, by the way, so adding additional pronoun sets is very easy for our developers. In fact, it’s even possible for us to apply an additional “override” on an individual, case-by-case basis: all we need to do is specify the five requisite personal pronouns, separated by slashes, and Three Rings understands how to use them.

Writing code that respects pronouns

Behind the scenes, the developers use a (binary-gendered, for simplicity) convenience function to produce output, and the system corrects for the pronouns appropriate to the volunteer in question:

<%= @volunteer.his_her.capitalize %>
account has been created for
<%= @volunteer.him_her %>
so
<%= @volunteer.he_she %>
can now log in.

The code above will, dependent on the pronouns specified for the volunteer @volunteer, output something like:

  • His account has been created for him so he can now log in.
  • Her account has been created for her so she can now log in.
  • Their account has been created for them so they can now log in.
  • Eir account has been created for em so ey can now log in.
  • Etc.

We’ve got extended functions to automatically detect cases where the use of second person pronouns might be required (“Your account has been created for you so you can now log in.”) as well as to help us handle the fact that we say “they are” but “he/she/ey/ze/etc. is“.

It’s all pretty magical and “just works” from a developer’s perspective. I’m sure most of our volunteer developers don’t think about the impact of pronouns at all when they code; they just get on with it.

Is that a complete solution?

Does this go far enough? Possibly not. This week, one of our customers contacted us to ask:

Is there any way to give the option to input your own pronouns? I ask as some people go by she/them or he/them and this option is not included…

You can probably see what’s happened here: some organisations have taken our pronouns property – which exists primarily to teach the system itself how to talk about volunteers – and are using it to facilitate their volunteers telling one another what their pronouns are.

What’s the difference? Well:

When a human discloses that their pronouns are “she/they” to another human, they’re saying “You can refer to me using either traditional feminine pronouns (she/her/hers etc.) or the epicene singular ‘they’ (they/their/theirs etc.)”.

But if you told Three Rings your pronouns were “she/her/their/theirs/themselves”, it would end up using a mixture of the two, even in the same sentence! Consider:

  • She has done 7 shifts by themselves.
  • She verified her email address was theirs.

That’s some pretty clunky English right there! Mixing pronoun sets for the same person within a sentence is especially ugly, but even mixing them within the same page can cause confusion. We can’t trivially meet this customer’s request simply by adding new pronoun sets which mix things up a bit! We need to get smarter.

A Newer Way?

Ultimately, we’re probably going to need to differentiate between a more-rigid “what pronouns should Three Rings use when talking about you” and a more-flexible, perhaps optional “what pronouns should other humans use for you”? Alternatively, maybe we could allow people to select multiple pronoun sets to display but Three Rings would only use one of them (at least, one of them at a time!): “which of the following sets of pronouns do you use: select as many as apply”?

Even after this, there’ll always be more work to do.

For instance: I’ve met at least one person who uses no pronouns! By this, they actually mean they use no third-person personal pronouns (if they actually used no pronouns they wouldn’t say “I”, “me”, “my”, “mine” or “myself” and wouldn’t want others to say “you”, “your”, “yours” and “yourself” to them)! Semantics aside… for these people Three Rings should use the person’s name rather than a pronoun.

Maybe we can get there one day.

Three Rings' "Pronouns" drop-down, with "Dan/Dan/Dan/Dan's/Dan's self" selected.
Three Rings is already capable of supporting people who use no pronouns, but we don’t yet have a user interface to help them specify this! Maybe it’d look like this?

But so long as Three Rings continues to remain ahead of the curve in its respect for and understanding of pronoun use then I’ll be happy.

Our mission is to focus on volunteers and make volunteering easier. At the heart of that mission is treating volunteers with respect. Making sure our system embraces the diversity of the 65,000+ volunteers who use it by using pronouns correctly might be a small part of that, but it’s a part of it, and I for one am glad we make the effort.

Flowchart showing how the pronouns used by Three Rings for a volunteer are contingent on what properties their org records, whether the volunteer has specified pronouns, failing that whether they've specified a gender, falling back to "singular they" pronouns.× Three Rings' "Pronouns" drop-down, with "Dan/Dan/Dan/Dan's/Dan's self" selected.× Three Rings' "Gender" drop-down, with "Female" selected.× Three Rings' "Pronouns" drop-down, with "ze/zir/zir/zirs/zemself" selected.×

Making an RSS feed of YOURLS shortlinks

As you might know if you were paying close attention in Summer 2019, I run a “URL shortener” for my personal use. You may be familiar with public URL shorteners like TinyURL and Bit.ly: my personal URL shortener is basically the same thing, except that only I am able to make short-links with it. Compared to public ones, this means I’ve got a larger corpus of especially-short (e.g. 2/3 letter) codes available for my personal use. It also means that I’m not dependent on the goodwill of a free siloed service and I can add exactly the features I want to it.

Diagram showing the relationships of the DanQ.me ecosystem. Highlighted is the injection of links into the "S.2" link shortener and the export of these shortened links by RSS into FreshRSS.
Little wonder then that my link shortener sat so close to me on my ecosystem diagram the other year.

For the last nine years my link shortener has been S.2, a tool I threw together in Ruby. It stores URLs in a sequentially-numbered database table and then uses the Base62-encoding of the primary key as the “code” part of the short URL. Aside from the fact that when I create a short link it shows me a QR code to I can easily “push” a page to my phone, it doesn’t really have any “special” features. It replaced S.1, from which it primarily differed by putting the code at the end of the URL rather than as part of the domain name, e.g. s.danq.me/a0 rather than a0.s.danq.me: I made the switch because S.1 made HTTPS a real pain as well as only supporting Base36 (owing to the case-insensitivity of domain names).

But S.2’s gotten a little long in the tooth and as I’ve gotten busier/lazier, I’ve leant into using or adapting open source tools more-often than writing my own from scratch. So this week I switched my URL shortener from S.2 to YOURLS.

Screenshot of YOURLS interface showing Dan Q's list of shortened links. Six are shown of 1,939 total.
YOURLs isn’t the prettiest tool in the world, but then it doesn’t have to be: only I ever see the interface pictured above!

One of the things that attracted to me to YOURLS was that it had a ready-to-go Docker image. I’m not the biggest fan of Docker in general, but I do love the convenience of being able to deploy applications super-quickly to my household NAS. This makes installing and maintaining my personal URL shortener much easier than it used to be (and it was pretty easy before!).

Another thing I liked about YOURLS is that it, like S.2, uses Base62 encoding. This meant that migrating my links from S.2 into YOURLS could be done with a simple cross-database INSERT... SELECT statement:

INSERT INTO yourls.yourls_url(keyword, url, title, `timestamp`, clicks)
  SELECT shortcode, url, title, created_at, 0 FROM danq_short.links

But do you know what’s a bigger deal for my lifestack than my URL shortener? My RSS reader! I’ve written about it a lot, but I use RSS for just about everything and my feed reader is my first, last, and sometimes only point of contact with the Web! I’m so hooked-in to my RSS ecosystem that I’ll use my own middleware to add feeds to sites that don’t have them, or for which I’m not happy with the feed they provide, e.g. stripping sports out of BBC News, subscribing to webcomics that don’t provide such an option (sometimes accidentally hacking into sites on the way), and generating “complete” archives of series’ of posts so I can use my reader to track my progress.

One of S.1/S.2’s features was that it exposed an RSS feed at a secret URL for my reader to ingest. This was great, because it meant I could “push” something to my RSS reader to read or repost to my blog later. YOURLS doesn’t have such a feature, and I couldn’t find anything in the (extensive) list of plugins that would do it for me. I needed to write my own.

Partial list of Dan's RSS feed subscriptions, including Jeremy Keith, Jim Nielson, Natalie Lawhead, Bruce Schneier, Scott O'Hara, "Yahtzee", BBC News, and several podcasts, as well as (highlighted) "Dan's Short Links", which has 5 unread items.
In some ways, subscribing “to yourself” is a strange thing to do. In other ways… shut up, I’ll do what I like.

I could have written a YOURLS plugin. Or I could have written a stack of code in Ruby, PHP, Javascript or some other language to bridge these systems. But as I switched over my shortlink subdomain s.danq.me to its new home at danq.link, another idea came to me. I have direct database access to YOURLS (and the table schema is super simple) and the command-line MariaDB client can output XML… could I simply write an XML Transformation to convert database output directly into a valid RSS feed? Let’s give it a go!

I wrote a script like this and put it in my crontab:

mysql --xml yourls -e                                                                                                                     \
      "SELECT keyword, url, title, DATE_FORMAT(timestamp, '%a, %d %b %Y %T') AS pubdate FROM yourls_url ORDER BY timestamp DESC LIMIT 30" \
    | xsltproc template.xslt -                                                                                                            \
    | xmllint --format -                                                                                                                  \
    > output.rss.xml

The first part of that command connects to the yourls database, sets the output format to XML, and executes an SQL statement to extract the most-recent 30 shortlinks. The DATE_FORMAT function is used to mould the datetime into something approximating the RFC-822 standard for datetimes as required by RSS. The output produced looks something like this:

<?xml version="1.0"?>
<resultset statement="SELECT keyword, url, title, timestamp FROM yourls_url ORDER BY timestamp DESC LIMIT 30" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <row>
        <field name="keyword">VV</field>
        <field name="url">https://webdevbev.co.uk/blog/06-2021/perfect-is-the-enemy-of-good.html</field>
        <field name="title"> Perfect is the enemy of good || Web Dev Bev</field>
        <field name="timestamp">2021-09-26 17:38:32</field>
  </row>
  <row>
        <field name="keyword">VU</field>
        <field name="url">https://webdevlaw.uk/2021/01/30/why-generation-x-will-save-the-web/</field>
        <field name="title">Why Generation X will save the web  Hi, Im Heather Burns</field>
        <field name="timestamp">2021-09-26 17:38:26</field>
  </row>

  <!-- ... etc. ... -->
  
</resultset>

We don’t see this, though. It’s piped directly into the second part of the command, which  uses xsltproc to apply an XSLT to it. I was concerned that my XSLT experience would be super rusty as I haven’t actually written any since working for my former employer SmartData back in around 2005! Back then, my coworker Alex and I spent many hours doing XML backflips to implement a system that converted complex data outputs into PDF files via an XSL-FO intermediary.

I needn’t have worried, though. Firstly: it turns out I remember a lot more than I thought from that project a decade and a half ago! But secondly, this conversion from MySQL/MariaDB XML output to RSS turned out to be pretty painless. Here’s the template.xslt I ended up making:

<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:template match="resultset">
    <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
      <channel>
        <title>Dan's Short Links</title>
        <description>Links shortened by Dan using danq.link</description>
        <link> [ MY RSS FEED URL ] </link>
        <atom:link href=" [ MY RSS FEED URL ] " rel="self" type="application/rss+xml" />
        <lastBuildDate><xsl:value-of select="row/field[@name='pubdate']" /> UTC</lastBuildDate>
        <pubDate><xsl:value-of select="row/field[@name='pubdate']" /> UTC</pubDate>
        <ttl>1800</ttl>
        <xsl:for-each select="row">
          <item>
            <title><xsl:value-of select="field[@name='title']" /></title>
            <link><xsl:value-of select="field[@name='url']" /></link>
            <guid>https://danq.link/<xsl:value-of select="field[@name='keyword']" /></guid>
            <pubDate><xsl:value-of select="field[@name='pubdate']" /> UTC</pubDate>
          </item>
        </xsl:for-each>
      </channel>
    </rss>
  </xsl:template>
</xsl:stylesheet>

That uses the first (i.e. most-recent) shortlink’s timestamp as the feed’s pubDate, which makes sense: unless you’re going back and modifying links there’s no more-recent changes than the creation date of the most-recent shortlink. Then it loops through the returned rows and creates an <item> for each; simple!

The final step in my command runs the output through xmllint to prettify it. That’s not strictly necessary, but it was useful while debugging and as the whole command takes milliseconds to run once every quarter hour or so I’m not concerned about the overhead. Using these native binaries (plus a little configuration), chained together with pipes, had already resulted in way faster performance (with less code) than if I’d implemented something using a scripting language, and the result is a reasonably elegant “scratch your own itch”-type solution to the only outstanding barrier that was keeping me on S.2.

All that remained for me to do was set up a symlink so that the resulting output.rss.xml was accessible, over the web, to my RSS reader. I hope that next time I’m tempted to write a script to solve a problem like this I’ll remember that sometimes a chain of piped *nix utilities can provide me a slicker, cleaner, and faster solution.

Update: Right as I finished writing this blog post I discovered that somebody had already solved this problem using PHP code added to YOURLS; it’s just not packaged as a plugin so I didn’t see it earlier! Whether or not I use this alternate approach or stick to what I’ve got, the process of implementing this YOURLS-database ➡ XML ➡  XSLTRSS chain was fun and informative.

Diagram showing the relationships of the DanQ.me ecosystem. Highlighted is the injection of links into the "S.2" link shortener and the export of these shortened links by RSS into FreshRSS.× Screenshot of YOURLS interface showing Dan Q's list of shortened links. Six are shown of 1,939 total.× Partial list of Dan's RSS feed subscriptions, including Jeremy Keith, Jim Nielson, Natalie Lawhead, Bruce Schneier, Scott O'Hara, "Yahtzee", BBC News, and several podcasts, as well as (highlighted) "Dan's Short Links", which has 5 unread items.×

Quickly Solving Jigidi Puzzles

tl;dr? Just want instructions on how to solve Jigidi puzzles really fast with the help of your browser’s dev tools? Skip to that bit.

This approach doesn’t work any more. Want to see one that still does (but isn’t quite so automated)? Here you go!

I don’t enjoy jigsaw puzzles

I enjoy geocaching. I don’t enjoy jigsaw puzzles. So mystery caches that require you to solve an online jigsaw puzzle in order to get the coordinates really don’t do it for me. When I’m geocaching I want to be outdoors exploring, not sitting at my computer gradually dragging pixels around!

A completed 1000-piece "Where's Wally?" jigsaw.
Don’t let anybody use my completion of this 1000-piece jigsaw puzzle over New Year as evidence that I’m lying and actually like jigsaws.

Many of these mystery caches use Jigidi to host these jigsaw puzzles. An earlier version of Jigidi was auto-solvable with a userscript, but the service has continued to be developed and evolve and the current version works quite hard to make it hard for simple scripts to solve. For example, it uses a WebSocket connection to telegraph back to the server how pieces are moved around and connected to one another and the server only releases the secret “you’ve solved it” message after it detects that the pieces have been arranged in the appropriate relative configuration.

A nine-piece jigsaw puzzle with the pieces numbered 1 through 9; only the ninth piece is detached.
I made a simple Jigidi puzzle for demonstration purposes. Do you think you can manage a nine-piece jigsaw?

If there’s one thing I enjoy more than jigsaw puzzles – and as previously established there are about a billion things I enjoy more than jigsaw puzzles – it’s reverse-engineering a computer system to exploit its weaknesses. So I took a dive into Jigidi’s client-side source code. Here’s what it does:

  1. Get from the server the completed image and the dimensions (number of pieces).
  2. Cut the image up into the appropriate number of pieces.
  3. Shuffle the pieces.
  4. Establish a WebSocket connection to keep the server up-to-date with the relative position of the pieces.
  5. Start the game: the player can drag-and-drop pieces and if two adjacent pieces can be connected they lock together. Both pieces have to be mostly-visible (not buried under other pieces), presumably to prevent players from just making a stack and then holding a piece against each edge of it to “fish” for its adjacent partners.
Javascirpt code where the truthiness of this.j affects whether or not the pieces are shuffled.
I spent some time tracing call stacks to find this line… only to discover that it’s one of only four lines to actually contain the word “shuffle” and I could have just searched for it…

Looking at that process, there’s an obvious weak point – the shuffling (point 3) happens client-side, and before the WebSocket sync begins. We could override the shuffling function to lay the pieces out in a grid, but we’d still have to click each of them in turn to trigger the connection. Or we could skip the shuffling entirely and just leave the pieces in their default positions.

An unshuffled stack of pieces from the nine-piece jigsaw. Piece number nine is on top of the stack.
An unshuffled jigsaw appears as a stack, as if each piece from left to right and then top to bottom were placed one at a time into a pile.

And what are the default positions? It’s a stack with the bottom-right jigsaw piece on the top, the piece to the left of it below it, then the piece to the left of that and son on through the first row… then the rightmost piece from the second-to-bottom row, then the piece to the left of that, and so on.

That’s… a pretty convenient order if you want to solve a jigsaw. All you have to do is drag the top piece to the right to join it to the piece below that. Then move those two to the right to join to the piece below them. And so on through the bottom row before moving back – like a typewriter’s carriage return – to collect the second-to-bottom row and so on.

How can I do this?

If you’d like to cheat at Jigidi jigsaws, this approach works as of the time of writing. I used Firefox, but the same basic approach should work with virtually any modern desktop web browser.

  1. Go to a Jigidi jigsaw in your web browser.
  2. Pop up your browser’s developer tools (F12, usually) and switch to the Debugger tab. Open the file game/js/release.js and uncompress it by pressing the {} button, if necessary.
  3. Find the line where the code considers shuffling; right now for me it’s like 3671 and looks like this:
    return this.j ? (V.info('board-data-bytes already exists, no need to send SHUFFLE'), Promise.resolve(this.j)) : new Promise(function (d, e) {
    Javascirpt code where the truthiness of this.j affects whether or not the pieces are shuffled.
    I spent some time tracing call stacks to find this line… only to discover that it’s one of only four lines to actually contain the word “shuffle” and I could have just searched for it…
  4. Set a breakpoint on that line by clicking its line number.
  5. Restart the puzzle by clicking the restart button to the right of the timer. The puzzle will reload but then stop with a “Paused on breakpoint” message. At this point the application is considering whether or not to shuffle the pieces, which normally depends on whether you’ve started the puzzle for the first time or you’re continuing a saved puzzle from where you left off.
    Paused on breakpoint dialog with play button.
  6. In the developer tools, switch to the Console tab.
  7. Type: this.j = true (this ensures that the ternary operation we set the breakpoint on will resolve to the true condition, i.e. not shuffle the pieces).
    this.j = true
  8. Press the play button to continue running the code from the breakpoint. You can now close the developer tools if you like.
  9. Solve the puzzle as described/shown above, by moving the top piece on the stack slightly to the right, repeatedly, and then down and left at the end of each full row.
    Jigsaw being solved by moving down-and-right.

Update 2021-09-22: Abraxas observes that Jigidi have changed their code, possibly in response to this shortcut. Unfortunately for them, while they continue to perform shuffling on the client-side they’ll always be vulnerable to this kind of simple exploit. Their new code seems to be named not release.js but given a version number; right now it’s 14.3.1977. You can still expand it in the same way, and find the shuffling code: right now for me this starts on line 1129:

Put a breakpoint on line 1129. This code gets called twice, so the first time the breakpoint gets hit just hit continue and play on until the second time. The second time it gets hit, move the breakpoint to line 1130 and press continue. Then use the console to enter the code d = a.G and continue. Only one piece of jigsaw will be shuffled; the rest will be arranged in a neat stack like before (I’m sure you can work out where the one piece goes when you get to it).

Update 2023-03-09: I’ve not had time nor inclination to re-“break” Jigidi’s shuffler, but on the rare ocassions I’ve needed to solve a Jigidi, I’ve come up with a technique that replaces a jigsaw’s pieces with ones that each show the row and column number they belong to, as well as colour-coding the rows and columns and drawing horizontal and vertical bars to help visual alignment. It makes the process significantly less-painful. It’s still pretty buggy code though and I end up tweaking it each and every time I use it, but it certainly works and makes jigsaws that lack clear visual markers (e.g. large areas the same colour) a lot easier.

An almost-solved Jigidi jigsaw striped and painted to make solving easier.

A completed 1000-piece "Where's Wally?" jigsaw.× Javascirpt code where the truthiness of this.j affects whether or not the pieces are shuffled.× An unshuffled stack of pieces from the nine-piece jigsaw. Piece number nine is on top of the stack.× ×