Article

Can I use HTTP Basic Auth in URLs?

7 September 2021 11:26 1

Web standards sometimes disappear

Sometimes a web standard disappears quickly at the whim of some company, perhaps to a great deal of complaint (and at least one joke).

But sometimes, they disappear slowly, like this kind of web address:

http://username:password@example.com/somewhere

If you’ve not seen a URL like that before, that’s fine, because the answer to the question “Can I still use HTTP Basic Auth in URLs?” is, I’m afraid: no, you probably can’t.

But by way of a history lesson, let’s go back and look at what these URLs were, why they died out, and how web browsers handle them today. Thanks to Ruth who asked the original question that inspired this post.

Basic authentication

The early Web wasn’t built for authentication. A resource on the Web was theoretically accessible to all of humankind: if you didn’t want it in the public eye, you didn’t put it on the Web! A reliable method wouldn’t become available until the concept of state was provided by Netscape’s invention of HTTP cookies in 1994, and even that wouldn’t see widespread for several years, not least because implementing a CGI (or similar) program to perform authentication was a complex and computationally-expensive option for all but the biggest websites.

Comic showing a conversation between a web browser and server. Browser: "Show me that page. (GET /)" Server: "No, take a ticket and fill this form. (Redirect, Set-Cookie)" Browser: "I've filled your form and here's your ticket (POST request with Cookie set)" Server: "Okay, Keep hold of your ticket. (Redirect, Set-Cookie)" Browser: "Show me that page, here's my ticket. (GET /, Cookie:)" — A simplified view of the form-and-cookie based authentication system used by virtually every website today, but which was too computationally-expensive for many sites in the 1990s.

1996’s HTTP/1.0 specification tried to simplify things, though, with the introduction of the WWW-Authenticate header. The idea was that when a browser tried to access something that required authentication, the server would send a 401 Unauthorized response along with a WWW-Authenticate header explaining how the browser could authenticate itself. Then, the browser would send a fresh request, this time with an Authorization: header attached providing the required credentials. Initially, only “basic authentication” was available, which basically involved sending a username and password in-the-clear unless SSL (HTTPS) was in use, but later, digest authentication and a host of others would appear.

Comic showing conversation between web browser and server. Browser: "Show me that page (GET /)" Server: "No. Send me credentials. (HTTP 401, WWW-Authenticate)" Browser: "Show me that page. I enclose credentials (Authorization)" Server: "Okay (HTTP 200)" — For all its faults, HTTP Basic Authentication (and its near cousins) are certainly elegant.

Webserver software quickly added support for this new feature and as a result web authors who lacked the technical know-how (or permission from the server administrator) to implement more-sophisticated authentication systems could quickly implement HTTP Basic Authentication, often simply by adding a .htaccess file to the relevant directory. .htaccess files would later go on to serve many other purposes, but their original and perhaps best-known purpose – and the one that gives them their name – was access control.

Credentials in the URL

A separate specification, not specific to the Web (but one of Tim Berners-Lee’s most important contributions to it), described the general structure of URLs as follows:

<scheme>://<username>:<password>@<host>:<port>/<url-path>#<fragment>

At the time that specification was written, the Web didn’t have a mechanism for passing usernames and passwords: this general case was intended only to apply to protocols that did have these credentials. An example is given in the specification, and clarified with “An optional user name. Some schemes (e.g., ftp) allow the specification of a user name.”

But once web browsers had WWW-Authenticate, virtually all of them added support for including the username and password in the web address too. This allowed for e.g. hyperlinks with credentials embedded in them, which made for very convenient bookmarks, or partial credentials (e.g. just the username) to be included in a link, with the user being prompted for the password on arrival at the destination. So far, so good.

Comic showing conversation between web browser and server. Browser asks for a page, providing an Authorization: header outright; server responds with the page immediately. — Encoding authentication into the URL provided an incredible shortcut at a time when Web round-trip times were much longer owing to higher latencies and no keep-alives.

This is why we can’t have nice things

The technique fell out of favour as soon as it started being used for nefarious purposes. It didn’t take long for scammers to realise that they could create links like this:

https://YourBank.com@HackersSite.com/

Everything we were teaching users about checking for “https://” followed by the domain name of their bank… was undermined by this user interface choice. The poor victim would actually be connecting to e.g. HackersSite.com, but a quick glance at their address bar would leave them convinced that they were talking to YourBank.com!

Theoretically: widespread adoption of EV certificates coupled with sensible user interface choices (that were never made) could have solved this problem, but a far simpler solution was just to not show usernames in the address bar. Web developers were by now far more excited about forms and cookies for authentication anyway, so browsers started curtailing the “credentials in addresses” feature.

Internet Explorer window showing https://YourBank.com@786590867/ in the address bar. — Users trained to look for “https://” followed by the site they wanted would often fall for scams like this one: the real domain name is *after* the @-sign. (This attacker is also using dword notation to obfuscate their IP address; this dated technique wasn’t often employed alongside this kind of scam, but it’s another historical oddity I enjoy so I’m shoehorning it in.)

(There are other reasons this particular implementation of HTTP Basic Authentication was less-than-ideal, but this reason is the big one that explains why things had to change.)

One by one, browsers made the change. But here’s the interesting bit: the browsers didn’t always make the change in the same way.

How different browsers handle basic authentication in URLs

Let’s examine some popular browsers. To run these tests I threw together a tiny web application that outputs the Authorization: header passed to it, if present, and can optionally send a 401 Unauthorized response along with a WWW-Authenticate: Basic realm="Test Site" header in order to trigger basic authentication. Why both? So that I can test not only how browsers handle URLs containing credentials when an authentication request is received, but how they handle them when one is not. This is relevant because some addresses – often API endpoints – have optional HTTP authentication, and it’s sometimes important for a user agent (albeit typically a library or command-line one) to pass credentials without first being prompted.

In each case, I tried each of the following tests in a fresh browser instance:

Go to http://<username>:<password>@<domain>/optional (authentication is optional).
Go to http://<username>:<password>@<domain>/mandatory (authentication is mandatory).
Experiment 1, then f0llow relative hyperlinks (which should correctly retain the credentials) to /mandatory.
Experiment 2, then follow relative hyperlinks to the /optional.

I’m only testing over the http scheme, because I’ve no reason to believe that any of the browsers under test treat the https scheme differently.

Chromium desktop family

Chrome 93 and Edge 93 both immediately suppressed the username and password from the address bar, along with the “http://” as we’ve come to expect of them. Like the “http://”, though, the plaintext username and password are still there. You can retrieve them by copy-pasting the entire address.

Opera 78 similarly suppressed the username, password, and scheme, but didn’t retain the username and password in a way that could be copy-pasted out.

Authentication was passed only when landing on a “mandatory” page; never when landing on an “optional” page. Refreshing the page or re-entering the address with its credentials did not change this.

Navigating from the “optional” page to the “mandatory” page using only relative links retained the username and password and submitted it to the server when it became mandatory, even Opera which didn’t initially appear to retain the credentials at all.

Navigating from the “mandatory” to the “optional” page using only relative links, or even entering the “optional” page address with credentials after visiting the “mandatory” page, does not result in authentication being passed to the “optional” page. However, it’s interesting to note that once authentication has occurred on a mandatory page, pressing enter at the end of the address bar on the optional page, with credentials in the address bar (whether visible or hidden from the user) does result in the credentials being passed to the optional page! They continue to be passed on each subsequent load of the “optional” page until the browsing session is ended.

Firefox desktop

Firefox 91 does a clever thing very much in-line with its image as a browser that puts decision-making authority into the hands of its user. When going to the “optional” page first it presents a dialog, warning the user that they’re going to a site that does not specifically request a username, but they’re providing one anyway. If the user says that no, navigation ceases (the GET request for the page takes place the same either way; this happens before the dialog appears). Strangely: regardless of whether the user selects yes or no, the credentials are not passed on the “optional” page. The credentials (although not the “http://”) appear in the address bar while the user makes their decision.

Similar to Opera, the credentials do not appear in the address bar thereafter, but they’re clearly still being stored: if the refresh button is pressed the dialog appears again. It does not appear if the user selects the address bar and presses enter.

Similarly, going to the “mandatory” page in Firefox results in an informative dialog warning the user that credentials are being passed. I like this approach: not only does it help protect the user from the use of authentication as a tracking technique (an old technique that I’ve not seen used in well over a decade, mind), it also helps the user be sure that they’re logging in using the account they mean to, when following a link for that purpose. Again, clicking cancel stops navigation, although the initial request (with no credentials) and the 401 response has already occurred.

Visiting any page within the scope of the realm of the authentication after visiting the “mandatory” page results in credentials being sent, whether or not they’re included in the address. This is probably the most-true implementation to the expectations of the standard that I’ve found in a modern graphical browser.

Safari desktop

Safari 14 never displays or uses credentials provided via the web address, whether or not authentication is mandatory. Mandatory authentication is always met by a pop-up dialog, even if credentials were provided in the address bar. Boo!

Once passed, credentials are later provided automatically to other addresses within the same realm (i.e. optional pages).

Older browsers

Let’s try some older browsers.

From version 7 onwards – right up to the final version 11 – Internet Explorer fails to even recognise addresses with authentication credentials in as legitimate web addresses, regardless of whether or not authentication is requested by the server. It’s easy to assume that this is yet another missing feature in the browser we all love to hate, but it’s interesting to note that credentials-in-addresses is permitted for ftp:// URLs…

…and if you go back a little way, Internet Explorer 6 and below supported credentials in the address bar pretty much as you’d expect based on the standard. The error message seen in IE7 and above is a deliberate design decision, albeit a somewhat knee-jerk reaction to the security issues posed by the feature (compare to the more-careful approach of other browsers).

These older versions of IE even (correctly) retain the credentials through relative hyperlinks, allowing them to be passed when they become mandatory. They’re not passed on optional pages unless a mandatory page within the same realm has already been encountered.

Pre-Mozilla Netscape behaved the same way. Truly this was the de facto standard for a long period on the Web, and the varied approaches we see today are the anomaly. That’s a strange observation to make, considering how much the Web of the 1990s was dominated by incompatible implementations of different Web features (I’ve written about the <blink> and <marquee> tags before, which was perhaps the most-visible division between the Microsoft and Netscape camps, but there were many, many more).

Interestingly: by Netscape 7.2 the browser’s behaviour had evolved to be the same as modern Firefox’s, except that it still displayed the credentials in the address bar for all to see.

Now here’s a real gem: pre-Chromium Opera. It would send credentials to “mandatory” pages and remember them for the duration of the browsing session, which is great. But it would also send credentials when passed in a web address to “optional” pages. However, it wouldn’t remember them on optional pages unless they remained in the address bar: this feels to me like an optimum balance of features for power users. Plus, it’s one of very few browsers that permitted you to change credentials mid-session: just by changing them in the address bar! Most other browsers, even to this day, ignore changes to HTTP Authentication credentials, which was sometimes be a source of frustration back in the day.

Finally, classic Opera was the only browser I’ve seen to mask the password in the address bar, turning it into a series of asterisks. This ensures the user knows that a password was used, but does not leak any sensitive information to shoulder-surfers (the length of the “masked” password was always the same length, too, so it didn’t even leak the length of the password). Altogether a spectacular design and a great example of why classic Opera was way ahead of its time.

The Command-Line

Most people using web addresses with credentials embedded within them nowadays are probably working with code, APIs, or the command line, so it’s unsurprising to see that this is where the most “traditional” standards-compliance is found.

I was unsurprised to discover that giving curl a username and password in the URL meant that username and password was sent to the server (using Basic authentication, of course, if no authentication was requested):

$ curl http://alpha:beta@localhost/optional Header: Basic YWxwaGE6YmV0YQ== $ curl http://alpha:beta@localhost/mandatory Header: Basic YWxwaGE6YmV0YQ==

However, wget did catch me out. Hitting the same addresses with wget didn’t result in the credentials being sent except where it was mandatory (i.e. where a HTTP 401 response and a WWW-Authenticate: header was received on the initial attempt). To force wget to send credentials when they haven’t been asked-for requires the use of the --http-user and --http-password switches:

$ wget http://alpha:beta@localhost/optional -qO- Header: $ wget http://alpha:beta@localhost/mandatory -qO- Header: Basic YWxwaGE6YmV0YQ==

lynx does a cute and clever thing. Like most modern browsers, it does not submit credentials unless specifically requested, but if they’re in the address bar when they become mandatory (e.g. because of following relative hyperlinks or hyperlinks containing credentials) it prompts for the username and password, but pre-fills the form with the details from the URL. Nice.

What’s the status of HTTP (Basic) Authentication?

HTTP Basic Authentication and its close cousin Digest Authentication (which overcomes some of the security limitations of running Basic Authentication over an unencrypted connection) is very much alive, but its use in hyperlinks can’t be relied upon: some browsers (e.g. IE, Safari) completely munge such links while others don’t behave as you might expect. Other mechanisms like Bearer see widespread use in APIs, but nowhere else.

The WWW-Authenticate: and Authorization: headers are, in some ways, an example of the best possible way to implement authentication on the Web: as an underlying standard independent of support for forms (and, increasingly, Javascript), cookies, and complex multi-part conversations. It’s easy to imagine an alternative timeline where these standards continued to be collaboratively developed and maintained and their shortfalls – e.g. not being able to easily log out when using most graphical browsers! – were overcome. A timeline in which one might write a login form like this, knowing that your e.g. “authenticate” attributes would instruct the browser to send credentials using an Authorization: header:

<form method="get" action="/" authenticate="Basic"> <label for="username">Username:</label> <input type="text" id="username" authenticate="username"> <label for="password">Password:</label> <input type="text" id="password" authenticate="password"> <input type="submit" value="Log In"> </form>

In such a world, more-complex authentication strategies (e.g. multi-factor authentication) could involve encoding forms as JSON. And single-sign-on systems would simply involve the browser collecting a token from the authentication provider and passing it on to the third-party service, directly through browser headers, with no need for backwards-and-forwards redirects with stacks of information in GET parameters as is the case today. Client-side certificates – long a powerful but neglected authentication mechanism in their own right – could act as first class citizens directly alongside such a system, providing transparent second-factor authentication wherever it was required. You wouldn’t have to accept a tracking cookie from a site in order to log in (or stay logged in), and if your browser-integrated password safe supported it you could log on and off from any site simply by toggling that account’s “switch”, without even visiting the site: all you’d be changing is whether or not your credentials would be sent when the time came.

The Web has long been on a constant push for the next new shiny thing, and that’s sometimes meant that established standards have been neglected prematurely or have failed to evolve for longer than we’d have liked. Consider how long it took us to get the <video> and <audio> elements because the “new shiny” Flash came to dominate, how the Web Payments API is only just beginning to mature despite over 25 years of ecommerce on the Web, or how we still can’t use Link: headers for all the things we can use <link> elements for despite them being semantically-equivalent!

The new model for Web features seems to be that new features first come from a popular JavaScript implementation, and then eventually it evolves into a native browser feature: for example HTML form validations, which for the longest time could only be done client-side using scripting languages. I’d love to see somebody re-think HTTP Authentication in this way, but sadly we’ll never get a 100% solution in JavaScript alone: (distributed SSO is almost certainly off the table, for example, owing to cross-domain limitations).

Or maybe it’s just a problem that’s waiting for somebody cleverer than I to come and solve it. Want to give it a go?

Get Lost on the Web

2 September 2021 19:06 1

Get lost

I got lost on the Web this week, but it was harder than I’d have liked.

The Ypsilanti Water Tower, at the intersection of Washtenaw Avenue and Cross Street, Ypsilanti, Michigan. The tower is listed in the National Register of Historic Places, and is a National Historic Civil Engineering Landmark. An American flag and a Greek flag are flying, and a bust of the Greek general, Demetrios Ypsilantis (also commonly spelled "Demetrius Ypsilanti"), for whom the city is named, is in the foreground. Photo by Dwight Burdette, used under a Creative Commons license. — Now that’s a suggestive erection. Photo by Dwight Burdette.

There was a discussion this week in the Abnib WhatsApp group about whether a particular illustration of a farm was full of phallic imagery (it was). This left me wondering if anybody had ever tried to identify the most-priapic buildings in the world. Of course towers often look at least a little bit like their architects were compensating for something, but some – like the Ypsilanti Water Tower in Michigan pictured above – go further than others.

I quickly found the Wikipedia article for the Most Phallic Building Contest in 2003, so that was my jumping-off point. It’s easy enough to get lost on Wikipedia alone, but sometimes you feel the need for a primary source. I was delighted to discover that the web pages for the Most Phallic Building Contest are still online 18 years after the competition ended!

1969 shot tower at Tower Wharf, Bristol. Photo by Anthony O'Neil, used under a Creative Commons license. — The Cheese Lane Shot Tower in Bristol – politely described as a “Q-tip” shape – was built in 1969 to replace the world’s first shot tower elsewhere in the city. Photo by Anthony O’Neil.

Link rot is a serious problem on the Web, to such an extent that it’s pleasing when it isn’t present. The other year, for example, I revisited a post I wrote in 2004 and was pleased to find that a linked 2003 article by Nicholas ‘Aquarion’ Avenell is still alive at its original address! Contrast Jonathan Ames, the author/columnist/screenwriter who created the Most Phallic Building Contest until as late as 2011 before eventually letting his site and blog lapse and fall off the Internet. It takes effort to keep Web content alive, but it’s worth more effort than it’s sometimes given.

Anyway: a shot tower in Bristol – a part of the UK with a long history of leadworking – was among the latecomer entrants to the competition, and seeing this curious building reminded me about something I’d read, once, about the manufacture of lead shot. The idea (invented in Bristol by a plumber called William Watts) is that you pour molten lead through a sieve at the top of a tower, let surface tension pull it into spherical drops as it falls, and eventually catch it in a cold water bath to finish solidifying it. I’d seen an animation of the process, but I’d never seen a video of it, so I went about finding one.

Cross-section animation showing lead shot being poured into a sieve, separating into pellets, and falling into a water bath. — The animation I saw might have been this one, or perhaps one that wasn’t so obviously-made-in-MS-Paint.

British Pathé‘s YouTube Channel provided me with this 1950 film, and if you follow only one hyperlink from this article, let it be this one! It’s a well-shot (pun intended, but there’s a worse pun in the video!), and while I needed to translate all of the references to “hundredweights” and “Fahrenheit” to measurements that I can actually understand, it’s thoroughly informative.

But there’s a problem with that video: it’s been badly cut from whatever reel it was originally found on, and from about 1 minute and 38 seconds in it switches to what is clearly a very different film! A mother is seen shepherding her young daughter off to bed, and a voiceover says:

Bedtime has a habit of coming round regularly every night. But for all good parents responsibility doesn’t end there. It’s just the beginning of an evening vigil, ears attuned to cries and moans and things that go bump in the night. But there’s no reason why those ears shouldn’t be your neighbours ears, on occasion.

Black & white framegrab showing a woman following her child, wearing pyjamas, towards a staircase up. — “Off to bed, you little monster. And no watching TikTok when you should be trying to sleep!”

Now my interest’s piqued. What was this short film going to be about, and where could I find it? There’s no obvious link; YouTube doesn’t even make it easy to find the video uploaded “next” by a given channel. I manipulated some search filters on British Pathé’s site until I eventually hit upon the right combination of magic words and found a clip called Radio Baby Sitter. It starts off exactly where the misplaced prior clip cut out, and tells the story of “Mr. and Mrs. David Hurst, Green Lane, Coventry”, who put a microphone by their daughter’s bed and ran a wire through the wall to their neighbours’ radio’s speaker so they can babysit without coming over for the whole evening.

It’s a baby monitor, although not strictly a radio one as the title implies (it uses a signal wire!), nor is it groundbreakingly innovative: the first baby monitor predates it by over a decade, and it actually did use radiowaves! Still, it’s a fun watch, complete with its contemporary fashion, technology, and social structures. Here’s the full thing, re-merged for your convenience:

Wait, what was I trying to do when I started, again? What was I even talking about…

It’s harder than it used to be

It used to be easier than this to get lost on the Web, and sometimes I miss that.

Obviously if you go back far enough this is true. Back when search engines were much weaker and Internet content was much less homogeneous and more distributed, we used to engage in this kind of meandering walk all the time: we called it “surfing” the Web. Second-generation Web browsers even had names, pretty often, evocative of this kind of experience: Mosaic, WebExplorer, Navigator, Internet Explorer, IBrowse. As people started to engage in the noble pursuit of creating content for the Web they cross-linked their sources, their friends, their affiliations (remember webrings? here’s a reminder; they’re not quite as dead as you think!), your favourite sites etc. You’d follow links to other pages, then follow their links to others still, and so on in that fashion. If you went round the circles enough times you’d start seeing all those invariably-blue hyperlinks turn purple and know you’d found your way home.

Screenshot showing Netscape Communicator running on Windows 98, showing Dan's vanity page circa 1999. — Some parts of the Web are perhaps *best* forgotten, though?

But even after that era, as search engines started to become a reliable and powerful way to navigate the wealth of content on the growing Web, links still dominated our exploration. Following a link from a resource that was linked to by somebody you know carried the weight of a “web of trust”, and you’d quickly come to learn whose links were consistently valuable and on what subjects. They also provided a sense of community and interconnectivity that paralleled the organic, chaotic networks of acquaintances people form out in the real world.

In recent times, that interpersonal connectivity has, for many, been filled by social networks (let’s ignore their failings in this regard for now). But linking to resources “outside” of the big social media silos is hard. These advertisement-funded services work hard to discourage or monetise activity that takes you off their platform, even at the expense of their users. Instagram limits the number of external links by profile; many social networks push for resharing of summaries of content or embedding content from other sources, discouraging engagement with the wider Web, Facebook and Twitter both run external links through a linkwrapper (which sometimes breaks); most large social networks make linking to the profiles of other users of the same social network much easier than to users anywhere else; and so on.

The net result is that Internet users use fewer different websites today than they did 20 years ago, and spend most of their “Web” time in app versions of websites (which often provide a better experience only because site owners strategically make it so to increase their lock-in and data harvesting potential). Truly exploring the Web now requires extra effort, like exercising an underused muscle. And if you begin and end your Web experience on just one to three services, that just feels kind of… sad, to me. Wasted potential.

A woman reading a map. Photo by Leah Kelly. — I suppose nowadays we don’t get lost as often outside of the Internet, either. Photo by Leah Kelly.

It sounds like I’m being nostalgic for a less-sophisticated time on the Web (that would certainly be in character!). A time before we’d fully-refined the technology that would come to connect us in an instant to the answers we wanted. But that’s not exactly what I’m pining for. Instead, what I miss is something we lost along the way, on that journey: a Web that was more fun-and-weird, more interpersonal, more diverse. More Geocities, less Facebook; there’s a surprising thing to find myself saying.

Somewhere along the way, we ended up with the Web we asked for, but it wasn’t the Web we wanted.

Quickly Solving Jigidi Puzzles

26 August 2021 16:06 6

tl;dr? Just want instructions on how to solve Jigidi puzzles really fast with the help of your browser’s dev tools? Skip to that bit.

I don’t enjoy jigsaw puzzles

I enjoy geocaching. I don’t enjoy jigsaw puzzles. So mystery caches that require you to solve an online jigsaw puzzle in order to get the coordinates really don’t do it for me. When I’m geocaching I want to be outdoors exploring, not sitting at my computer gradually dragging pixels around!

A completed 1000-piece "Where's Wally?" jigsaw. — Don’t let anybody use my completion of this 1000-piece jigsaw puzzle over New Year as evidence that I’m lying and actually like jigsaws.

Many of these mystery caches use Jigidi to host these jigsaw puzzles. An earlier version of Jigidi was auto-solvable with a userscript, but the service has continued to be developed and evolve and the current version works quite hard to make it hard for simple scripts to solve. For example, it uses a WebSocket connection to telegraph back to the server how pieces are moved around and connected to one another and the server only releases the secret “you’ve solved it” message after it detects that the pieces have been arranged in the appropriate relative configuration.

A nine-piece jigsaw puzzle with the pieces numbered 1 through 9; only the ninth piece is detached. — I made a simple Jigidi puzzle for demonstration purposes. Do you think you can manage a nine-piece jigsaw?

If there’s one thing I enjoy more than jigsaw puzzles – and as previously established there are about a billion things I enjoy more than jigsaw puzzles – it’s reverse-engineering a computer system to exploit its weaknesses. So I took a dive into Jigidi’s client-side source code. Here’s what it does:

Get from the server the completed image and the dimensions (number of pieces).
Cut the image up into the appropriate number of pieces.
Shuffle the pieces.
Establish a WebSocket connection to keep the server up-to-date with the relative position of the pieces.
Start the game: the player can drag-and-drop pieces and if two adjacent pieces can be connected they lock together. Both pieces have to be mostly-visible (not buried under other pieces), presumably to prevent players from just making a stack and then holding a piece against each edge of it to “fish” for its adjacent partners.

Javascirpt code where the truthiness of this.j affects whether or not the pieces are shuffled. — I spent some time tracing call stacks to find this line… only to discover that it’s one of only four lines to actually contain the word “shuffle” and I could have just searched for it…

Looking at that process, there’s an obvious weak point – the shuffling (point 3) happens client-side, and before the WebSocket sync begins. We could override the shuffling function to lay the pieces out in a grid, but we’d still have to click each of them in turn to trigger the connection. Or we could skip the shuffling entirely and just leave the pieces in their default positions.

An unshuffled stack of pieces from the nine-piece jigsaw. Piece number nine is on top of the stack. — An unshuffled jigsaw appears as a stack, as if each piece from left to right and then top to bottom were placed one at a time into a pile.

And what are the default positions? It’s a stack with the bottom-right jigsaw piece on the top, the piece to the left of it below it, then the piece to the left of that and son on through the first row… then the rightmost piece from the second-to-bottom row, then the piece to the left of that, and so on.

That’s… a pretty convenient order if you want to solve a jigsaw. All you have to do is drag the top piece to the right to join it to the piece below that. Then move those two to the right to join to the piece below them. And so on through the bottom row before moving back – like a typewriter’s carriage return – to collect the second-to-bottom row and so on.

How can I do this?

If you’d like to cheat at Jigidi jigsaws, this approach works as of the time of writing. I used Firefox, but the same basic approach should work with virtually any modern desktop web browser.

Go to a Jigidi jigsaw in your web browser.
Pop up your browser’s developer tools (F12, usually) and switch to the Debugger tab. Open the file game/js/release.js and uncompress it by pressing the {} button, if necessary.
Find the line where the code considers shuffling; right now for me it’s like 3671 and looks like this:
return this.j ? (V.info('board-data-bytes already exists, no need to send SHUFFLE'), Promise.resolve(this.j)) : new Promise(function (d, e) {

I spent some time tracing call stacks to find this line… only to discover that it’s one of only four lines to actually contain the word “shuffle” and I could have just searched for it…
Set a breakpoint on that line by clicking its line number.
Restart the puzzle by clicking the restart button to the right of the timer. The puzzle will reload but then stop with a “Paused on breakpoint” message. At this point the application is considering whether or not to shuffle the pieces, which normally depends on whether you’ve started the puzzle for the first time or you’re continuing a saved puzzle from where you left off.
In the developer tools, switch to the Console tab.
Type: this.j = true (this ensures that the ternary operation we set the breakpoint on will resolve to the true condition, i.e. not shuffle the pieces).
Press the play button to continue running the code from the breakpoint. You can now close the developer tools if you like.
Solve the puzzle as described/shown above, by moving the top piece on the stack slightly to the right, repeatedly, and then down and left at the end of each full row.

Heatmapping my Movements

22 August 2021 12:50 4

As I mentioned last year, for several years I’ve collected pretty complete historic location data from GPSr devices I carry with me everywhere, which I collate in a personal μlogger server.

Going back further, I’ve got somewhat-spotty data going back a decade, thanks mostly to the fact that I didn’t get around to opting-out of Google’s location tracking until only a few years ago (this data is now also housed in μlogger). More-recently, I now also get tracklogs from my smartwatch, so I’m managing to collate more personal location data than ever before.

Inspired perhaps at least a little by Aaron Parecki, I thought I’d try to do something cool with it.

Heatmapping my movements

The last year

Heatmap showing Dan's movements around Oxford since moving house in 2020. There's a strong cluster around Stanton Harcourt with heavy tendrils around Witney and Eynsham and along the A40 to Summertown, and lighter tendrils around North and Central Oxford. — My movements over the last year have been relatively local, but there are some interesting hotspots and common routes.

What you’re looking at is a heatmap showing my location over the last year or so since I moved to The Green. Between the pandemic and switching a few months prior to a job that I do almost-entirely at home there’s not a lot of travel showing, but there’s some. Points of interest include:

The blob around my house, plus some of the most common routes I take to e.g. walk or cycle the children to school.
A handful of my favourite local walking and cycling routes, some of which stand out very well: e.g. the “loop” just below the big blob represents a walk around the lake at Dix Pit; the blob on its right is the Devils Quoits, a stone circle and henge that I thought were sufficiently interesting that I made a virtual geocache out of them.
The most common highways I spend time on: two roads into Witney, the road into and around Eynsham, and routes to places in Woodstock and North Oxford where the kids have often had classes/activities.
I’ve unsurprisingly spent very little time in Oxford City Centre, but when I have it’s most often been at the Westgate Shopping Centre, on the roof of which is one of the kids’ favourite restaurants (and which we’ve been able to go to again as Covid restrictions have lifted, not least thanks to their outdoor seating!).

One to eight years ago

Let’s go back to the 7 years prior, when I lived in Kidlington. This paints a different picture:

Heatmap showing Dan's movements around Kidlington, including a lot of time in the village and in Oxford City Centre, as well as hotspots at the hospital, parks, swimming pools, and places that Dan used to volunteer. Individual expeditions can also be identified. — For the seven years I lived in Kidlington I moved around a lot more than I have since: each hotspot tells a story, and some tell a few.

This heatmap highlights some of the ways in which my life was quite different. For example:

Most of my time was spent in my village, but it was a lot larger than the hamlet I live in now and this shows in the size of my local “blob”. It’s also possible to pick out common destinations like the kids’ nursery and (later) school, the parks, and the routes to e.g. ballet classes, music classes, and other kid-focussed hotspots.
I worked at the Bodleian from early 2011 until late in 2019, and so I spent a lot of time in Oxford City Centre and cycling up and down the roads connecting my home to my workplace: Banbury Road glows the brightest, but I spent some time on Woodstock Road too.
For some of this period I still volunteered with Samaritans in Oxford, and their branch – among other volunteering hotspots – show up among my movements. Even without zooming in it’s also possible to make out individual venues I visited: pubs, a cinema, woodland and riverside walks, swimming pools etc.
Less-happily, it’s also obvious from the map that I spent a significant amount of time at the John Radcliffe Hospital, an unpleasant reminder of some challenging times from that chapter of our lives.
The data’s visibly “spottier” here, mostly because I built the heatmap only out of the spatial data over the time period, and not over the full tracklogs (i.e. the map it doesn’t concern itself with the movement between two sampled points, even where that movement is very-guessable), and some of the data comes from less-frequently-sampled sources like Google.

Eight to ten years ago

Let’s go back further:

Heatmap showing Dan's movements around Oxford during the period he lived in Kennington. Again, it's dominated by time at home, in the city centre, and commuting between the two. — Back when I lived in Kennington I moved around a lot less than I would come to later on (although again, the spottiness of the data makes that look more-significant than it is).

Before 2011, and before we bought our first house, I spent a couple of years living in Kennington, to the South of Oxford. Looking at this heatmap, you’ll see:

I travelled a lot less. At the time, I didn’t have easy access to a car and – not having started my counselling qualification yet – I didn’t even rent one to drive around very often. You can see my commute up the cyclepath through Hinksey into the City Centre, and you can even make out the outline of Oxford’s Covered Market (where I’d often take my lunch) and a building in Osney Mead where I’d often deliver training courses.
Sometimes I’d commute along Abingdon Road, for a change; it’s a thinner line.
My volunteering at Samaritans stands out more-clearly, as do specific venues inside Oxford: bars, theatres, and cinemas – it’s the kind of heatmap that screams “this person doesn’t have kids; they can do whatever they like!”

Every map tells a story

I really love maps, and I love the fact that these heatmaps are capable of painting a picture of me and what my life was like in each of these three distinct chapters of my life over the last decade. I also really love that I’m able to collect and use all of the personal data that makes this possible, because it’s also proven useful in answering questions like “How many times did I visit Preston in 2012?”, “Where was this photo taken?”, or “What was the name of that place we had lunch when we got lost during our holiday in Devon?”.

There’s so much value in personal geodata (that’s why unscrupulous companies will try so hard to steal it from you!), but sometimes all you want to do is use it to draw pretty heatmaps. And that’s cool, too.

How these maps were generated

I have a μlogger instance with the relevant positional data in. I’ve automated my process, but the essence of it if you’d like to try it yourself is as follows:

First, write some SQL to extract all of the position data you need. I round off the latitude and longitude to 5 decimal places to help “cluster” dots for frequency-summing, and I raise the frequency to the power of 3 to help make a clear gradient in my heatmap by making hotspots exponentially-brighter the more popular they are:

SELECT ROUND(latitude, 5) lat, ROUND(longitude, 5) lng, POWER(COUNT(*), 3) `count`
FROM positions
WHERE `time` BETWEEN '2020-06-22' AND '2021-08-22'
GROUP BY ROUND(latitude, 5), ROUND(longitude, 5)

This data needs converting to JSON. I was using Ruby’s mysql2 gem to fetch the data, so I only needed a .to_json call to do the conversion – like this:

db = Mysql2::Client.new(host: ENV['DB_HOST'], username: ENV['DB_USERNAME'], password: ENV['DB_PASSWORD'], database: ENV['DB_DATABASE'])
db.query(sql).to_a.to_json

Approximately following this guide and leveraging my Mapbox subscription for the base map, I then just needed to include leaflet.js, heatmap.js, and leaflet-heatmap.js before writing some JavaScript code like this:

body.innerHTML = '<div id="map"></div>';
let map = L.map('map').setView([51.76, -1.40], 10);
// add the base layer to the map
L.tileLayer('https://api.mapbox.com/styles/v1/{id}/tiles/{z}/{x}/{y}?access_token={accessToken}', {
  maxZoom: 18,
  id: 'itsdanq/ckslkmiid8q7j17ocziio7t46', // this is the style I defined for my map, using Mapbox
  tileSize: 512,
  zoomOffset: -1,
  accessToken: '...' // put your access token here if you need one!
}).addTo(map);
// fetch the heatmap JSON and render the heatmap
fetch('heat.json').then(r=>r.json()).then(json=>{
  let heatmapLayer = new HeatmapOverlay({
    "radius": parseFloat(document.querySelector('#radius').value),
    "scaleRadius": true,
    "useLocalExtrema": true,
  });
  heatmapLayer.setData({ data: json });
  heatmapLayer.addTo(map);
});

That’s basically all there is to it!

Holidays in the Age of COVID

15 August 2021 22:59 0

We’ve missed out on or delayed a number of trips and holidays over the last year and a half for, you know, pandemic-related reasons. So this summer, in addition to our trip to Lichfield, we arranged a series of back-to-back expeditions.

1. Alton Towers

The first leg of our holiday saw us spend a long weekend at Alton Towers, staying over at one of their themed hotels in between days at the water park and theme park:

Children playing outside the CBeebies Land Hotel — The CBeebies Land hotel is… whimsical.

A muppet assists with hotel checkin. — Yes, there’s a puppeteer (somewhere) in that cabinet processing checkins.

Kids in a decorated elevator. — Even the elevators play tunes and put on a light show with every journey.

A rabbit... playing a saxophone? — It took me a while to see this rabbit as playing music, rather than, y’know… vomiting.

The Windmill restaurant at the CBeebies Land Hotel — The whimsy continues in the theming of the restaurant. Yes, that windmill turns.

The kids on an enormous chair. — Technically, this is part of one of the other (similarly whimsical) hotels on the site, but nobody seemed to mind our exploring.

Ruth, JTA and the kids eat dinner. — On at least one occasion we ate dinner in “The Library”, which turns out (disappointingly) not to be a library but a room with pictures of books on the wall.

Children watching a show in the hotel. — The hotel puts on a series of show somewhat reminiscent of an “upscale” Pontins.

Bing the rabbit at the hotel. — Bing the rabbit made me glad that the other end of this hotel room had a bar.

JTA and the kids relaxing in our Octonauts-themed hotel room. — The (Octonauts) theming of our hotel room even extended as far as the phone, TVs, and that notice they put up about towel washing. Neat.

Mini-golf game in progress. — Hole in… seven?

Dan, Ruth, JTA and the kids at the water park. — For obvious reasons, we don’t have photos inside the water park. Ignore the sign, we asked permission before taking this one!

The kids by the pond between the hotels. — Befriending wildfowl is what people with kids come to theme parks for, right?

Kids at the "frog" fountains near the entrance to Alton Towers' theme park. — Yay! Frog fountains!

Riding the In The Night Garden boat ride. — The In The Night Garden Boat Ride was a particular favourite.

Ubercorn poses for a photo with a fan. — Social distancing was imposed reasonably wall, all things considered, although (as you’d expect) many fairgoers were less-disciplined than we’d have liked.

The Octonauts rollercoaster. — We spent a lot of our first day in the theme park in CBeebies Land, but we only had the patience to queue for the Octonauts rollercoaster the once.

Duggee at Alton Towers — CBeebies Land itself had a variety of shows with different characters.

The kids learn from a Mr. Tumble screen. — Makaton for “friend” is just cute. Thanks, Mr. Tumble.

Dan and JTA wear masks indoors. — On these, among the hottest days of the year, we got quite sweaty inside our masks.

Charlie and Lola photo opportunity. — I’m not sure we optimised this photo opportunity for height.

Imprisoned nose-picker. — An unrepentant nose-picker gets jail time in Mutinty Bay.

The kids drive through the Postman Pat ride. — The Postman Pat ride designer had the foresight to provide two steering wheels in case the participants weren’t good at sharing… but still provided an odd number of buttons to use to “deliver parcels” at key points during the ride.

Congo River Rapids ride.. — Later in the first day and into the second day we stepped up to more-exciting family rides, like the Congo River Rapids, and even a few thrill rides that interested only a subset of our party.

Ride photo from the Congo River Rapids — The Congo River Rapids had a tediously long queue (not least because no groups, however small, were allowed to share a boat), but at least we all got to take part together as a family.

Ride photo from the Runaway Mind Train. — The kids had to make a solemn promise to stop their bickering before they were allowed to sit together for this ride.

Ride photo from Duel — Not everybody was equally-enthusiastic about haunted house shoot-’em-up ride Duel.

2. Darwin Forest

The second leg of our holiday took us to a log cabin in the Darwin Forest Country Park for a week:

Feeding a sheep. — We punctuated our journey from Alton Towers with a trip to Chatsworth House to feed some livestock.

Rope climbing. — Chatsworth House also has a spectacular adventure playground.

Kid carrying huge stick. — “I found a stick!” “That’s… half a tree!”

Log cabin in the Landal Darwin Forest. — Eventually we reached our cabin, unpacked… and jumped into the hot tub!

Lounging on the sofa with a CBeebie toy. — For the duration of the week we instituted “holiday mornings”, permitting the children to get up by themselves, assemble their own breakfast, and watch broadcast TV… so long as they did so without disturbing the adults. It worked pretty well.

Ruth hops across balancing logs in the Darwin Forest. — The forest trails are full of obstacles fun for children… and adults.

Kids different ways up. — No, THIS way up!

Dan on a climbing wall. — Further along the trail, there’s a traverse wall.

Fairy houses in a wood. — Off the trails, all kinds of curious sculptures – like these fairy houses – can be found.

Running around on a football pitch. — We forgot to bring a football, but we played lots of imaginary sports.

Soft play dome. — The on-site soft play centre operated at minimal capacity, which felt about right.

Soft play slides. — A soft play centre with a low population means you’re often alone on the racing slides…

Dan and the kids on a racing slide. — …that is, unless you bring your own racers to compete with!

Mini-golf in the Darwin Forest. — The second mini-golf course of our holiday was deceptively harder than the first thanks to awkwardly-shaped obstacles that reflected the ball out at terrible angles.

Dan, Ruth, JTA and the kids outside St. Elphin's Park Retirement Village. — Ruth’s old secondary school is now an old folks home and an attached fancy tea room, so – as we were in the vicinity – we had to go visit!

Stirring tea. — Remarkably, the kids (for once) showed impeccable table manners.

Wiping with a napkin. — I’m almost embarrassed to say that this, one of the best photos I’ve ever taken, was snapped accidentally when a 4-year-old reached over and touched my camera.

Dan, Ruth, JTA and the kids outside the Devil's Arse. — What else does one do in the Derbyshire Dales? That’s right: go down caves!

The kids visit a restored ropemaker's hut in the Devil's Arse. — Staff at the Devil’s Arse were amazing and even took the kids on their own personal tour of a ropemaker’s hut (not part of the routine tour).

Ropemaking demonstration. — We also got to bring home a length of rope that the kids helped make.

Family down a cave. — It turns out to be hard to take a good photo down a cave. Can’t think why.

Dan, Ruth and the kids in a boat down a cave. — Our second spelunking expedition took us into Speedwell Cavern by boat.

Dan, Ruth, JTA and the kids above the "bottomless pit" in Speedwell Cavern. — The bottomless pit, behind us, turns out to be less-bottomless than advertised.

Ice cream. — When it wasn’t raining, it was hot. Either way, we were on holiday, so ice cream was in order.

The kids asleep. — The kids shared a bedroom for pretty-much the first time since we moved house last year, and routinely sat up late reading stories to one another until they zonked out mid-book.

Dan, geocaching. — Naturally, I took one of our “chill out and rest” days as an excuse for a nice long hike and a geocaching expedition to the Sydnope Valley.

Dan, Ruth, JTA and the kids at the Crazy Cat Lady escape room. — We also tried an local escape room and it was spectacularly well-designed (and amazingly family-friendly).

Dan drinking from a champagne flute. — Oh, and there might have been a modicum of drinking, both in and out of the hot tub.

3. Preston

Kicking off the second week of our holiday, we crossed the Pennines to Preston to hang out with my family (with the exception of JTA, who had work to do back in Oxfordshire that he needed to return to):

Boy eating a crumpet alongside a bowl of fruit. — Our resident 4-year-old foodie claims that no crumpets are as good as Nanna Doreen’s “giant” ones.

Boy lying on Dan, waving, eating an ice lolly. — Sometimes you just need an ice lolly and a bean bag to chill on. If you can’t find a bean bag, use Dan.

Children running out of a shed. — Now that she doesn’t have any cats, dogs, or chickens, my mother’s old sheds have been converted into playhouses.

Ruth provides tickling. — Who needs a tickle? You need a tickle.

A girl with felt-tip whiskers drawn on to look like a cat. — Fortunately, the felt-tips she found were of the washable variety.

Dan takes a picture of Ruth taking a picture of the kids. — Why yes, this is a picture of me taking a picture of Ruth taking a picture.

A woman and a girl press their faces together and rub noses. — I’m not sure what this pose was MEANT to achieve…

A woman and a girl; the girl is face-painted as a cat, the woman has bit of the face paint rubbed off onto her. — …but what it ACTUALLY achieved was my mother getting ink splotches on her face.

Two adults and a child using mobile phones and tablets. — Syncing everything up in anticipation of a Pokewalk.

A woman and a girl outdoors looking at their mobile devices. — “Which way to the Pokestop?” (I’m guessing. I don’t understand Pokemon Go.)

A woman and a girl on a cycleway, holding hands, between trees and weeds. — “How far to the Pokegym?” (I still don’t know anything about Pokemon Go. Don’t flame me.)

Sarah and Ruth pull strange pouty faces together. — I feel like every time I visit my family I acquire a larger number of photos that I can’t begin to explain.

My sisters wrestle(?) on the floor. — Like this one. Is this some kind of game? What are the rules? Who’s winning? I just don’t know.

Becky lies on the floor, surrounded by/covered in children. — Is this the same game? Are the children all on the same team (against Becky) or not? I have so many unanswered questions.

Dan and Sarah hold their heads with their hands, at a strange angle. — And this one. What are Sarah and I doing? Simultaneously cracking our necks, perhaps?

Hedgehog on a garden path. — The kids put food out for the hedgehogs and attracted a big one.

A boy, holding a GPS receiver, runs across a park lawn. — Now THIS kind of GPS-based sport I can get behind. Lead on to the cache!

Two children hold a geocache. — The kids were less impressed by this geocache than I was. And I wasn’t THAT impressed.

A girl does sit-ups in an outdoor gym. — Not content with sprinting around the 400m track wearing her wellies, our 7-year-old then proceeded to dominate on the park gym.

A boy on a cross-trainer. — If you pedal one of these things hard enough, does it take off?

Sarah helps a child on a zipline. — Wheeee!

A boy enjoys an ice cream at a park cafe. — So. Many. Sprinkles.

A boy runs in the distance. — How do they find so much energy?

A girl runs with a small dog on a lead. — We borrowed a dog from a family friend. If our 7-year-old had her way, we wouldn’t have given it back.

Jemma blows out candles on a birthday cake. — It was Jemma’s birthday, apparently, so we marked it at a family barbecue.

Ruth photographs a boy on a pontoon. — Nearby Brockholes nature reserve provided a wonderful outing.

Ruth photographs her 4-year-old as he stands on a pontoon dock. — This one’s the less-accident-prone of our children. Otherwise he’d probably already be in the water, somehow.

Ducks approach a girl on a pontoon bridge. — Our 7-year-old likes to befriend ducks wherever she goes.

Children run away into a meadow. — Again with the boundless energy as the kids disappear into one of Brockholes’ meadows.

A boy runs away from two adults. — That boy loves a big open space to run in, for sure.

A girl shouts with excitement as she crosses a wobbly bridge. — Brockholes’ adventure play area is pretty exciting too.

A boy concentrates hard as he crosses a wobbly bridge. — But crossing a wobbly bridge isn’t the easiest thing when you’re little.

A boy climbs a two-rope crossing, balancing on the bottom rope while holding the top rope. — Luckily our children are both pretty fearless and adventurous and will give pretty much anything a go.

A boy shouts from the top of a grassy knoll near a car park. — It’s hard to tell, but he’s not shouting in distress here, but in joy.

Children petting an English Longhorn cattle. — I believe this is an English Longhorn, a traditional draft animal of North-West England in centuries past.

Two adults and two children pose with a wooden sculpture of a doe lying down. — I made several attempts to get my mother, my sister Sarah, and both our kids into a single frame in which they were all looking at the camera and none of them were blinking. I failed, but this was the closest I managed.

4. Forest of Bowland

Ruth and I then left the kids with my mother and sisters for a few days to take an “anniversary mini-break” of glamping in the gorgeous Forest of Bowland:

Dan and Ruth outside a wooden "pod" glamping structure. — This caravan-sized wooden hut became our delightful little home for a few days.

A river snakes through a lightly-wooded valley. — Here, at the edge of the Forest, the Bier Beck snakes lazily towards the River Ribble.

Dan strokes a large brown horse wearing a red jacket. — The farm on which our pod was situated kept horses.

A horse wearing a zebra-print jacket. — I just loved the fact that this horse had a zebra-print jacket.

Ruth and Dan at the trig point at the summit of Pendle Hill. — One morning, we took a scorching hike up Pendle Hill. Didn’t see any witches, but also didn’t receive any mission from God, so I guess all’s well.

Steve Taylor wearing a bathtub and climbing Pendle Hill. — We DID get to meet Steve Taylor, though, who’s repeatedly climbing the hill, carrying a bathtub, until he’s ascended to the consecutive height of Mount Everest. He’s raising money for the Cystic Fibrosis Trust, the hero.

Dan and Ruth enjoy cocktails in a beer garden. — That pubs expanded their outdoor seating provision to handle social distancing works wonderfully when you get to sit in a gorgeous but quiet beer garden.

Ruth plays a pipe organ. — During a self-guided tour of the nearest village we bumped into the vicar who showed us his church’s pipe organ. Ruth tried to explain to me how (and why) this particular pipe organ was unusual and cool, but I’m not sure I’ll ever “get it” as well as she does.

Dan, sweaty, at the top of a rise. — I found a second wind on a walk around Gisburn Forest and jogged up to the trail’s highest point, leaving Ruth far behind the sweaty mess ahead of her.

Orange-red river water beneath Dan's feet. — I’m not aware of any iron mining operations anywhere near this forest, but there must be iron oxide in the rocks to turn this stream so red.

Ruth and Dan eat (fake) duck pancakes outside their lodge. — Each evening we’d record a quick selfie video to say hi to the kids. Sometimes (with some help!) they’d send one back.

A fire pit rages and sparks. — A particular treat was stargazing by the fire pit on an evening.

Dan and Ruth with the sea in the background. — As we checked-out of our accommodation, Ruth suggested we see the sea (we’re a long way from it, normally), so we diverted via the coast.

(If you’re interested in Steve Taylor’s bathtub-carrying virtual-Everest expedition, here’s his Facebook page and JustGiving profile.)

5. Meanwhile, in Preston

The children, back in Preston, were apparently having a whale of a time:

Children pet a starfish — Starfish-petting.

Starfish with childrens' hands nearby. — (Yes, actual starfish.)

Boy in the (bones of the) mouth of a shark. — That’ll be Seaworld, of course.

Underwater transparent tube with fish outside and children inside. — Tube. Tube tube tube.

Child peering into a fishtank using a dome underneath it. — Do you think the fish think that humans are the exhibit?

Children in a VR motion simulator. — Both kids play with VR at home and are way less susceptible to VR-nausea than I am (even with all the practice I’ve had!).

Girl in front of a LEGO mosaic of butterfly wings. — Legoland Manchester. (Did you even know there was a Legoland in Manchester? I didn’t.)

Kids with Lego Batman. — The 4-year-old took “meeting” Lego Batman way more-seriously than the 7-year-old, I think.

Child in a lego space suit. — That’s one small step for…. OW I STOOD ON A LEGO BRICK!

Child with a contributory LEGO mosaic. — This brick was her contribution, I guess?

Boy balancing on a beam at a park. — I think this must be Avenham Park in Preston.

A boy doing finger painting. — Many of his paintings start out as beautiful coloured stripes and end up as brown handprints. I can’t imagine how.

Children play shop: a boy operates a checkout and a girl buys shopping. — “Do you do contactless?”

Children gardening. — I gather the children even got a little gardening done… or at least, grazed on the entirety of my mother’s herb garden.

6. Suddenly, A Ping

The plan from this point was simple: Ruth and I would return to Preston for a few days, hang out with my family some more, and eventually make a leisurely return to Oxfordshire. But it wasn’t to be…

Screenshot from the NHS Covid App: "You need to self-isolate." — Well that’s not the kind of message you want to get from your phone.

I got a “ping”. What that means is that my phone was in close proximity to somebody else’s phone on 29 August and that other person subsequently tested positive for COVID-19.

My risk from this contact is exceptionally low. There’s only one place that my phone was in close proximity to the phone of anybody else outside of my immediate family, that day, and it’s when I left it in a locker at the swimming pool near our cabin in the Darwin Forest. Also, of course, I’d been double-jabbed for a month and a half and I’m more-cautious than most about contact, distance, mask usage etc. But my family are, for their own (good) reasons, more-cautious still, so self-isolating at Preston didn’t look like a possibility for us.

Ruth and Dan in a car, in a car park. — Ruth and I went directly to a drive-through PCR testing facility.

As soon as I got the notification we redirected to the nearest testing facility and both got swabs done. 8 days after possible exposure we ought to have a detectable viral load, if we’ve been infected. But, of course, the tests take a day or so to process, so we still needed to do a socially-distanced pickup of the kids and all their stuff from Preston and turn tail for Oxfordshire immediately, cutting our trip short.

The results would turn up negative, and subsequent tests would confirm that the “ping” was a false positive. And in an ironic twist, heading straight home actually put us closer to an actual COVID case as Ruth’s brother Owen turned out to have contracted the bug at almost exactly the same time and had, while we’d been travelling down the motorway, been working on isolating himself in an annex of the “North wing” of our house for the duration of his quarantine.

Barricade with signs reading "Quarantine: Zombie Outbreak" — I set up a “yellow zone” between Owen’s quarantine area and the rest of the house into which we could throw supplies. And I figured I’d have fun with the signage.

7. Ruth & JTA go to Berwick

Thanks to negative tests and quick action in quarantining Owen, Ruth and JTA were still able to undertake the next part of this three-week holiday period and take their anniversary break (which technically should be later in the year, but who knows what the situation will be by then?) to Berwick-upon-Tweed. That’s their story to tell, if they want to, but the kids and I had fun in their absence:

Dan and his 4-year-old wearing silly hats. — Silly Hat Morning!

Dan and the kids on a swan boat. — Swan boat ride. (I had to do all the pedalling, but the kids were good at shouting orders and threatening to fire upon or board the other boats!)

Monkey on roof of car. — A trip to Woburn Safari Park isn’t complete without a photo of a monkey on the roof of a car.

A child watches a giant tortoise try to open a gate. — That giant tortoise was determined that it could open the (latched) gate if only it pushed hard enough.

Children petting a goat. — Kids love a kid (goat).

Girl eats half a sausage roll. — Nothing beats a picnic lunch right as everybody else rushes to queue for food in the heat.

A boy pours a juice drink onto some candy floss. — Ever the culinary experimenter, our 4-year-old tries pouring a Fruit Shoot onto some candy floss.

A boy holds a GPS receiver as he walks down a lane. — And of course, some geocaching might have happened.

Children look around a field. — “The cache is this way!” “No, it’s THIS way.” “Kids! It’s still 200 metres away, keep walking!”

Children hold a toy plane with "geocache airlines" written on it. — GC98N1P – whose cache container is this aeroplane and is hidden in a “crash site”! – is one of my favourite local caches.

A girl plays a kazoo. — I gave the 7-year-old a kazoo. #parentingmistakes

Children painting — We start with nice colourful lines…

Boy with painted brown hands. — …and end up with brown hands! It’s part of his artistic process, I guess.

Children in pyjamas at a breakfast table greet Robin outside the patio doors. — We enjoyed a socially-distanced visit from Robin one breakfast time.

A girl wearing headphones programs in Scrach. On her screen a witch is hunting for a broomstick. — We also extended our practice in programming with Scratch.

Dan holding the Wild Wolf 3 geocache. — I found an opportunity to retrieve a much-loved but no-longer-sustainable geocache of my own. Look at this monster!

Butter pies in an oven. — I tapped into my Lancashire heritage and had a go at making “butter pie”, a regional dish distinct to (pretty much) just Preston and Chorley.

A butter pie with heart-shaped pastry decorations. — This one turned out pretty well, but I’ve still got ideas about how I’ll improve for the next one.

8. Reunited again

Finally, Ruth and JTA returned from their mini-break and we got to do a few things together as a family again before our extended holiday drew to a close:

A hug with Ruth. — The children were glad to see mummy and daddy return.

JTA greets the kids. — Also, to explain everything that’d they been up to. (Possibly just as a ruse to keep from being sent to bed for a moment longer!)

A girl looks lost in a maize maze. — I specifically said, “Look like you’re NOT completely lost in this maize maze,” I swear.

Children playing crazy golf. — I feel like I’ve played a lot of crazy golf these last few weeks. This course was perhaps the second-craziest.

A girl digs in a tabletop sandpit. — There was some kind of puzzle to solve in a maze. Then you had to dig in the sand to find a token of the right colour and put it in a box to show you’d solved it. I don’t know.

A boy dives onto a trampoline. — I’m not sure this angle of approach is going to end well.

A girl on a carousel horse called "Annie". — Nice horse.

Two children crawl through a den of sticks. — Den-building? Or poster art for some gritty new Netflix series?

A girl hangs upside-down in a soft play area. — “I’m being a bat.” She hung there for some time, greeting other children as they entered her “cave”.

Dan and Ruth with their 4-year-old. — “Smile!” “Wait… what’s in your mou-?” [click]

A girl picks gooseberries. — Gooseberry-picking.

Ruth and the children pick gooseberries. — The gooseberries later made a great reduction to go on our Sunday brunch pancakes. Except for all the ones the kids ate before then.

A girl looks at a strawberry. — Choosing exactly the right strawberry requires significant mental effort.

A boy inspects a strawberry on a vine. — You need to check the fruit from every side before picking it.

A girl pulls a face as she looks at a strawberry. — This one failed quality control.

People stand around in a garden at a barbecue. — We got to go to Liz & Simon’s barbecue party and it was awesome to catch up with everybody.

Dan plays with a small baby. — Plus there are a few new faces in our friend group who I hadn’t had a chance to meet before.

Children at a picnic table in a garden. — Making (and re-making) friends is so much easier as a child.

Adults eat barbecue food. — For us adults, though, sharing food and alcohol goes a long way.

A child in the smoke of a barbecue holds his hand over his eyes. — “It’s smokey over here.” “By the barbecue, yes.” “Why?”

Children play together; adults sit and talk in the background. — It’s amazing how a toy designed for somebody WAY younger than you is fascinating if it’s just different enough from one you have at home. Novelty wins.

Children watch as our 7-year-old plays Pokemon with Simon. — She came here to kick ass at Pokemon and eat your burgers. And you’re all out of burgers.

The kids pose with a fruit-topped cake. — The fruits we picked earlier in the week made a great addition to a cake.

Fairpoirt Convention's livestream plays on TV. — Of course, we were SUPPOSED to be at Fairport’s Cropredy Convention this weekend, until it was cancelled for the second year in a row. But we still enjoyed Fairport’s livestream mini-concert.

Children dancing. — We adults felt too old and/or self-conscious to dance in our living room, but the kids had no such limitations.

Ruth and the kids in a heap at the end of the virtual concert. — By the end of the virtual concert we were all ready to flump into bed.

A fully-assembled 'hamster heaven' cage. — We built an enclosure for a new pet we’re expecting in the coming week (the kids’ first pet; let’s see how that goes…).

Robin reverse-abseilling up a tree. — Robin (and Owen – now recovered but not featured in this picture) – were instrumental in helping us run some ropes over a high bough of one of our garden’s trees…

The kids play switch while they hang in a 'nest swing'. — …to facilitate the installation of a wonderful new ‘nest swing’ Ruth had bought the kids but that they’d not really been able to use until now.

9. Back to work?

Tomorrow I’m back at work, and after 23 days “off” I’m honestly not sure I remember what I do for a living any more. Something to do with the Internet, right? Maybe ecommerce?

I’m sure it’ll all come right back to me, at least by the time I’ve read through all the messages and notifications that doubtless await me (I’ve been especially good at the discipline, this break, of not looking at work notifications while I’ve been on holiday; I’m pretty proud of myself.)

But looking back, it’s been a hell of a three weeks. After a year and a half of being pretty-well confined to one place, doing a “grand tour” of so many destinations as a family and getting to do so many new and exciting things has made the break feel even longer than it was. It seems like it must have been months since I last had a Zoom meeting with a work colleague!

For now, though, it’s time to try to get the old brain back into work mode and get back to making the Web a better place!

Lichfield

19 July 2021 19:16 6

We took a family trip up to Lichfield this weekend. I don’t know if I can give a “review” of a city-break as a whole, but if I can: I give you five stars, Lichfield.

Dan in front of Lichfield Cathedral, early on Sunday morning. — It’s got a cathedral, which is quite pretty.

Maybe it’s just because we’ve none of us had a night away from The Green… pretty-much since we moved in, last year. But there was something magical about doing things reminiscent of the “old normal”.

Dan and the kids in a bed at a hotel. — “I’m so excited! We get to stay… *at a Premier Inn*!” At first I rolled my eyes at this joyous line from our 4-year-old (I mean… it’s just a Premier Inn…), but it did feel good to go somewhere and do something.

It’s not that like wasn’t plenty of mask-wearing and social distancing and hand sanitiser and everything that we’ve gotten used to now: there certainly was. The magic, though, came from getting to do an expedition further away from home than we’re used to. And, perhaps, with that happening to coincide with glorious weather and fun times.

A balloon artist wearing a unicorn on her head makes sculptures for children. — Socially-distanced balloon modelling turns out to work, not least because you can hand one of those long balloons to somebody without getting anywhere near them.

We spent an unimaginably hot summer’s day watching an outdoor interpretation of Peter and the Wolf, which each of the little ones has learned about in reasonable depth, at some point or another, as part of the (fantastic) “Monkey Music” classes of which they’re now both graduates.

Ruth and John sit on a picnic blanket in a painted circle; the maquee for the band is behind them. — So long as you weren’t staring at the painted circles on the grass – for corralling families apart from one another – you’d easily forget how unusual things are, right now.

And maybe it’s that they’ve been out-of-action for so long and are only just beginning to once again ramp up… or maybe I’ve just forgotten what the hospitality industry is like?… but man, we felt well-looked after.

From the staff at the hotel who despite the clear challenges of running their establishment under the necessary restrictions still went the extra mile to make the kids feel special to the restaurant we went to that pulled out all the stops to give us all a great evening, I basically came out of the thing with the impression of Lichfield as a really nice place.

Dan in Lichfield city centre, deserted early on a Sunday morning. — Take social distancing to the next level: do your urban geocaching at the crack of dawn.

I’m not saying that it was perfect. A combination of the intolerable heat (or else the desiccating effect of the air conditioner) and a mattress that sagged with two adults on it meant that I didn’t sleep much on Saturday night (although that did mean I could get up at 5am for a geocaching expedition around the city before it got too hot later on). And an hour and a half of driving to get to a place where you’re going to see a one-hour show feels long, especially in this age where I don’t really travel anywhere, ever.

But that’s not the point.

Ruth and the kids eat breakfast — The buffet was closed, of course, but these kids were made for an “all you can eat” breakfast.

The point is that Lichfield made me happy, this weekend. And I don’t know how much of that is that it’s just a nice place and how much is that I’ve missed going anywhere or doing anything, but either way, it lead to a delightful weekend.

Higher/Lower Datepicker

6 July 2021 11:43 2

I’ve written before about the trend in web development to take what the web gives you for free, throw it away, and then rebuild it in Javascript. The rebuilt version is invariably worse in many ways – less-accessible, higher-bandwidth, reduced features, more fragile, etc. – but it’s more convenient for developers. Personally, I try not to value developer convenience at the expense of user experience, but that’s an unpopular opinion lately.

Screenshot showing a hovered hyperlink to "Digital Forest" on a list of green hosting providers in France. — Here’s a perfect example I bumped into earlier this week, courtesy of The Green Web Foundation. This *looks* like a hyperlink… but if you open it in a new tab/window, you see a page (not even a 404 page!) with the text “It looks like nothing was found at this location.”

In the site shown in the screenshot above, the developer took something the web gave them for free (a hyperlink), threw it away (by making it a link-to-nowhere), and rebuilt its functionality with Javascript (without thinking about the fact that you can do more with hyperlinks than click them: you can click-and-drag them, you can bookmark them, you can share them, you can open them in new tabs etc.). Ugh.

Date pickers

Particularly egregious are the date pickers. Entering your date of birth on a web form ought to be pretty simple: gov.uk pretty much solved it based on user testing they did in 2013.

Here’s the short of it:

Something you can clearly type a numeric day, month and year into is best.
Three dropdowns are slightly worse, but at least if you use native HTML <select> elements keyboard users can still “type” to filter.
Everything else – including things that look like <select>s but are really funky React <div>s, is pretty terrible.

Calendar datepicker with slider-based timepicker and no text-based fallback. — Calendars can be great for choosing your holiday date range. But pressing “Prev” ~480 times to get to my month of birth isn’t good. Also: what’s with the time “sliders”? (Yes, I know I’ve implemented these myself, in the past, and I’m sorry.)

My fellow Automattician Enfys recently tweeted:

People designing webforms that require me to enter my birthdate:

I am begging you: just let me type it in.

Typing it in is 6-8 quick keystrokes. Trying to navigate a little calendar or spinny wheels back to the 1970s is time-consuming, frustrating and unnecessary.

They’re right. Those little spinny wheels are a pain in the arse if you’ve got to use one to go back 40+ years.

Date "spinner" currently showing 20 December 2012. — These things are *okay* (I guess) on mobile/touchscreen devices, though I’d still prefer the *option* to type in my date of birth. But send one to my desktop and I will curse your name.

Can we do worse?

If there’s one thing we learned from making the worst volume control in the world, the other year, it’s that you can always find a worse UI metaphor. So here’s my attempt at making a date of birth field that’s somehow even worse than “date spinners”:

<span style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" data-mce-type="bookmark" class="mce_SELRES_start"></span><span style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" data-mce-type="bookmark" class="mce_SELRES_start"></span><span style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" data-mce-type="bookmark" class="mce_SELRES_start"></span><a href="https://danq.me/wp-content/uploads/2021/07/higher-lower-datepicker.html">See the “higher/lower datepicker” at https://danq.me/wp-content/uploads/2021/07/higher-lower-datepicker.html</a>

My datepicker implements a game of “higher/lower”. Starting from bounds specified in the HTML code and a random guess, it narrows-down its guess as to what your date of birth is as you click the up or down buttons. If you make a mistake you can start over with the restart button.

Amazingly, this isn’t actually the worst datepicker into which I’ve entered my date of birth! It’s cognitively challenging compared to most, but it’s relatively fast at narrowing down the options from any starting point. Plus, I accidentally implemented some good features that make it better than plenty of the datepickers out there:

It’s progressively enhanced – if the Javascript doesn’t load, you can still enter your date of birth in a sensible way.
Because it leans on a <input type="date"> control, your browser takes responsibility for localising, so if you’re from one of those weird countries that prefers mm-dd-yyyy then that’s what you should see.
It’s moderately accessible, all things considered, and it could easily be improved further.

It turns out that even when you try to make something terrible, so long as you’re building on top of the solid principles the web gives you for free, you can accidentally end up with something not-so-bad. Who knew?

Goose-Related Etymologies

5 July 2021 10:02 2

My favourite thing about geese… is the etymologies of all the phrases relating to geese. There’s so many, and they’re all amazing. I started reading about one, then – silly goose that I am – found another, and another, and another…

A Canada goose at a waterside accompanied by seven goslings. Photo by Brandon Montrone from Pexels. — Have a gander at this photo.

For example:

Barnacle geese are so-called because medieval Europeans believed that they grew out of a kind of barnacle called a goose barnacle, whose shell pattern… kinda, sorta looks like barnacle goose feathers? Barnacle geese breed on remote Arctic islands and so people never saw their chicks, which – coupled with the fact that migration wasn’t understood – lead to a crazy myth that lives on in the species name to this day. Incidentally, this strange belief led to these geese being classified as a fish for the purpose of fasting during Lent, and so permitted. (This from the time period that brought us the Vegetable Lamb of Tartary, of course. I’ve written about both previously.)
Gooseberries may have a similar etymology. Folks have tried to connect it to old Dutch or Germanic words, but inconclusively: given that they appear at the opposite end of the year to some of the migratory birds goose, the same kind of thinking that gave us “barnacle geese” could be seen as an explanation for gooseberries’ name, too. But really: nobody has a clue about this one. Fun fact: the French name for the fruit is groseille à maquereau, literally “mackerel currant”!
A gaggle is the collective noun for geese, seemingly derived from the sound they make. It’s also been used to describe groups of humans, especially if they’re gossiping (and disproportionately directed towards women). “Gaggle” is only correct when the geese are on the ground, by the way: the collective noun for a group of airborne geese is skein or plump depending on whether they’re in a delta shape or not, respectively. What a fascinating and confusing language we have!
John Stephen Farmer helps us with a variety of goose-related sexual slang though, because, well, that was his jam. He observes that a goose’s neck was a penis and gooseberries were testicles, goose-grease is vaginal juices. Related: did you ever hear the euphemism for where babies come from “under a gooseberry bush“? It makes a lot more sense when you realise that gooseberry bush was slang for pubic hair.

Face of a gosse, looking into the camera. Other geese can be seen swimming in the background. — Hey there, you big honker.

An actor whose performance wasn’t up to scratch might describe the experience of being goosed; that is – hissed at by the crowd. Alternatively, goosing can refer to a a pinch on the buttocks possibly in reference to geese pecking humans at about that same height.
If you have a gander at something you take a good look at it. Some have claimed that this is rhyming slang – “have a look” coming from “gander and duck” – but I don’t buy it. Firstly, why wouldn’t it be “goose and duck” (or “gander and drake“, which doesn’t rhyme with “look” at all). And fake, retroactively-described rhyming roots are very common: so-called mockney rhyming slang! I suspect it’s inspired by the way a goose cranes its neck to peer at something that interests it! (“Crane” as a verb is of course also a bird-inspired word!)
Goosebumps might appear on your skin when you’re cold or scared, and the name alludes to the appearance of plucked poultry. Many languages use geese, but some use chickens (e.g. French chair de poule, “chicken flesh”). Fun fact: Slavic languages often use anthills as the metaphor for goosebumps, such as Russian мурашки по коже (“anthill skin”). Recently, people talk of tapping into goosebumps if they’re using their fear as a motivator.
A tailor’s goose is a traditional kind of iron so-named for the shape of its handle.
The childrens game of duck duck goose is played by declaring somebody to be a “goose” and then running away before they catch you. Chasing – or at risk of being chased by! – geese is common in metaphors: if somebody wouldn’t say boo to a goose they’re timid. A wild goose chase (yet another of the many phrases for which we can possibly thank Shakespeare, although he probably only popularised this one) begins without consideration of where it might end up.

A Canada goose and young gosling swim together, side-by-side. Photo by Erick Todd from Pexels. — If humans tell children they were found under a gooseberry bush, where do geese tell their chicks they came from?

If those children are like their parents, you might observe that a wild goose never laid a tame egg: that traits are inherited and predetermined.
Until 1889, the area between Blackfriars and Tower Bridge in London – basically everything around Borough tube station up to the river – was considered to be outside the jurisdiction of both London and Surrey, and fell under the authority of the Bishop of Winchester. For a few hundred years it was the go-to place to find a prostitute South of the Thames, because the Bishop would license them to be able to trade there. These prostitutes were known as Winchester geese. As a result, to be bitten by a Winchester goose was to contract a venereal disease, and goosebumps became a slang term for the symptoms of some such diseases.
Perennial achillea ptarmica is known, among other names, as goose tongue, and I don’t know why. The shape of the plant isn’t particularly similar to that of a goose’s tongue, so I think it might instead relate to the effect of chewing the leaves, which release a spicy oil that might make your tongue feel “pecked”? Goose tongue can also refer to plantago maritima, whose dense rosettes do look a little like goose tongues, I guess. Honestly, I’ve no clue about this one.
If you’re sailing directly downwind, you might goose-wing your sails, putting the mainsail away from the wind and the jib towards it, for balance and to easily maintain your direction. Of course, a modern triangular-sailed boat usually goes faster broad reach (i.e. at an angle of about 45º to the wind) by enough that it’s faster to zig-zag downwind rather than go directly downwind, but I can see how one might sometimes want to try this anatidaetian maneuver.

Plaque with a picture of a goose running and text: "Cross Bones Graveyard. In medieval times this was an unconsecreated graveyard for prostitutes of 'Winchester Geese'. By the 18th century it had become a paupers burial ground, which closed in 1853. Here, local people have created a memorial shrine. The Outcast Dead R.I.P." A smiley face sticker has been attached to the plaque and ribbons and silk flowers are tied nearby. — I feel like the “Cross Bones Graveyard” ought to have been where pirates were buried, but prostitutes is pretty good too.

Geese make their way all over our vocabulary. If it’s snowing, the old woman is plucking her goose. If it’s fair to give two people the same thing (and especially if one might consider not doing so on account of their sex), you might say that what’s good for the goose is good for the gander, which apparently used to use the word “sauce” instead of “good”. I’ve no idea where the idea of cooking someone’s goose comes from, nor why anybody thinks that a goose step march might look anything like the way a goose ~~walks~~ waddles.

With apologies to Beverley, whose appreciation of geese (my take, previously) is something else entirely but might well have got me thinking about this in the first instance.

Parcel Delivery Scammers Could Try Harder

30 June 2021 18:49 4

There’s a lot of talk lately about scam texts pretending to be from Royal Mail (or other parcel carriers), tricking victims into paying a fee to receive a parcel. Hearing of recent experiences with this sort of scam inspired me to dissect the approach the scammers use… and to come up with ways in which the scams could be more-effective.

Let’s take a look at a scam:

Anatomy of a Parcel Fee Scam

A parcel fee scam begins with a phishing email or, increasingly, text message, telling the victim that they need to pay a fee in order to receive a parcel and directing them to a website to make payment.

Scam SMS from "Royal Mail" asking the recipient to go to myparce-uk-manage.com to pay a "fee required for shipping", shown on an iPhone screen — This text message was received by a friend of mine the other week, and it’s pretty typical. Don’t type in that web address, obviously.

If the victim clicks the link, they’ll likely see a fake website belonging to the company who allegedly have the victim’s parcel. They’ll be asked for personal and payment information, after which they’ll be told that their parcel is scheduled for redelivery. They’ll often be redirected back to the real website as a “convincer”. The redirects often go through a third-party redirect site so that your browser’s “Referer:” header doesn’t give away the scam to the legitimate company (if it did, they could e.g. detect it and show you a “you just got scammed by somebody pretending to be us” warning!).

Many scammers also set a cookie so they’ll recognise you if you come back: if you return to the scam site with this cookie in-place, they’ll redirect you instantly to the genuine company’s site. This means that if you later try to follow the link in the text message you’ll see e.g. the real Royal Mail website, which makes it harder for you to subsequently identify that you’ve been scammed. (Some use other fingerprinting methods to detect that you’ve been victimised already, such as your IP address.)

Spoofed Royal Mail webpage saying "Royal Mail: Your Package Has A £2.99 Unpaid Shipping Fee, To Pay This Now Please Visit www.myparcel-uk-manage.com If You Do Not Pay This Your Package Will Be Returned To Sender" and asking for personal details — The spoofed websites usually use HTTPS (“padlock icon” etc.) and have convincing branding (lifted directly from the real company’s website). They frequently – but not always – ask for information that seems… suspicious and unnecessary, like date of birth or bank account sort code.

Typically, no payment is actually taken. Often, the card number and address aren’t even validated, and virtually any input is accepted. That’s because this kind of scam isn’t about tricking you into giving the scammers money. It’s about harvesting personal information for use in a second phase.

Once the scammers have your personal information they’ll either use your card details to make purchases of hard-to-trace, easy-to-resell goods like gift cards or, increasingly, use all of the information you’ve provided in order to perform an even more-insidious trick. Knowing your personal, contact and bank details, they can convincingly call you and pretend to be your bank! Some sophisticated fraudsters will even highlight the parcel fee scam you just fell victim to in order to gain your trust and persuade you that they’re genuinely your bank, which is a very powerful convincer.

"SCAM" spelled out using keycaps from a cyrillic keyboard. Photo by Mikhail Nilov from Pexels. — SCAM > ЫСФЬ? Who knew translation was as simple as these keycaps suggest!

Why does the scam work?

A scam like the one described above works because each individual part of it is individually convincing, but the parts are delivered separately.

Email, reading: Ihre Sendung CH63 **** 26 wurde noch nicht geliefert. — Parcel fee scams aren’t limited to the Anglophone world. Apparently Swiss Post tried to visit me on Monday, even though I’m about 500 miles outside of their delivery area!

Being asked to pay a fee to receive a parcel is a pretty common experience, and getting texts from carriers is too. A lot of people are getting a lot more stuff mail-ordered than they used to, right now, and that – along with the Brexit-related import duties that one in ten people have had to pay – means that it seems perfectly reasonable to get a message telling you that you need to pay a fee to get your parcel.

Similarly, I’m sure we’ve all been called by our bank to discuss a suspicious transaction. (When this happens to me, I’ve always said that I’ll call them back on the number on my card or my bank statements rather than assume that they are who they claim to be. When I first started doing this, 20 years ago, this sometimes frustrated bank policies, but nowadays they’re more accepting.) Most people though will willingly believe the legitimacy of a person who calls them up, addresses them by name and claims to be from their bank.

Separating the scam into two separate parts, each of which is individually unsuspicious, makes it more effective at tricking the victim than simpler phishing scams.

Delivery man, wearing a face mask, holding a parcel and checking his mobile phone. Photo by Kindel Media from Pexels. — “You know these £50 headphones you bought? Yeah, they came from the EU so you owe another £25 somehow.” Fuck Brexit.

Anybody could fall for this. It’s not about being smart and savvy; lots of perfectly smart people become victims of this kind of fraud. Certainly, there are things you can do (like learning to tell a legitimate domain name from a probably-fake one and only ever talking to your bank if you were the one who initiated the call), but we’re all vulnerable sometimes. If you were expecting a delivery, and it’s really important, and you’re tired, and you’re distracted, and then a text message comes along pressuring you to pay the fee right now… anybody could make a mistake.

The scammers aren’t really trying

But do you know what: these scammers aren’t even trying that hard. There’s so much that they could be doing so much “better”. I’m going to tell you, off the top of my head, four things that they could do to amplify their effect.

Wait a minute: am I helping criminals by writing this? No, I don’t think so. I believe that these are things that they’ve thought of already. Right now, it’s just not worthwhile for them to pull out all the stops… they can make plenty of money conning people using their current methods: they don’t need to invest the time and energy into doing their shitty job better.

But if there’s one thing we’ve learned it’s that digital security is an arms race. If people stop falling for these scams, the criminals will up their game. And they don’t need me to tell them how.

"Hacker type" man in hoodie between two computer monitors, looking at his phone. — He ain’t even breaking a sweat. But if the economic pressure was there, he might.

I’m a big fan of trying to make better attacks. Even just looking at site-spoofing scams I’ve been doing this for a couple of decades. Because if we can collectively get ahead of security threats, we’re better able to defend against them.

So no: this isn’t about informing criminals – it’s about understanding what they might do next.

How could the scammers be more effective?

I’d like to highlight four ways that this scam could be made more-effective. Again, this isn’t about helping the criminals: it’s about thinking about and planning for what tomorrow’s attacks might look like.

1. SMS Spoofing

Most of these text messages appear to come from random mobile numbers, which can be an red flag. But it’s distressingly easy to send a text message “from” any other number or even from a short string of text. Imagine how much more-convincing one of these messages would be if it appeared to come from e.g. “Royal Mail” instead?

Text message from "Mum", but actually a marketing text. — Organisers of Parklife Festival were fined £70K for causing distress by texting participants from “Mum” in 2014.

A further step would be to spoof the message to appear to come from the automated redelivery line of the target courier. Many parcel delivery services have automated lines you can call, provide the code from the card dropped through your door, and arrange redelivery: making the message appear to come from such a number means that any victim who calls it will hear a genuine message from the real company, although they won’t be able to use it because they don’t have a real redelivery card. Plus: any efforts to search for the number online (as is done automatically by scam-detection apps) will likely be confused by the appearance of the legitimate data.

"Royal Mail" text message reading: You owe, like, a billion pounds for a parcel we tried to deliver. Go to DanQ.me/royal-mail-scam and pay us. This is totally a legit text message. — This took me literally seconds and fractions of a penny.

SMS spoofing is getting harder as the underlying industry that supports bulk senders tries to clean up its image, but it’s still easy enough to be a real (yet underexploited) threat.

2. Attention to detail

Scammers routinely show a lack of attention to detail that can help give the game away to an attentive target. Spelling and grammar mistakes are commonplace, and compared to legitimate messages the scams generally have suspicious features like providing few options for arranging redelivery or asking for unusual personal information.

A "Royal Mail" scam message that's full of little errors that make it unlike a legitimate one. — Also: where would you even get this email address from, “Royal Mail”? Can’t be from a merchant because I give a different one to each store…

They’re getting a lot better at this already: text messages and emails this year are far more-convincing, from an attention-to-detail perspective, than they were three years ago. And because improvements to the scam can be made iteratively, it’s probably already close to the “sweet spot” at the intersection of effort required versus efficacy. But the bad guys’ attention to detail will only grow and in future they’ll develop richer, more-believable designs and content based on whatever success metrics they collect.

3. Tracking tokens

On which note: it amazes me that these SMS scams don’t yet seem to include any identifier unique to the victim. Spam email does this all the time, but a typical parcel scam text directs you to a simple web address like https://royalmail.co.uk.scamsite.com/. A smarter scam could send you to e.g. https://royalmail.co.uk.scamsite.com/YRC0D35 and/or tell you that your parcel tracking number was e.g. YRC0D35.

"DHL" scam email encouraging you to click a link to arrange redelivery. — Click a link (or even just view the images!) in this phishing email and the sender knows that you read it. SMS scammers could learn from this.

Not only would this be more-convincing for anybody who’s familiar with the kind of messages that are legitimately left by couriers, it would also facilitate the gathering of a great deal of additional metrics which scammers could use to improve their operation. For example:

How many, and which, potential victims clicked the link? Knowing this helps plan future scams, or for follow-up attacks.
Pre-filling personal data, even just a phone number, acts as an additional convincer, or else needn’t be asked at all.
Multivariate testing can determine which approaches work best: show half the victims one form and half the victims another and use the results as research for future evolution.

These are exactly the same techniques that legitimate marketers (and email spammers) use to track engagement with emails and advertisements. It stands to reason that any sufficiently-large digital fraud operation could benefit from them too.

4. Partial submission analysis

I’ve reverse-engineered quite a few parcel scams to work out what they’re recording, and the summary is: not nearly as much as they could be. A typical parcel scam site will ask for your personal details and payment information, and when you submit it will send that information to the attacker. But they could do so much more…

C5 envelope with a yellow "Item underpaid. Fee to pay £1.50" sticker attached. — Real couriers put a card through your door with a code on. Or just put a sticker on your letter and never actually claim the fee, as recently happened to my friends Kit and Matt.

I’ve spoken to potential victims, for example, who got part way through filling the form before it felt suspicious enough that they stopped. Coupled with tracking tokens, even this partial data would have value to a determined fraudster. Suppose the victim only gets as far as typing their name and address… the scammer now has enough information to convincingly call them up, pretending to be the courier, ask for them by name and address, and con them out of their card details over the phone. Every single piece of metadata has value; even just having the victim’s name is a powerful convincer for a future text message campaign.

Summary

There’s so much more that parcel fee SMS scammers could be doing to increase the effectiveness of their campaigns, such as the techniques described above. It’s not rocket science, and they’ll definitely have considered them (they won’t learn anything new from this post!)… but if we can start thinking about them it’ll help us prepare to educate people about how to protect themselves tomorrow, as well as today.

Creed Cult-ure

15 June 2021 19:34 2

My employer, Automattic, has a creed. Right now it reads:

I will never stop learning. I won’t just work on things that are assigned to me. I know there’s no such thing as a status quo. I will build our business sustainably through passionate and loyal customers. I will never pass up an opportunity to help out a colleague, and I’ll remember the days before I knew everything. I am more motivated by impact than money, and I know that Open Source is one of the most powerful ideas of our generation. I will communicate as much as possible, because it’s the oxygen of a distributed company. I am in a marathon, not a sprint, and no matter how far away the goal is, the only way to get there is by putting one foot in front of another every day. Given time, there is no problem that’s insurmountable.

Lots of companies have something like this, even if it falls short of a “creed”. It could be a “vision”, or a set of “values”, or something in that line.

Of course, sometimes that just means they’ve strung three clichéd words together because they think it looks good under their company logo, and they might as well have picked any equally-meaningless words.

Company Name

respect
integrity
teamwork

Future logo and values of of Any Company, Anywhere.

But while most companies (and their staff) might pay lip service to their beliefs, Automattic’s one of few that seems to actually live it. And not in an awkward, shoehorned-in way: people here actually believe this stuff.

By way of example:

A woman in a wheelchair waves to a colleague via her laptop screen; she's smiling and has a cup of coffee by her side. Photo by Marcus Aurelius from Pexels. — Coffee: check. Webcam: check. Let’s touch bases, random colleague!

We’ve got a bot that, among other things, pairs up people from across the company for virtual “watercooler chat”/”coffee dates”/etc. It’s cool: I pair-up with random colleagues in my division, or the whole company, or fellow queermatticians… and collectively these provide me a half-hour hangout about once a week. It’s a great way to experience the diversity of culture, background and interests of your colleagues, as well as being a useful way to foster idea-sharing and “watercooler effect” serendipity.

For the last six months or so, I’ve been bringing a particular question to almost every random-chat I’ve been paired into:

What part of the Automattic creed resonates most-strongly for you right now?

On a good day, I’m at least 90% certain I’m a senior software engineer and not a cult member.

I volunteer my own answer first. It’s varied over time. Often I’m most-attached to “I will never stop learning.” Other times I connect best to “I will communicate as much as possible…” or “I am in a marathon, not a sprint…”. Lately I’ve felt a particular engagement with “I will never pass up the opportunity to help a colleague…”.

It varies for other people too. But every single person I’ve asked this question has been able to answer it. And they’ve been able to answer it confidently and with justifications for or examples of their choice.

Have you ever worked anywhere before where seemingly all your coworkers profess a genuine belief in the corporate creed? Like, enough that some of them get it tattooed onto their bodies. Unless you’ve been brainwashed by a cult, the answer is probably no.

If Automattic is a cult, then it might be too late for me.

Why are Automatticians like that?

For some folks, of course, the creed is descriptive rather than prescriptive. Regarding its initial creation, Matt says that “as a hack to introduce new folks to our culture, we put a beta Automattic Creed, basically a statement of things important to us, written in the first person.”

But this alone isn’t an explanation, because back then there were only around a hundred people in the company: nowadays there are over 1,500. So how can the creed continue to be such a pervasive influence? Or to put it another way: why are Automatticians… like that?

Do we simply attract like-minded individuals? The creed is highly visible and cross-referenced by our recruitment pages, so it wouldn’t be entirely surprising.
Maybe we filter for people who are ideologically-compatible with the creed? Insofar as the qualities it describes are essential to integrating into our corporate culture, yes: our recruitment process does a great job of testing for those qualities.
Perhaps we converge on these values as a result of our experience as Automatticians? Once you’re in, you’re indoctrinated into the tenets of the creed and internalise its ideas.
Or perhaps it’s a combination of the three, in some ratio or another. (What’s the ratio?)

I’ve been here 1⅔ years and don’t know the answer yet. But I’ll tell you this: it’s inspiring to be part of a team that really seem to believe in what they do.

The Automattic Creed presented as an infographic with icons accompanying each tenet. — People keep making infographics of the creed, just for fun. Even if they’re not Automatticians (any longer). That’s not creepy, right?

Incidentally: if the creed speaks to you too, you might like to look at some of the many open positions! I promise we’re not actually a cult.

Plus we’ve been doing “work anywhere” for longer than almost anybody else and we’re really, really good at it.

If you enjoyed this, you might also like other blog posts about my time with Automattic: the recruitment process, accepting an offer, my induction, and the experience of lockdown in a distributed company, among others.

Getting Twitter Avatars (without the Twitter API)

2 June 2021 12:53 0

Among Twitter’s growing list of faults over the years are various examples of its increasing divergence from open Web standards and developer-friendly endpoints. Do you remember when you used to be able to subscribe to somebody’s feed by RSS? When you could see who follows somebody without first logging in? When they were still committed to progressive enhancement and didn’t make your browser download ~5MB of Javascript or else not show any content whatsoever? Feels like a long time ago, now.

Lighthouse Performance score for Twitter's Twitter account page on mobile, scoring 50%. — For one of the most-popular 50 websites in the world, this score is frankly shameful.

But those complaints aside, the thing that bugged me most this week was how much harder they’ve made it to programatically get access to things that are publicly accessible via web pages. Like avatars, for example!

If you’re a human and you want to see the avatar image associated with a given username, you can go to twitter.com/that-username and – after you’ve waited a bit for all of the mandatory JavaScript to download and run (I hope you’re not on a metered connection!) – you’ll see a picture of the user, assuming they’ve uploaded one and not made their profile private. Easy.

If you’re a computer and you want to get the avatar image, it used to be just as easy; just go to twitter.com/api/users/profile_image/that-username and you’d get the image. This was great if you wanted to e.g. show a Facebook-style facepile of images of people who’d retweeted your content.

But then Twitter removed that endpoint and required that computers log in to Twitter, so a clever developer made a service that fetched avatars for you if you went to e.g. twivatar.glitch.com/that-username.

But then Twitter killed that, too. Because despite what they claimed 5½ years ago, Twitter still clearly hates developers.

You want to that image? Well you’ll need a Twitter account, a developer account, an OAuth token set, a stack of code…

Recently, I needed a one-off program to get the avatars associated with a few dozen Twitter usernames.

First, I tried the easy way: find a service that does the work for me. I’d used avatars.io before but it’s died, presumably because (as I soon discovered) Twitter had made things unnecessarily hard for them.

Second, I started looking at the Twitter API documentation but it took me in the region of 30-60 seconds before I said “fuck that noise” and decided that the set-up overhead in doing things the official way simply wasn’t justified for my simple use case.

So I decided to just screen-scrape around the problem. If a human can just go to the web page and see the image, a computer pretending to be a human can do exactly the same. Let’s do this:

/* Copyright (c) 2021 Dan Q; released under the MIT License. */

const Puppeteer = require('puppeteer');

getAvatar = async (twitterUsername) => {
  const browser = await Puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox']});
  const page = await browser.newPage();
  await page.goto(`https://twitter.com/${twitterUsername}`);
  await page.waitForSelector('a[href$="/photo"] img[src]');
  const url = await page.evaluate(()=>document.querySelector('a[href$="/photo"] img').src);
  await browser.close();
  console.log(`${twitterUsername}: ${url}`);
};

process.argv.slice(2).forEach( twitterUsername => getAvatar( twitterUsername.toLowerCase() ) );

The code is ludicrously simple. It took less time, energy, and code to write this than to follow Twitter’s “approved” procedure. You can download the code via Gist.

Obviously, using this code would violate Twitter’s terms of use for automation, so… don’t, I guess?

Given that I only needed to run it once, on a finite list of accounts, I maintain that my approach was probably kinder on their servers than just manually going to every page and saving the avatar from it. But if you set up a service that uses this approach then you’ll certainly piss off somebody at Twitter and history shows that they’ll take their displeasure out on you without warning.

$ node get-twitter-avatar.js alexsdutton richove geohashing TailsteakAD LilFierce1 ninjanails
alexsdutton: https://pbs.twimg.com/profile_images/740505937039986688/F9gUV0eK_200x200.jpg
lilfierce1: https://pbs.twimg.com/profile_images/1189417235313561600/AZ2eLjAg_200x200.jpg
richove: https://pbs.twimg.com/profile_images/1576438972/2011_My_picture4_200x200.jpeg
geohashing: https://pbs.twimg.com/profile_images/877137707939581952/POzWWV2d_200x200.jpg
ninjanails: https://pbs.twimg.com/profile_images/1146364466801577985/TvCfb49a_200x200.jpg
tailsteakad: https://pbs.twimg.com/profile_images/1118738807019278337/y5WWkLbF_200x200.jpg

This output shows the avatar URLs of a half a dozen Twitter accounts. It took minutes to write the code and takes seconds to run, but if I’d have done it the “right” way I’d still be unnecessarily wading through Twitter’s sprawling documentation.

But it works. It was fast and easy and I got what I was looking for.

And the moral of the story is: if you make an API and it’s terrible, don’t be surprised if people screen-scape your service instead. (You can’t spell “scraping” without “API”, amirite?)

Ireland and the UK Aren’t In The Same Timezone!

11 May 2021 13:44 109

This weekend, while investigating a bug in some code that generates iCalendar (ICS) feeds, I learned about a weird quirk in the Republic of Ireland’s timezone. It’s such a strange thing (and has so little impact on everyday life) that I imagine that even most Irish people don’t even know about it, but it’s important enough that it can easily introduce bugs into the way that computer calendars communicate:

Most of Europe put their clocks forward in Summer, but the Republic of Ireland instead put their clocks backward in Winter.

If that sounds to you like the same thing said two different ways – or the set-up to a joke! – read on:

Map showing timezones of Europe. The UK and Ireland are grouped (along with Iceland) in a zone labelled as being UTC+0. — The timezones of Europe look pretty simple compared to some parts of the world, but the illustration of the British Isles hides an interesting eccentricity.

A Brief History of Time (in Ireland)

Poster titled "Time (Ireland) Act 1916", advising that "On and after Sunday 1st October 1916 Western European Time will be ovserved throughout Ireland" asking people to set their clocks and watches back 35 minutes. — Spring forward, fall back… just a little bit back, though. Not too much.

After high-speed (rail) travel made mean solar timekeeping problematic, Great Britain in 1880 standardised on Greenwich Mean Time (UTC+0) as the time throughout the island, and Ireland standardised on Dublin Mean Time (UTC-00:25:21). If you took a ferry from Liverpool to Dublin towards the end of the 19th century you’d have to put your watch back by about 25 minutes. With air travel not yet being a thing, countries didn’t yet feel the need to fixate on nice round offsets in the region of one-hour (today, only a handful of regions retain UTC-offsets of half or quarter hours).

That’s all fine in peacetime, but by the First World War and especially following the Easter Rising, the British government decided that it was getting too tricky for their telegraph operators (many of whom operated out of Ireland, which provided an important junction for transatlantic traffic) to be on a different time to London.

1885 GPO telegraph instrument from the Porthcurno Telegraph Museum, which Dan almost visited the other week but it was closed. — It’s widely believed that the world’s first “U UP? [STOP]” message never got a response as a direct result of Anglo-Irish timezone confusion.

So the Time (Ireland) Act 1916 was passed, putting Ireland on Greenwich Mean Time. Ireland put her clocks back by 35 minutes and synched-up with the rest of the British Isles. And from then on, everything was simple and because nothing ever went wrong in Ireland as a result of the way it was governed by by Britain, nobody ever had to think about the question of timezones on the island again.

Ah. Hmm.

December 1920 photograph showing St Patrick's Street, Cork, following the burning of the city by British forces. — “Those Irish people want to govern their *own* country, do they? After we so kindly shared *our* king with them? Right-ho: let’s set fire to their cities and see how they feel then.”

Following Irish independence, the keeping of time carried on in much the same way for a long while, which will doubtless have been convenient for families spread across the Northern Irish border. But then came the Second World War.

Summers in the 1940s saw Churchill introduce Double Summer Time which he believed would give the UK more daylight, saving energy that might otherwise be used for lighting and increasing production of war materiel.

Ireland considered using the emergency powers they’d put in place to do the same, as a fuel saving measure… but ultimately didn’t. This was possibly because aligning her time with Britain might be seen as undermining her neutrality, but was more likely because the government saw that such a measure wouldn’t actually have much impact on fuel use (it certainly didn’t in Britain). Whatever the reason, though, Britain and Northern Ireland were again out-of-sync with one another until the war ended.

Newspaper clipping advising that "Double Summer Time comes to an end on Saturday night, August 8-9, when all clocks and watches should be put back one hour, thus reverting to British Summer Time, which will probably be maintained throughout the winter." — I like to imagine that the development of powerful computers by the folks at Bletchley Park was a result of needing to keep track of timezones across the British Isles.

From 1968 to 1971 Britain experimented with “British Standard Time” – putting the clocks forward in Summer once, to UTC+1, and then leaving them there for three years. This worked pretty well except if you were Scottish in which case you’ll have found winter mornings to be even gloomier than you were used to, which was already pretty gloomy. Conveniently: during much of this period Ireland was also on UTC+1, but in their case it was part of a different experiment. Ireland were working on joining the European Economic Community, and aligning themselves with “Paris time” year-round was an unnecessary concession but an interesting idea.

But here’s where the quirk appears: the Standard Time Act 1968, which made UTC+1 the “standard” timezone for the Republic of Ireland, was not repealed and is still in effect. Ireland could have started over in 1971 with a new rule that made UTC+0 the standard and added a “Summer Time” alternative during which the clocks are put forward… but instead the Standard Time (Amendment) Act 1971 left UTC+1 as Ireland’s standard timezone and added a “Winter Time” alternative during which the clocks are put back.

Two clocks, both showing the same time. One has a sign reading "LONDON", the other "DUBLIN, I GUESS?" — It all seems so simple until you actually think about it.

(For a deeper look at the legal history of time in the UK and Ireland, see this timeline. Certainly don’t get all your history lessons from me.)

So what?

You might rightly be thinking: so what! Having a standard time of UTC+0 and going forward for the Summer (like the UK), is functionally-equivalent to having a standard time of UTC+1 and going backwards in the Winter, like Ireland, right? It’s certainly true that, at any given moment, a clock in London and a clock in Dublin should show the same time. So why would anybody care?

Perl Data::ICal::TimeZone implementation of Dublin timezone, incorrectly showing summer DST at +1 rather than winter DST of -1. — This code for `Europe/Dublin`, from the Perl module Data::ICal::TimeZone, is technically-incorrect because it states that the winter time is the standard and daylight savings of +1 hour apply in the summer, rather than the opposite.

But declaring which is “standard” is important when you’re dealing with computers. If, for example, you run a volunteer rota management system that supports a helpline charity that has branches in both the UK and Ireland, then it might really matter that the computer systems involved know what each other mean when they talk about specific times.

The author of an iCalendar file can choose to embed timezone information to explain what, in that file, a particular timezone means. That timezone information might say, for example, “When I say ‘Europe/Dublin’, I mean UTC+1, or UTC+0 in the winter.” Or it might say – like the code above! – “When I say ‘Europe/Dublin’, I mean UTC+0, or UTC+1 in the summer.” Both of these declarations would be technically-valid and could be made to work, although only the first one would be strictly correct in accordance with the law.

Stressed programmer hunched over a MacBook. Photo by Anna Shvets from Pexels. — Clients who need solid timezone support represent 50% of a programmer’s production of stress hormones. See also Falsehoods Programmers Believe About Time.

But if you don’t include timezone information in your iCalendar file, you’re relying on the feed subscriber’s computer (e.g. their calendar software) to make a sensible interpretation.. And that’s where you run into trouble. Because in cases like Ireland, for which the standard is one thing but is commonly-understood to be something different, there’s a real risk that the way your system interprets and encodes time won’t necessarily be the same as the way somebody else’s does.

If I say I’ll meet you at 12:00 on 1 January, in Ireland, you rightly need to know whether I’m talking about 12:00 in Irish “standard” time (i.e. 11:00, because daylight savings are in effect) or 12:00 in local-time-at-the-time-of-the-meeting (i.e. 12:00). Humans usually mean the latter because we think in terms of local time, but when your international computer system needs to make sure that people are on a shift at the same time, but in different timezones, it needs to be very clear what exactly it means!

And when your daylight savings works “backwards” compared to everybody else’s… that’s sure to make a developer somewhere cry. And, possibly, blog about your weird legislation.

Spy’s Guidebook Reborn

9 April 2021 14:48 3

When I was a kid of about 10, one of my favourite books was Usborne’s Spy’s Guidebook. (I also liked its sister the Detective’s Handbook, but the Spy’s Guidebook always seemed a smidge cooler to me).

Detective's Handbook andSpy's Guidebook on a child's bookshelf. — I imagine that a younger version of me would approve of our 7-year-old’s bookshelf, too.

So I was pleased when our eldest, now 7, took an interest in the book too. This morning, for example, she came to breakfast with an encrypted message for me (along with the relevant page in the book that contained the cipher I’d need to decode it).

Usborne Spy's Guidebook showing the "Pocket code card" page and a coded message — Decryption efforts were hampered by sender’s inability to get her letter “Z”s the right damn way around.

Later, as we used the experience to talk about some of the easier practical attacks against this simple substitution cipher (letter frequency analysis, and known-plaintext attacks… I haven’t gotten on to the issue of its miniscule keyspace yet!), she asked me to make a pocket version of the code card as described in the book.

Three printed pocket code cards — A three-bit key doesn’t make a simple substitution cipher significantly safer, but it does serve as a vehicle to teach elementary cryptanalysis!

While I was eating leftover curry for lunch with one hand and producing a nice printable, foldable pocket card for her (which you can download here if you like) with the other, I realised something. There are likely to be a lot more messages in my future that are protected by this substitution cipher, so I might as well preempt them by implementing a computerised encoder/decoder right away.

So naturally, I did. It’s at danq.dev/spy-pocket-code and al l the source code is available to do with as you please.

Key 4-1 being used to decode the message: UOMF0 7PU9V MMFKG EH8GE 59MLL GFG00 8A90P 5EMFL — Uh-oh: my cover is blown!

If you’ve got kids of the right kind of age, I highly recommend picking up a copy of the Spy’s Guidebook (and possibly the Detective’s Handbook). Either use it as a vehicle to talk about codes and maths, like I have… or let them believe it’s secure while you know you can break it, like we did with Enigma machines after WWII. Either way, they eventually learn a valuable lesson about cryptography.

Twinebook – Printable Interactive Fiction

5 April 2021 13:43 2

Twine 2 is a popular tool for making hypertext interactive fiction, but there’s something about physical printed “choose your own adventure”-style gamebooks that isn’t quite replicated when you’re playing on the Web. Maybe it’s the experience of keeping your finger in a page break to facilitate a “save point” for when you inevitably have to backtrack and try again?

Annabe enjoying Choose Your Own (Minecraft) Story books. — These are the first branching novels I’ve introduced her to for which she’s felt the need to *take notes*.

As a medium for interactive adventures, paper isn’t dead! Our 7-year-old is currently tackling the second part of a series of books by John Diary, the latest part of which was only published in December! But I worry that authors of printed interactive fiction might have a harder time than those producing hypertext versions. Keeping track of all of your cross-references and routes is harder than writing linear fiction, and in the hypertext

Hand-written note showing branching path story plan, from John Diary's Twitter. — John Diary tweeted about his process back in 2017 and it looks… more manual than *I’d* want.

Twinebook

So I’ve thrown together Twinebook, an experimental/prototype tool which aims to bring the feature-rich toolset of Twine to authors of paper-based interactive fiction. Simply: you upload your compiled Twine HTML to Twinebook and it gives you a printable PDF file, replacing the hyperlinks with references in the style of “turn to 27” to instruct the player where to go next. By default, the passages are all scrambled to keep it interesting, but with the starting passage in position 1… but it’s possible to override this for specific passages to facilitate puzzles that require flipping to specific numbered passages.

Thumb in the page of a Sorcery choose-your-own-adventure gamebook. — In some adventure games, keeping your thumb in the page feels like it’s essential.

Obviously, it doesn’t work with any kind of “advanced” Twine game – anything that makes use of variables, Javascript, etc., for example! – unless you can think of a way to translate these into the written word… which is certainly possible – see Fighting Fantasy‘s skill, stamina, luck and dice-rolling mechanics, for example! – but whether it’s desirable is up to individual authors.

If this tool is valuable to anybody, that’s great! Naturally I’ve open-sourced the whole thing so others can expand on it if they like. If you find it useful, let me know.

Twine screenshot showing many branching paths of the game "Inpatient". — Mapping a complex piece of interactive fiction is a job for a computer, not a human.

If you’re interested in the possibility of using Twine to streamline the production of printable interactive fiction, give my Twinebook prototype a try and let me know what you think.

The Ballad of John Crawford

9 March 2021 12:19 1

Following the success of our last game of Dialect the previous month and once again in a one-week hiatus of our usual Friday Dungeons & Dragons game, I hosted a second remote game of this strange “soft” RPG with linguistics and improv drama elements.

Thieves’ Cant

Our backdrop to this story was Portsmouth in 1834, where we were part of a group – the Gunwharf Ants – who worked as stevedores and made our living (on top of the abysmal wages for manual handling) through the criminal pursuit of “skimming a little off the top” of the bulk-break cargo we moved between ships and onto and off the canal. These stolen goods would be hidden in the basement of nearby pub The Duke of Wellington until they could be safely fenced, and this often-lucrative enterprise made us the envy of many of the docklands’ other criminal gangs.

I played Katie – “Kegs” to her friends – the proprietor of the Duke (since her husband’s death) and matriarch of the group. I was joined by Nuek (Alec), a Scandinavian friend with a wealth of criminal experience, John “Tuck” Crawford (Matt), adoptee of the gang and our aspiring quartermaster, and “Yellow” Mathias Hammond (Simon), a navy deserter who consistently delivers better than he expects to.

Thieves' Cant tableau at the end of a game of Dialect, with cards strewn around the table. — Our second tableau was somehow more-chaotic than the first, even after I accidentally removed several cards before taking this picture!

While each of us had our stories and some beautiful and hilarious moments, I felt that we all quickly converged on the idea that the principal storyline in our isolation was that of young Tuck. The first act was dominated by his efforts to proof himself to the gang, and – with a little snuff – shake off his reputation as the “kid” of the group and gain acceptance amongst his peers. His chance to prove himself with a caper aboard the Queen Anne went proper merry though after she turned up tin-ful and he found himself kept in a second-place position for years longer. Tuck – and Yellow – got proofed eventually, but the extra time spent living hand-to-mouth might have been what first planted the seed of charity in the young man’s head, and kept most of his numbers out of his pocket and into those of the families he supported in the St. Stevens area.

The second act turned political, as Spiky Dave, leader of the competing gang The Barbados Boys, based over Gosport way, offered a truce between the two rivals in exchange for sharing the manpower – and profits – of a big job against a ship from South Africa… with a case of diamonds aboard. Disagreements over the deal undermined Kegs’ authority over the Ants, but despite their March it went ahead anyway and the job was a success. Except… Spiky Dave kept more than his share of the loot, and agreed to share what was promised only in exchange for the surrender of the Ants and their territory to his gang’s rulership.

We returned to interpersonal drama in the third act as Katie – tired of the gang wars and feeling her age – took perhaps more than her fair share of the barrel (the gang’s shared social care fund) and bought herself clearance to leave aboard a ship to a beachside retirement in Jamaica. She gave up her stake in the future of the gang and shrugged off their challenges in exchange for a quiet life, leaving Nuek as the senior remaining leader of the group… but Tuck the owner of the Duke of Wellington. The gang split into those that integrated with their rivals and those that went their separate ways… and their curious pidgin dissolved with them. Well, except for a few terms which hung on in dockside gang chatter, screeched amongst the gulls of Portsmouth without knowing their significance, for years to come.

Playing Out

Despite being fundamentally the same game and a similar setting to when we played The Outpost the previous month, this game felt very different. Dialect is versatile enough that it can be used to write… adventures, coming-of-age tales, rags-to-riches stories, a comedies, horror, romance… and unless the tone is explicitly set out at the start then it’ll (hopefully) settle somewhere mutually-acceptable to all of the players. But with a new game, new setting, and new players, it’s inevitable that a different kind of story will be told.

But more than that, the backdrop itself impacted on the tale we wove. On Mars, we were physically isolated from the rest of humankind and living in an environment in which the necessities of a new lifestyle and society necessitates new language. But the isolation of criminal gangs in Portsmouth docklands in the late Georgian era is a very different kind: it’s a partial isolation, imposed (where it is) by its members and to a lesser extent by the society around them. Which meant that while their language was still a defining aspect of their isolation, it also felt more-artificial; deliberately so, because those who developed it did so specifically in order to communicate surreptitiously… and, we discovered, to encode their group’s identity into their pidgin.

While our first game of Dialect felt like the language lead the story, this second game felt more like the language and the story co-evolved but were mostly unrelated. That’s not necessarily a problem, and I think we all had fun, but it wasn’t what we expected. I’m glad this wasn’t our first experience of Dialect, because if it were I think it might have tainted our understanding of what the game can be.

As with The Outpost, we found that some of the concepts we came up with didn’t see much use: on Mars, the concept of fibs was rooted in a history of of how our medical records were linked to one another (for e.g. transplant compatibility), but aside from our shared understanding of the background of the word this storyline didn’t really come up. Similarly, in Thieves Cant’ we developed a background about the (vegan!) roots of our gang’s ethics, but it barely got used as more than conversational flavour. In both cases I’ve wondered, after the fact, whether a “flashback” scene framed from one of our prompts might have helped solidify the concept. But I’m also not sure whether or not such a thing would be necessary. We seemed to collectively latch onto a story hook – this time around, centred around Matt’s character John Crawford’s life and our influences on it – and it played out fine.

And hey; nobody died before the epilogue, this time!

I’m looking forward to another game next time we’re on a D&D break, or perhaps some other time.