Blog

More than you expected?

You're reading everything on Dan's blog - including notes, reposts, checkins, videos and comics.
That might be more than you wanted to see, if you're only interested in blog posts (articles) Dan has written.

We Are The Martians

This week our usual Dungeons & Dragons group took a week off while our DM recovered from a long and tiring week. As a “filler”, I offered to facilitate a game of Dialect: A Game About Language and How It Dies, from Thorny Games, who I discovered through a Metafilter post about their latest free print-and-play game, Sign: A Game about Being Understood. Yes, all of their games about about language and communication; what of it?

Dialect

Dialect could be described as a rules-light, GM-less (it has a “facilitator” role, but they have no more authority than any player on anything), narrative-driven/storytelling roleplaying game based on the concept of isolated groups developing their own unique dialect and using the words they develop as a vehicle to tell their stories.

Dialect's rulebook and card deck.
It’s also super-pretty to leaf through and hold.

This might not be the kind of RPG that everybody likes to play – if you like your rules more-structured, for example, or you’re not a fan of “one-shot”/”beer and pretzels” gaming – but I was able to grab a subset of our usual roleplayers – Alec, Matt R, Penny, and I – and have a game (with thanks to Google Meet for videoconferencing and Roll20 for the virtual tabletop: I’d have used Foundry but its card support is still pretty terrible!).

The Outpost

A game of Dialect begins with a backdrop – what other games might call a scenario or adventure – to set the scene. We opted for The Outpost, which put the four of us among the first two thousand humans to colonise Mars, landing in 2045. With help from some prompts provided by the backdrop we expanded our situation in order to declare the “aspects” that would underpin our story, and then expand on these to gain a shared understanding of our world and society:

  • Refugees from plague: Our expedition left Earth to escape from a series of devastating plagues that were ravaging the planet, to try to get a fresh start on another world.
  • Hostile environment: Life on Mars is dominated by the ongoing struggle for sufficient food and water; we get by, but only thanks to ongoing effort and discipline and we lack some industries that we haven’t been able to bootstrap in the five years we’ve been here (we had originally thought that others would follow).
  • Functionalist, duty-driven society: The combination of these two factors led us to form a society based on supporting its own needs; somewhat short of a caste system, our culture is one of utilitarianism and unity.
Finished game board from The Outpost backdrop of our game of Dialect.
Our finished game board, or tableau.

It soon became apparent that communication with Earth had been severed, at least initially, from our end: radicals, seeing the successes of our new social and economic systems, wanted to cement our differences by severing ties with the old world. And so our society lives in a hub-and-spoke cave system beneath the Martian desert, self-sustaining except for the need to send rovers patrolling the surface to scout for and collect valuable surface minerals.

In this world, and prompted by our cards, we each developed a character. I was Jeramiah, the self-appointed “father” of the expedition and of this unusual new social order, who remembers the last disasters and wars of old Earth and has revolutionary plans for a better world here on Mars, based on controlled growth and a planned economy. Alec played Sandy – “Tyres” to their friends – a rover-driving explorer with one eye always on the horizon and fresh stories for the colony brought back from behind every new crater and mountain. Penny played Susie, acting not only as the senior medic to the expedition but something more: sort-of the “mechanic” of our people-driven underground machine, working to keep alive the genetic records we’d brought from Earth and keep them up-to-date as our society eventually grew, in order to prevent the same kinds of catastrophe happening here. “Picker” Ben was our artist, for even a functionalist society needs somebody to record its stories, celebrate its accomplishments, and inspire its people. It’s possible that the existence of his position was Jeramiah’s doing: the two share a respect for the stark, barren, undeveloped beauty of the Martian surface.

We developed our language using prompt cards, improvised dialogue, and the needs of our society. But the decades that followed brought great change. More probes began to land from Earth, more sophisticated than the ones that had delivered us here. They brought automated terraforming equipment, great machines that began to transform Mars from a barren wasteland into a place for humans to thrive. These changes fractured our society: there were those that saw opportunity in this change – a chance to go above ground and live in the sun, to expand across the planet, to make easier the struggle of our day-to-day lives. But others saw it as a threat: to our way of life, which had been shaped by our challenging environment; to our great social experiment, which could be ruined by the promise of an excessive lifestyle; to our independence, as these probes were clearly the harbingers of the long-promised second wave from Earth.

Even as new colonies were founded, the Martians of the Hub (the true Martians, who’d been here for yams time, lived and defibed here, not these tanning desert-dwelers that followed) resisted the change, but it was always going to be a losing battle. Jeramiah took his last breath in an environment suit atop a dusty Martian mountain a day’s drive from the Hub, watching the last of the nearby deserts that was still untouched by the new green plants that had begun to spread across the surface. He was with his friend Sandy, for despite all of the culture’s efforts to paint them as diametrically opposed leaders with different ideas of the future, they remained friends until the end. As the years went by and more and more colonists arrived, Sandy left for Phobos, always looking for a new horizon to explore. Sick of the growing number of people who couldn’t understand his language or his art, Ben pioneered an expedition to the far side of the planet where he lived alone, running a self-sustaining agri-home and exploring the hills until his dying day. We were never sure where Susie ended up, but it wasn’t Mars: she’d talked about joining humanity’s next big jump, to the moons of Jupiter, so perhaps she’s out there on one of the colonies of Titan or Europa. Maybe, low clicks, she’s even keeping our language alive out there.

Retrospective

The whole event was a lot of fun and I’m keen to repeat it, perhaps with a different group and a different backdrop. The usual folks know who they are, but if you’re not one of those and you want in next time we play, drop me a message of some kind.

Confused.com Confuses Me

It’s that time of year again when I comparison-shop for car insurance, and every time I come across a new set of reasons to hate the developers at Confused.com. How do you confuse me? Let me count the ways.

No means yes

I was planning to enumerate my concerns to them directly, via their contact form, but when I went to do so I spotted this bit of genius, which clinched it and made me write a blog post instead:

Animated GIF showing how clicking on "No" on Confused.com's contact form checks the "Yes" box.
Clicking the word “Yes” means “Yes”. Clicking the word “No” means “Yes” as well.

Turns out that there’s a bit of the old sloppy-paste going on there:

<input type="radio" value="Yes" id="ContactByPhoneYes" name="contactByPhone" />
<label for="ContactByPhoneYes" class="label">Yes</label>
<input type="radio" value="No" id="ContactByPhoneNo" name="contactByPhone" />
<label for="ContactByPhoneYes" class="label">No</label>

I guess nobody had the “consent talk” with Confused.com?

That’s not my name!

Error message "Please enter a name between 2 and 30 letters long..." when Dan enters "Q" as his surname.
Somebody needs to brush up on their falsehoods programmers believe about names.

Honestly, I’m used to my unusual name causing trouble by now and I know how to work around it in the way that breaks the fewest systems (I can even usually get airline tickets without too much difficulty nowadays). But these kinds of (arbitrary) restrictions must frustrate folks like Janice Keihanaikukauakahihulihe’ekahaunaele.

I guess their developers didn’t realise that this blog post was parody?

Also, that’s not my title!

This one, though, pisses me off:

Animation showing title selector with options "Mr", "Mrs", "Miss", and "More...". Clicking "More..." reveals three more: "Ms", "Dr (Male)" and "Dr (Female)"
As everybody knows, there are only six titles, and two of them are “Dr”.

This is a perfect example of why your forms should ask for what you actually want to know, not for what you think people want to tell you. Just ask!

  1. If you want to know my gender, ask for my gender! (I’m a man, by the way.)
    I don’t understand why you want to know – after all, it’s been illegal since 2012 to risk-assess/price car insurance differently on the grounds of gender – but maybe you’ve got a valid reason. Which hopefully you’ll tell me in a tooltip. Like you’re using it as a (terrible checksum) when you check my driving license details, that’s fine!
  2. If you want to know my title, ask for my title! (I prefer not to use one, but if you must use one I’d prefer Mx.)
    This ought to be an optional field, of course, and ideally you want a free text input or else you’ll always have missed somebody (Lord, Reverend, Prince, Wing Commander…). It’s in your interests because I’m totally going to pick at random otherwise. Today I’m a Ms.

Consistency? Never heard of it.

It’s not a big thing, but if you come up with a user interface paradigm like “clicking More… shows more buttons”, you ought to stick to it.

Animation of marital statuses: clicking "More..." shows a dropdown instead of more buttons.
Maybe their internal style guide says “a More… button with three additional options should use buttons, but four additional options should be a drop-down”. But it seems more-likely that they just don’t have one.

Again, I’m not sure exactly what all of this data is used for, nor why there’s a need to differentiate between married couples and civil partnerships, but let’s just assume this is all necessary and legitimate and just ask ourselves: why are we using drop-downs now for “More…”? We were using buttons just a second ago!

"How many cars are at your home?" has a "More..." box that shows more buttons.
This was just crying out for a type-in field. But I guess the same developer who did the “Title” question did this one too, and wanted to show off the fancy “more buttons” control they’d written. (Imaginary style guide be damned!)

What’s my occupation again?

There’s so much to unpack in the “occupation” part of the form that I’m not even sure where to begin. Let’s just pick out a few things:

What type of student are you? List of options, many of which intersect.
I never answered a question this hard even in the exams I did when I was a student. Why do we care where students live… except if they’re postgrads? If I’m a mature student studying a postgraduate course in medicine while living at home with my parents… which of the five possible options should I pick? And, again: what difference could it conceivably make?

The student thing is just the beginning, though. You can declare up to two jobs, but if the first one is “house person/parent” you can’t have a second one. If you’re self-employed, that has to be your first job even though the guidance says that the one you spend most time on must be the first one (this kind of thing infuriated me when I used to spend 60% of my work time employed, 20% self-employed, and 20% studying).

I’m not saying it’s easy to make a form like this. I know from experience that it’s not. I am saying that Confused.com make it look a lot harder than it is.

Tooltip reading "Please choose the employment status that reflects the majority of the work you do. For example if you are a house person and have a part time job of 5 hours a week, you should select 'House person/parent' as your primary job.
Well that clears everything up. Also, I think you mean “houseperson”, unless you’re referring to somebody who is half-house/half-person, like some kind of architectural werewolf.

What do you mean, you live with your partner?

At a glance, this sounds like a “poly world problem”, but hear me out:

Relationship to policy holder: Living together (couple) results in the error "The driver's marital status must be Living With Partner" if their relationship to the proposer is Living Together (Couple)".
What you’re seeing here is a reference-identity error. I can’t possibly be living together with somebody as a couple if their marital status isn’t “Living With Partner”.

I put Ruth‘s martial status as married, because she’s married to JTA. But then when it asked how she was related to me, it wouldn’t accept “Living together (couple)”.

Relationship to proposer question with 'spouse' option but not 'living with partner'.
If I put Ruth as the primary policyholder (proposer) though, I don’t even get the option of “living together (couple)” to describe her relationship with me. ‘Cos it’s physically impossible to have a partner and be married, right?

Even if you don’t think it’s odd that they hide “living with partner” button as an option to describe a married person’s relationship to somebody other than their spouse… you’ve still got to agree that it’s a little bit odd that they don’t hide the “spouse” button. In other words, this user interface is more-okay with you having multiple spouses than it is with you having a spouse and an unmarried partner!

And of course this isn’t just about polyamorous folks: there are perfectly “normal” reasons that a person might end up confused by this interface, too. For example a separated (but not yet divorced) couple, one of whom has a new partner (it’s not even inconceivable that such a pair might share custody of a car). Also interesting is the fact that the form doesn’t care about the gender of your spouse (it doesn’t ask for “husband” or “wife”) but does care about the gender of your parent, child, or sibling. What gives?

Half a dozen easy fixes. Go for it, Confused.com.

Given that their entire marketing plan for most of the last two decades has been that they reduce customer confusion, Confused.com’s user interface leaves a lot to be desired. As I’ve mentioned before – and speaking as a web developer that’s been in the game for longer than their company has – it’s not necessarily easy to get this kind of thing right. But you can improve a form like this, a little at a time. And every little win counts for something: a more-satisfied returning customer, perhaps, or a new word-of-mouth recommendation.

Or you can just let it languish and continue to have the kind of form that people mock on the public Internet.

It’ll be a year until I expect to comparison-shop for car insurance again: let’s see how they get on, shall we?

The Diamonds, The Dagger and One Classy Dame

On account of the pandemic, I’d expected my fortieth birthday to be a somewhat more-muted affair than I’d hoped. I had a banner, I got trolled by bagels, and I received as a gift a pizza oven with which I immediately set fire to several pieces of cookware, but I hadn’t expected to be able to do anything like the “surprise” party of my thirtieth, and that saddened me a little. So imagine my surprise when I come back from an evening walk the day after my birthday to discover than an actual (remote) surprise party really had been arranged without my knowing!

Matt, Suz, Alec, Jen, Dermot and Doreen on a Google Meet screen.
“Hello, remote guests! What are you doing here?”

Not content with merely getting a few folks together for drinks, though, Ruth and team had gone to great trouble (involving lots of use of the postal service) arranging a “kit” murder mystery party in the Inspector McClue series – The Diamonds, The Dagger, and One Classy Dame – for us all to play. The story is sort-of a spiritual successor to The Brie, The Bullet, and The Black Cat, which we’d played fifteen years earlier. Minor spoilers follow.

JTA (wearing a string of pearls) and Robin (wearing sunglasses)
“Hello, local guests. Wait… why are you all in costumes…?”

Naturally, I immediately felt underdressed, having not been instructed that I might need a costume, and underprepared, having only just heard for the first time that I would be playing the part of German security sidekick Lieutenant Kurt Von Strohm minutes before I had to attempt my most outrageous German accent.

Dan with his tongue out holding a glass of champagne.
Fortunately I was able to quickly imbibe a few glasses of champagne and quickly get into the spirit. Hic.

The plot gave me in particular a certain sense of deja vu. In The Brie, The Bullet, and The Black Cat, I played a French nightclub owner who later turned out to be an English secret agent supplying the French Resistance with information. But in The Diamonds, The Dagger, and One Classy Dame I played a Gestapo officer who… also later turned out to be an English secret agent infiltrating the regime and, you guessed it, supplying the French Resistance.

Jen drinking from the neck of a nearly-empty wine bottle.
As she had previously with Sour Grapes, Ruth had worked to ensure that a “care package” had reached each murder mystery guest. Why yes, it was a boozy care package.

It was not the smoothest nor the most-sophisticated “kit” murder mystery we’ve enjoyed. The technology made communication challenging, the reveal was less-satisfying than some others etc. But the company was excellent. (And the acting way pretty good too, especially by our murderer whose character was exquisitely played.)

JTA downing a Jeroboam of champagne.
The largest bottle, though, was with us: we opened the Jeroboam of champagne Ruth and JTA had been saving from their anniversary (they have a tradition involving increasing sizes of bottle; it’s a whole thing; I’ll leave them to write about it someday).

And of course the whole thing quickly descended into a delightful shouting match with accusations flying left, right, and centre and nobody having a clue what was going on. Like all of our murder mystery parties!

Google Meet transcript with the words "You are the Jewish", which nobody said.
I’m not sure how I feel about Google Meet’s automatic transcription feature. It was generally pretty accurate, but it repeatedly thought that it heard the word “Jewish” being spoken by those of us who were putting on German accents, even though none of us said that.

In summary, the weekend of my fortieth birthday was made immeasurably better by getting to hang out with (and play a stupid game with) some of my friends despite the lockdown, and I’m ever so grateful that those closest to me were able to make such a thing happen (and without me even noticing in advance).

How Not to use A Pizza Oven

Clearly those closest to me know me well, because for my birthday today I received a beautiful (portable: it packs into a bag!) wood-fired pizza oven, which I immediately assembled, test-fired, cleaned, and prepped with the intention of feeding everybody some homemade pizza using some of Robin‘s fabulous bread dough, this evening.

Ooni Fyra portable wood-fired pizza oven.

Fuelled up with wood pellets the oven was a doddle to light and bring up to temperature. It’s got a solid stone slab in the base which looked like it’d quickly become ideal for some fast-cooked, thin-based pizzas. I was feeling good about the whole thing.

But then it all began to go wrong.

Animated GIF showing the fire blazing as seen through the viewing window on the front of the oven.
The confined space quickly heats up to a massive 400-500ºC.

If you’re going to slip pizzas onto hot stone – especially using a light, rich dough like this one – you really need a wooden peel. I own a wooden peel… somewhere: I haven’t seen it since I moved house last summer. I tried my aluminium peel, but it was too sticky, even with a dusting of semolina or a light layer of oil. This wasn’t going to work.

I’ve got some stone slabs I use for cooking fresh pizza in a conventional oven, so I figured I’d just preheat them, assemble pizzas directly on them, and shunt the slabs in. Easy as (pizza) pie, right?

Pizza on fire in oven.
Within 60 seconds the pizza was cooked and, in its elevated position atop a second layer of stone, the crust began to burn. The only-mildy-charred bits were delicious, though.

This oven is hot. Seriously hot. Hot enough to cook the pizza while I turned my back to assemble the next one, sure. But also hot enough to crack apart my old pizza stone. Right down the middle. It normally never goes hotter than the 240ºC of my regular kitchen oven, but I figured that it’d cope with a hotter oven. Apparently not.

So I changed plan. I pulled out some old round metal trays and assembled the next pizza on one of those. I slid it into the oven and it began to cook: brilliant! But no sooner had I turned my back than… the non-stick coating on the tray caught fire! I didn’t even know that was a thing that could happen.

Flames flickering at the back of a pizza oven.
Hello fire. I failed to respect you sufficiently when I started cooking. I’ve learned my lesson now.

Those first two pizzas may have each cost me a piece of cookware, but they tasted absolutely brilliant. Slightly coarse, thick, yeasty dough, crisped up nicely and with a hint of woodsmoke.

But I’m not sure that the experience was worth destroying a stone slab and the coating of a metal tray, so I’ll be waiting until I’ve found (or replaced) my wooden peel before I tangle with this wonderful beast again. Lesson learned.

Story Time

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

First frame of "Story Time" comic from Channelate. An old man sits in an armchair and talks to two small children sitting on the floor in front of him. The old man says "I ever tel you squirts about the year 2020?"

When this comic (go read the full thing) came out at the tail end of last year, I thought to myself: yeah, that’s about right. I’m resharing that on my birthday in a week or so.

‘Cos I’m forty today, and I sort of had a half-baked dream that I’d throw some kind of big party and get people together. My surprise party for my thirtieth birthday party was an excellent (and much-needed) bash, and I guess I’d thought I’d try to replicate the feel of that, but a decade on (and not a surprise party… although in the end the last one wasn’t either).

But 2020’s the year that keeps on giving, so I’m postponing my party plans to… “some other time”. And so this comic really spoke to me.

Having Kids In Our Poly Triad by Sara Valta

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Having Kids In Our Poly Triad by Sara Valta title showing picture of two men and a woman hugging the outline of a baby

Sara’s back! You might remember a couple of years ago she’d shared with us a comic on her first year in a polyamory! We’re happy to have her back with a slice of life and a frank n’ real conversation about having kids in her Poly Triad relationship.

This sort of wholesome loving chat is just the thing we need for the start of 2021.

Start your year with a delightful comic about the author negotiating possible future children in a queer polyamorous triad, published via Oh Joy Sex Toy. Sara previously published a great polyamory-themed comic via OJST too, which is also worth a look.

The Fourth Way to Inject CSS

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Last week, Jeremy Keith wrote an excellent summary of the three ways to inject CSS into a web document. In short, he said:

There are three ways—that I know of—to associate styles with markup.

External CSS

<link rel="stylesheet" href="/path/to/styles.css">

Embedded CSS

<style>element { property: value; }</style>

Inline CSS

<element style="property: value"></element>

While talking about external CSS, he hinted at what I consider to be a distinct fourth way with its own unique use cases:; using the Link: HTTP header. I’d like to share with you how it works and why I think it needs to be kept in people’s minds, even if it’s not suitable for widespread deployment today.

Injecting CSS using the Link: HTTP Header

Every one of Jeremy’s suggestions involve adding markup to the HTML document itself. Which makes sense; you almost always want to associate styles with a document regardless of the location it’s stored or the medium over which it’s transmitted. The most popular approach to adding CSS to a page uses the <link> HTML element, but did you know… the <link> element has a semantically-equivalent HTTP header, Link:.

Bash shell running the command "curl -i https://hell.meiert.org/core/php/link.php". The response includes a HTTP Link: header referencing a CSS stylesheet; this is highlighted.
The only active example I’ve been able to find are test pages like Jens Oliver Meiert’s (pictured), Louis Lazaris’s , and Anne van Kesteren’s, but it’s possible that others are hiding elsewhere on the Web.

According to the specifications, the following HTTP responses are equivalent in terms of the CSS that would be loaded and applied to the document:

Traditional external CSS injection:

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8

<!doctype html>
<html>
  <head>
    <title>My page</title>
    <link rel="stylesheet" href="/style/main.css">
  </head>
  <body>
    <h1>My page</h1>
  </body>
</html>

Link: header CSS injection:

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Link: </style/main.css>; rel="stylesheet"

<!doctype html>
<html>
  <head>
    <title>My page</title>
  </head>
  <body>
    <h1>My page</h1>
  </body>
</html>
Diagram showing a web browser requesting a document from a web server, the web server finding the document and returning it after attaching HTTP headers.
A webserver adds headers when it serves a document anyway. Adding one more is no big deal.

Why is this important?

This isn’t something you should put on your website right now. This (21-year-old!) standard is still only really supported in Firefox and pre-Blink Opera, so you lose perhaps 95% of the Web (it could be argued that because CSS ought to be considered progressive enhancement, it’s tolerable so long as your HTML is properly-written).

If it were widely-supported, though, that would be a really good thing: HTTP headers beat meta/link tags for configurability, performance management, and separation of concerns. Need some specific examples? Sure: here’s what you could use HTTP stylesheet linking for:

Diagram showing a reverse proxy server modifying the headers set by an upstream web server in response to a request by a web browser.
You have no idea how many times in my career I’d have injected CSS Link: headers using a reverse proxy server the standard was universally-implemented. This technique would have made one of my final projects at the Bodleian so much easier…
  • Performance improvement using aggressively preloaded “top” stylesheets before the DOM parser even fires up.
  • Stylesheet injection by edge caches to provide regionalised/localised changes to brand identity.
  • Strong separation of content and design by hosting content and design elements in different systems.
  • Branding your staff intranet differently when it’s accessed from outside the network than inside it.
  • Rebranding proprietary services on your LAN without deep inspection, using reverse proxies.
  • Less-destructive user stylesheet injection by plugins etc. that doesn’t risk breaking icky on-page Javascript (e.g. theme switchers).
  • Browser detection? 😂 You could use this technique today to detect Firefox. But you absolutely shouldn’t; if you think you need browser detection in CSS, use this instead.

Unfortunately right now though, stylesheet Link: headers remain consigned to the bin of “cool stylesheet standards that we could probably use if it weren’t for fucking Google”; see also alternate stylesheets.

Downloading a YouTube Music Playlist for Offline Play

Now that Google Play Music has been replaced by YouTube Music, and inspired by the lampshading the RIAA did recently with youtube-dl, a friend asked me: “So does this mean I could download music from my Google Play Music/YouTube Music playlists?”

A Creative MuVo MP3 player (and FM radio), powered off, on a white surface.
My friend still uses a seriously retro digital music player, rather than his phone, to listen to music. It’s not a Walkman or a Minidisc player, I suppose, but it’s still pretty elderly. But it’s not one of these.

I’m not here to speak about the legality of retaining offline copies of music from streaming services. YouTube Music seems to permit you to do this using their app, but I’ll bet there’s something in their terms and conditions that specifically prohibits doing so any other way. Not least because Google’s arrangement with rights holders probably stipulates that they track how many times tracks are played, and using a different player (like my friend’s portable device) would throw that off.

But what I’m interested in is the feasibility. And in answering that question, in explaining how to work out that it’s feasible.

A "Your likes" playlist in the YouTube Music interface, with 10 songs showing.
The web interface to YouTube Music shows playlists of songs and streaming is just a click away.

Spoiler: I came up with an approach, and it looks like it works. My friend can fill up their Zune or whatever the hell it is with their tunes and bop away. But what I wanted to share with you was the underlying technique I used to develop this approach, because it involves skills that as a web developer I use most weeks. Hold on tight, you might learn something!

youtube-dl can download “playlists” already, but to download a personal playlist requires that you faff about with authentication and it’s a bit of a drag. Just extracting the relevant metadata from the page is probably faster, I figured: plus, it’s a valuable lesson in extracting data from web pages in general.

Here’s what I did:

Step 1. Load all the data

I noticed that YouTube Music playlists “lazy load”, and you have to scroll down to see everything. So I scrolled to the bottom of the page until I reached the end of the playlist: now everything was in the DOM, I could investigate it with my inspector.

Step 2. Find each track’s “row”

Using my browser’s debugger “inspect” tool, I found the highest unique-sounding element that seemed to represent each “row”/track. After a little investigation, it looked like a playlist always consists of a series of <ytmusic-responsive-list-item-renderer> elements wrapped in a <ytmusic-playlist-shelf-renderer>. I tested this by running document.querySelectorAll('ytmusic-playlist-shelf-renderer ytmusic-responsive-list-item-renderer') in my debug console and sure enough, it returned a number of elements equal to the length of the playlist, and hovering over each one in the debugger highlighted a different track in the list.

A browser debugger inspecting a "row" in a YouTube Music playlist. The selected row is "Baba Yeta" by Peter Hollens and Malukah, and has the element name "ytmusic-responsive-list-item-renderer" shown by the debugger.
The web application captured right-clicks, preventing the common right-click-then-inspect-element approach… so I just clicked the “pick an element” button in the debugger.

Step 3. Find the data for each track

I didn’t want to spend much time on this, so I looked for a quick and dirty solution: and there was one right in front of me. Looking at each track, I saw that it contained several <yt-formatted-string> elements (at different depths). The first corresponded to the title, the second to the artist, the third to the album title, and the fourth to the duration.

Better yet, the first contained an <a> element whose href was the URL of the piece of music. Extracting the URL and the text was as simple as a .querySelector('a').href on the first <yt-formatted-string> and a .innerText on the others, respectively, so I ran [...document.querySelectorAll('ytmusic-playlist-shelf-renderer ytmusic-responsive-list-item-renderer')].map(row=>row.querySelectorAll('yt-formatted-string')).map(track=>[track[0].querySelector('a').href, `${track[1].innerText} - ${track[0].innerText}`]) (note the use of [...*] to get an array) to check that I was able to get all the data I needed:

Debug console running on YouTube Music. The output shows an array of 256 items; items 200 through 212 are visible. Each item is an array containing a YouTube Music URL and a string showing the artist and track name separated by a hyphen.
Lots of URLs and the corresponding track names in my friend’s preferred format (me, I like to separate my music into folders by album, but I suppose I’ve got a music player with more than a floppy disk’s worth of space on it).

Step 4. Sanitise the data

We’re not quite good-to-go, because there’s some noise in the data. Sometimes the application’s renderer injects line feeds into the innerText (e.g. when escaping an ampersand). And of course some of these song titles aren’t suitable for use as filenames, if they’ve got e.g. question marks in them. Finally, where there are multiple spaces in a row it’d be good to coalesce them into one. I do some experiments and decide that .replace(/[\r\n]/g, '').replace(/[\\\/:><\*\?]/g, '-').replace(/\s{2,}/g, ' ') does a good job of cleaning up the song titles so they’re suitable for use as filenames.

I probably should have it fix quotes too, but I’ll leave that as an exercise for the reader.

Step 5. Produce youtube-dl commands

Okay: now we’re ready to combine all of that output into commands suitable for running at a terminal. After a quick dig through the documentation, I decide that we needed the following switches:

  • -x to download/extract audio only: it defaults to the highest quality format available, which seems reasomable
  • -o "the filename.%(ext)s" to specify the output filename but accept the format provided by the quality requirement (transcoding to your preferred format is a separate job not described here)
  • --no-playlist to ensure that youtube-dl doesn’t see that we’re coming from a playlist and try to download it all (we have our own requirements of each song’s filename)
  • --download-archive downloaded.txt to log what’s been downloaded already so successive runs don’t re-download and the script is “resumable”

The final resulting code, then, looks like this:

console.log([...document.querySelectorAll('ytmusic-playlist-shelf-renderer ytmusic-responsive-list-item-renderer')].map(row=>row.querySelectorAll('yt-formatted-string')).map(track=>[track[0].querySelector('a').href, `${track[1].innerText} - ${track[0].innerText}`.replace(/[\r\n]/g, '').replace(/[\\\/:><\*\?]/g, '-').replace(/\s{2,}/g, ' ')]).map(trackdata=>`youtube-dl -x "${trackdata[0]}" -o "${trackdata[1]}.%(ext)s" --no-playlist --download-archive downloaded.txt`).join("\n"));

Code running in a debugger and producing a list of youtube-dl commands to download a playlist full of music.
The output isn’t pretty, but it’s suitable for copy-pasting into a terminal or command prompt where it ought to download a whole lot of music for offline play.

This isn’t an approach that most people will ever need: part of the value of services like YouTube Music, Spotify and the like is that you pay a fixed fee to stream whatever you like, wherever you like, obviating the need for a large offline music collection. And people who want to maintain a traditional music collection offline are most-likely to want to do so while supporting the bands they care about, especially as (with DRM-free digital downloads commonplace) it’s never been easier to do so.

But for those minority of people who need to play music from their streaming services offline but don’t have or can’t use a device suitable for doing so on-the-go, this kind of approach works. (Although again: it’s probably not permitted, so be sure to read the rules before you use it in such a way!)

Step 6. Learn something

But more-importantly, the techniques of exploring and writing console Javascript demonstrated are really useful for extracting all kinds of data from web pages (data scraping), writing your own userscripts, and much more. If there’s one lesson to take from this blog post it’s not that you can steal music on the Internet (I’m pretty sure everybody who’s lived on this side of 1999 knows that by now), but that you can manipulate the web pages you see. Once you’re viewing it on your computer, a web page works for you: you don’t have to consume a page in the way that the author expected, and knowing how to extract the underlying information empowers you to choose for yourself a more-streamlined, more-personalised, more-powerful web.

New Blood Donation Rules Better, I Suppose

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

So the NHS blood donation rules are changing again. And while they’re certainly getting closer, they’re still not quite hitting the bullseye yet.

That’s great. Prior to 2011 men who’d ever had sex with men, as well as women who’d had sex with such a man within the last 6 months, were banned from donating blood. That rule clearly spun out of the AIDS hysteria of the 1980s and generally entrenched homophobia. It probably did little to protect the recipients of blood, and certainly did a lot to increase the stigma experienced by non-straight men.

A shooting target with a great many holes.
You throw enough policies at a problem, eventually one will get close-enough, right?

The 2011 change permitted donation by men who’d previously had sex with men… so long as they hadn’t done so within the last year. Which opened the doors to donation by a lot of men: e.g. bisexual men who’d been in relationships exclusively with women, gay men who’d been celibate for a period, etc. It still wasn’t great, but it was a step in the right direction.

So when I saw that the rules were changing to better target only risky behaviours, rather than behaviours that are so broad-brush as to target identities, I was initially delighted. Evidence-based medicine, you say? For the win.

A nurse wearing gloves uses a hyperdermic needle to take a blood sample from a patients' arm, as seen from over the patient's shoulder.
Go on! Stick it in me! I’ll still be able to give blood, right?

But… it’s not all sunshine and rainbows. The new rules prohibit blood donation regardless of gender by people who’ve had sex with more than one person in the last three months.

Diagram showing a relationship between Andre and Brandon (married), and between Carlos and Brandon (partners). Andre and Carlos are now allowed to give blood, but Brandon still can't.
Sorry Brandon, we only want Andre and Carlos’ blood.

So if for example if there’s a V-shaped relationship consisting of three men, who only have sex within their thruple… two of them are now allowed to give blood but the third isn’t? (This isn’t a contrived example. I know such a thruple.)

Stranger still: if you swap Brandon in the diagram above for a woman then you get a polycule that’s a lot like mine, but the woman in the middle used to be allowed to give blood… and now can’t! My partner Ruth is in exactly the position: her situation hasn’t changed, but because she’s been in a long-term relationship with exactly two people she’s now not allowed to give blood. Wot?

On the whole, this rule change is an improvement. We’re getting closer to a perfect answer. But it’s amusing to see where the policy misses again and excludes donors who would otherwise be perfectly viable.

Update: as this is attracting a lot of attention I just wanted to remind people that the whole discussion is, of course, a lot more complicated than can be summarised in a single, short, opinionated blog post. Take a look at the FAIR steering group’s recommendations and compare to the government’s press release.

Update #2: justifying choice of words – “AIDS hysteria” refers specifically to the media (and to a lesser extent the policy) reactions to the (very real, very devastating) pandemic. For a while there it was perfectly normal to see (often misguided, sometimes homophobic) scaremongering news coverage suggesting that everybody was at enormous risk from HIV.

Banners Begone!

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

It’s a clicker quest of purging banners from your homepage!

Push through dozens of banners and upgrade various web-tools to grow your ad-blocking power.
Hone your clicking skills – matters are in your own fingers hands!

There’s no time for idling, show these ads their place!

I quite enjoyed this progressive game: it’s a little bit different than most, the theming is fun, it lends itself to multiple strategies, and it’s not geared towards making you wait for longer and longer intervals (as is common in this genre): there’s always something you can be doing to get closer to your goal.

You’ll need a mouse with left and right buttons.