Octave Compass

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Octave Compass showing the scale and chords of D Major

This is cool. Twist the outside dial to transpose the tonal centre of your key. Twist the inner dial to shift the mode of the scale. Turn on- or off- individual tones to shift into more-exotic modes. Use triangles to illustrate the triads of your major, minor, and sustained chords, or add the sixth or seventh with the help of a trapezoid.

The amateur music theorist in me continually struggles to visualise what and why a key is what it is. This kind of thing helps. Plus, what a cool software toy!

Downloading a YouTube Music Playlist for Offline Play

Now that Google Play Music has been replaced by YouTube Music, and inspired by the lampshading the RIAA did recently with youtube-dl, a friend asked me: “So does this mean I could download music from my Google Play Music/YouTube Music playlists?”

A Creative MuVo MP3 player (and FM radio), powered off, on a white surface.
My friend still uses a seriously retro digital music player, rather than his phone, to listen to music. It’s not a Walkman or a Minidisc player, I suppose, but it’s still pretty elderly. But it’s not one of these.

I’m not here to speak about the legality of retaining offline copies of music from streaming services. YouTube Music seems to permit you to do this using their app, but I’ll bet there’s something in their terms and conditions that specifically prohibits doing so any other way. Not least because Google’s arrangement with rights holders probably stipulates that they track how many times tracks are played, and using a different player (like my friend’s portable device) would throw that off.

But what I’m interested in is the feasibility. And in answering that question, in explaining how to work out that it’s feasible.

A "Your likes" playlist in the YouTube Music interface, with 10 songs showing.
The web interface to YouTube Music shows playlists of songs and streaming is just a click away.

Spoiler: I came up with an approach, and it looks like it works. My friend can fill up their Zune or whatever the hell it is with their tunes and bop away. But what I wanted to share with you was the underlying technique I used to develop this approach, because it involves skills that as a web developer I use most weeks. Hold on tight, you might learn something!

youtube-dl can download “playlists” already, but to download a personal playlist requires that you faff about with authentication and it’s a bit of a drag. Just extracting the relevant metadata from the page is probably faster, I figured: plus, it’s a valuable lesson in extracting data from web pages in general.

Here’s what I did:

Step 1. Load all the data

I noticed that YouTube Music playlists “lazy load”, and you have to scroll down to see everything. So I scrolled to the bottom of the page until I reached the end of the playlist: now everything was in the DOM, I could investigate it with my inspector.

Step 2. Find each track’s “row”

Using my browser’s debugger “inspect” tool, I found the highest unique-sounding element that seemed to represent each “row”/track. After a little investigation, it looked like a playlist always consists of a series of <ytmusic-responsive-list-item-renderer> elements wrapped in a <ytmusic-playlist-shelf-renderer>. I tested this by running document.querySelectorAll('ytmusic-playlist-shelf-renderer ytmusic-responsive-list-item-renderer') in my debug console and sure enough, it returned a number of elements equal to the length of the playlist, and hovering over each one in the debugger highlighted a different track in the list.

A browser debugger inspecting a "row" in a YouTube Music playlist. The selected row is "Baba Yeta" by Peter Hollens and Malukah, and has the element name "ytmusic-responsive-list-item-renderer" shown by the debugger.
The web application captured right-clicks, preventing the common right-click-then-inspect-element approach… so I just clicked the “pick an element” button in the debugger.

Step 3. Find the data for each track

I didn’t want to spend much time on this, so I looked for a quick and dirty solution: and there was one right in front of me. Looking at each track, I saw that it contained several <yt-formatted-string> elements (at different depths). The first corresponded to the title, the second to the artist, the third to the album title, and the fourth to the duration.

Better yet, the first contained an <a> element whose href was the URL of the piece of music. Extracting the URL and the text was as simple as a .querySelector('a').href on the first <yt-formatted-string> and a .innerText on the others, respectively, so I ran [...document.querySelectorAll('ytmusic-playlist-shelf-renderer ytmusic-responsive-list-item-renderer')].map(row=>row.querySelectorAll('yt-formatted-string')).map(track=>[track[0].querySelector('a').href, `${track[1].innerText} - ${track[0].innerText}`]) (note the use of [...*] to get an array) to check that I was able to get all the data I needed:

Debug console running on YouTube Music. The output shows an array of 256 items; items 200 through 212 are visible. Each item is an array containing a YouTube Music URL and a string showing the artist and track name separated by a hyphen.
Lots of URLs and the corresponding track names in my friend’s preferred format (me, I like to separate my music into folders by album, but I suppose I’ve got a music player with more than a floppy disk’s worth of space on it).

Step 4. Sanitise the data

We’re not quite good-to-go, because there’s some noise in the data. Sometimes the application’s renderer injects line feeds into the innerText (e.g. when escaping an ampersand). And of course some of these song titles aren’t suitable for use as filenames, if they’ve got e.g. question marks in them. Finally, where there are multiple spaces in a row it’d be good to coalesce them into one. I do some experiments and decide that .replace(/[\r\n]/g, '').replace(/[\\\/:><\*\?]/g, '-').replace(/\s{2,}/g, ' ') does a good job of cleaning up the song titles so they’re suitable for use as filenames.

I probably should have it fix quotes too, but I’ll leave that as an exercise for the reader.

Step 5. Produce youtube-dl commands

Okay: now we’re ready to combine all of that output into commands suitable for running at a terminal. After a quick dig through the documentation, I decide that we needed the following switches:

  • -x to download/extract audio only: it defaults to the highest quality format available, which seems reasomable
  • -o "the filename.%(ext)s" to specify the output filename but accept the format provided by the quality requirement (transcoding to your preferred format is a separate job not described here)
  • --no-playlist to ensure that youtube-dl doesn’t see that we’re coming from a playlist and try to download it all (we have our own requirements of each song’s filename)
  • --download-archive downloaded.txt to log what’s been downloaded already so successive runs don’t re-download and the script is “resumable”

The final resulting code, then, looks like this:

console.log([...document.querySelectorAll('ytmusic-playlist-shelf-renderer ytmusic-responsive-list-item-renderer')].map(row=>row.querySelectorAll('yt-formatted-string')).map(track=>[track[0].querySelector('a').href, `${track[1].innerText} - ${track[0].innerText}`.replace(/[\r\n]/g, '').replace(/[\\\/:><\*\?]/g, '-').replace(/\s{2,}/g, ' ')]).map(trackdata=>`youtube-dl -x "${trackdata[0]}" -o "${trackdata[1]}.%(ext)s" --no-playlist --download-archive downloaded.txt`).join("\n"));

Code running in a debugger and producing a list of youtube-dl commands to download a playlist full of music.
The output isn’t pretty, but it’s suitable for copy-pasting into a terminal or command prompt where it ought to download a whole lot of music for offline play.

This isn’t an approach that most people will ever need: part of the value of services like YouTube Music, Spotify and the like is that you pay a fixed fee to stream whatever you like, wherever you like, obviating the need for a large offline music collection. And people who want to maintain a traditional music collection offline are most-likely to want to do so while supporting the bands they care about, especially as (with DRM-free digital downloads commonplace) it’s never been easier to do so.

But for those minority of people who need to play music from their streaming services offline but don’t have or can’t use a device suitable for doing so on-the-go, this kind of approach works. (Although again: it’s probably not permitted, so be sure to read the rules before you use it in such a way!)

Step 6. Learn something

But more-importantly, the techniques of exploring and writing console Javascript demonstrated are really useful for extracting all kinds of data from web pages (data scraping), writing your own userscripts, and much more. If there’s one lesson to take from this blog post it’s not that you can steal music on the Internet (I’m pretty sure everybody who’s lived on this side of 1999 knows that by now), but that you can manipulate the web pages you see. Once you’re viewing it on your computer, a web page works for you: you don’t have to consume a page in the way that the author expected, and knowing how to extract the underlying information empowers you to choose for yourself a more-streamlined, more-personalised, more-powerful web.

Mouth Dreams is so stupid and so good

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

For the sake of our own sanity, if nothing else, we wanted to take a minute to dig into the most wonderfully dumb song on the entire album—although technically it’s two songs, since tracks 13 and 14, “Fredhammer” and “Limp Wicket,” both share a single unifying sound: Limp Bizkit’s ode to heartbreak, “Nookie”.

Together, these two tracks cover so much of what makes Cicierega so great, from the unexpected sample choices, to the step-stuttering repetition of lyrics, to the moment when you realize he’s snuck the Seinfeld baseline into the middle of the song. There’s also the fact that the whole thing works irritatingly well, from Durst rapping over the “Sledgehammer” horns, to the undeniably triumphant feel of the “Yub nubs” kicking in.

I confess to a genuine and unironic love of Mouth Moods (and, to a lesser extent, Neil Cicierega’s other Mouth* work). I don’t know if I enjoy Mouth Dreams even more, but it’s certainly a close thing.

William Hughes succinctly describes what makes Mouth Dreams so good. I promise you that if you start down this rabbit hole you’ll soon be lost (what does it all mean? what are the secret messages hidden in the spectrogram output? why, just why?), but in the most wonderful way. You can listen to the entire album on Soundcloud.

The Unexpected Solace in Learning to Play Piano

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

…while I practice, I have to simultaneously read, listen, think, translate. Every synapse of my brain is so utterly overwhelmed, there is no capacity left to think about the world out there.

When Christoph Niemann published this piece about learning to play the piano during the most-lockdown-y parts of the Coronavirus lockdown, it rang a chord with me (hah!). I, too, have experimented with learning to play the piano this spring/summer, and found a similar kind of Zen-like focussed calm emerge out of the frustration of staring at a piece of sheet music and wondering why I couldn’t for the life of me get me fingers to remember to do when they got to that point.

I started out with – after following some random links off the back of finishing the last bit of work for my recent masters degree – a free course in music theory by the OU, because I figured that coming in from a theoretical perspective would help with the way my brain thinks about this kind of thing. I supplemented that with a book we got for the kids to use to learn to play, and now I’ve now graduated to very gradually hunt-and-pecking my way through Disney’s back catalogue. I can play Go The Distance, Colors of the Wind and most of Can You Feel The Love Tonight barely well enough that I don’t feel the need to tear my own ears off, so I guess I’m making progress, though I still fall over my own hands every time I try to play any bloody thing from Moana. 20 minutes at a time, here and there, and I’m getting there. I don’t expect to ever be good at it, but I’m enjoying it nonetheless.

But anyway: this piece in the NYT Magazine really spoke to me, and to hear that somebody with far more music experience than me can struggle with all the same things I do when getting started with the piano was really reassuring.

Dune (2020)

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Oh my god I’m so excited. I’m afraid they might fuck up the story even more than David Lynch did in 1984 (not that I don’t love that film, too, but in a very different way than the books). I mean: I’d have hoped a modern adaptation would have a bigger part for Chani than it clearly does. And I know nothing at all about the lead, Timothée Chalamet. If only there was something I could do about these fears?

I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.

Yeah, that’s the kind of thing.

The supporting cast look excellent. I think Josh Brolin will make an awesome Gurney Halleck, Jason Momoa will rock Duncan Idaho, and I’m looking forward to seeing Stephen McKinley Henderson play Thufir Hawat. But if there’s just one thing you should watch the trailer for… it’s to listen to fragments of Hans Zimmer’s haunting, simplistic choral adaptation of Pink Floyd’s Eclipse.

Bodley and the Bookworms – Scan and Deliver

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

You know that strange moment when you see your old coworkers on YouTube doing a cover of an Adam and the Ants song? No: just me?

Still good to see the Bodleian put a fun spin on promoting their lockdown-friendly reader services. For some reason they’ve marked this video “not embeddable” (?) in their YouTube settings, so I’ve “fixed” the copy above for you.

Lindsey Stirling/Johnny Rzeznik String Session

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

One of the last “normal” things I got to do before the world went full lockdown was to attend a Goo Goo Dolls concert with Ruth, and so to see two musicians I enjoy team up to perform a song and share some words of hope and encouragement for a better future beyond these troubled times… feels fitting and inspiring.

Also awesome to see that Stirling’s perhaps as much a fan of Live in Buffalo as I am.

Fun diversion: I never know how to answer the question “what kind of music do you like?”, because I increasingly (and somewhat deliberately) find that I enjoy a wider and wider diversity of different genres and styles. But perhaps the right answer might be: “I like music that makes me feel the way I feel when I hear Cuz You’re Gone recorded from the Goo Goo Dolls’ concert in Buffalo on 4 July 2004, specifically the bit between 4 minutes 10 seconds and 4 minutes 33 seconds into the song, right at the end of the extended bridge. It’s full of anticipatory energy and building to a wild crescendo that seems to mirrors the defiance of both the band and the crown in the face of the torrential rain that repeatedly almost brought an end to the concert. Music that makes me feel like that bit does: that’s the kind of music I like. Does that help?”

Fox In Sox

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

This is the best rap music-related thing to happen to Dr. Seuss since Wubble Down, over 10 years ago. And understand that there have been a lot of strong candidates.

(I maintain that I’m not that “into” rap music. But somehow I keep proving myself wrong. Loving this song right now, for example.)

When you’re in the club and you’re hungry… (Misheard Lyrics… Round Two!)

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

My mother has long argued that a large category of popular music, second only to those on the subjects of sex and drugs, are about food. This so-called corpus of food songs is, I’m pretty confident, mostly based on mishearing lyrics, but I think she’d have a friend in the fabulous Bec Hill who’s this month made a follow-up to her video When You Listen to the Radio When You’re Hungry. And it’s even better (and to my delight, paella still manages to make a cameo appearance).

Unfortunately Warner Music Group don’t seem to have a sense of humour and you might find that you can’t watch her new video on YouTube. But thankfully that’s not how the Internet works (somebody should tell them!) and if proxying isn’t the best solution for you then you can just watch her new video on the BBC’s Facebook page instead.

Without The Bod

Eight years, six months, and one week after I started at the Bodleian, we’ve gone our separate ways. It’s genuinely been the nicest place I’ve ever worked; the Communications team are a tightly-knit, supportive, caring bunch of diverse misfits and I love them all dearly, but the time had come for me to seek my next challenge.

(For anybody out-of-the-loop, I’m moving to Automattic after surviving their amazing, mind-expanding recruitment process).

Bodleian staff badge and keyring
My imminent departure began to feel real when I turned over my badge and gun card and keys.

Being awesome as they are, my team threw a going-away party for me, complete with food from Najar’s Place, about which I’d previously raved as having Oxford’s best falafels. I wasn’t even aware that Najar’s place did corporate catering… actually, it’s possible that they don’t and this was just a (very) special one-off.

Partry platters courtesy of Najar's Place along with drinks and cake.
Start from the left, work towards the right.

Following in the footsteps of recent team parties, they’d even gotten a suitably-printed cake with a picture of my face on it. Which meant that I could leave my former team with one final magic trick, the never-before-seen feat of eating my own head (albeit in icing form).

Dan on a cake
Of course, the first thing I was asked to do was to put a knife through my own neck.

As the alcohol started to work, I announced an activity I’d planned: over the weeks prior I’d worked to complete but not cash-in reward cards at many of my favourite Oxford eateries and cafes, and so I was now carrying a number of tokens for free burritos, coffees, ice creams, smoothies, pasta and more. Given that I now expect to spend much less of my time in the city centre I’d decided to give these away to people who were able to answer challenge questions presented – where else? – on our digital signage simulator.

"Play Your Shards Right" on the big screen.
Among the games was Play Your Shards Right, a game of “higher/lower” played across London’s skyscrapers.

I also received some wonderful going-away gifts, along with cards in which a few colleagues had replicated my long tradition of drawing cartoon animals in other people’s cards, by providing me with a few in return.

Coworkers competing agressively for tiny prizes.
“Wait… all of these Javascript frameworks look like they’re named after Pokémon!”

Later, across the road at the Kings’ Arms and with even more drinks inside of me, I broke out the lyrics I’d half-written to a rap song about my time at the Bodleian. Because, as I said at the time, there’s nothing more-Oxford than a privileged white boy rapping about how much he’d loved his job at a library (video also available on QTube [with lyrics] and on Videopress).

It’s been an incredible 8½ years that I’ll always look back on with fondness. Don’t be strangers, guys!

Dan says goodbye to Bodleian colleagues
My department’s made far too much use out of that “Sorry you’re leaving” banner, this year. Here’s hoping they get a stabler, simpler time next year.