preservation – Dan Q

Today I Rescued 7,234 Old GIFs

This week, GlitchyZorua brought to my attention the Ibiblio Icon Browser, a collection of many thousands of GIF icons curated in the 1990s by Gioacchino La Vecchia. Glitchy’s goal was to archive a copy of all of the icons, which was turning out to be… challenging.

A more-90s website you’re unlikely to see today.

It looks pretty simple: (a) an index page, leading to (b) 24 sub-index pages, leading to (c) 57 icon directory pages, representing (d) 114 icon collections, containing anywhere up to (e) 7,296 icons, mostly but not always 32×32 pixels. Right?

But the challenge comes when you try to go from a directory page to an icon file. It looks like you’re clicking a link, but really you’re clicking… an imagemap.

I’ve talked about imagemaps before, but the essence of them is that you define areas of an image that, when clicked, hyperlink to different places. The most-common way of doing these was always client-side imagemaps, where the HTML code itself contained all of the coordinates and, crucially, the resulting destinations. But that’s not what kind of imagemap this is.

This one’s a server-side imagemap. The HTML code looks like this… and there are no URLs for the resulting library of GIF files anywhere to be seen:

<a href="/iconbin/imagemap/icon3">
  <img src="destic3/icons.gif" ismap>
</a>

That ismap attribute is what tells your browser to send the coordinates that you clicked-at.

Directory indexing is disabled, so we can’t just knock the image filename off the end of the URL and inspect. So how are we to get these images, short of manually, painstakingly, clicking on each one of them? That’s what GlitchyZorua was wondering when I turned up with some bright ideas…

(We’re clearly not the only people who struggled: archive.org hadn’t managed to collect a full set of the icons either.)

Fortunately, we can work out a little something about the gallery images. Exploration of the site shows that they’re always laid out in a grid of up to 8×8, with each (including its size information) occupying a space of 72×89 pixels:

Gallery of 64 images with a particular row and column highlighted to show the boundaries of what's believed to be a particular 'hit target' within it. — A little experimentation shows that clicking anywhere within the intersection area results in a redirect to the same image.

The webserver seems to be running Apache, so it’s probably using something like mod_imagemaps to manage its server-side imagemaps. We can imagine that somewhere on the server there’s probably a file that looks a bit like this, mapping rectangular coordinate pairs to redirect URLs:

# icon3 images:
base destic3/
#    filename  |  top left  |  bottom right
# -------------+------------+----------------
rect 49ers.gif         0,0            72,89
rect 49ers1.gif       73,0           145,89
rect 4dos.2.gif      146,0           217,89
rect 4dos.gif        218,0           289,89
# ... and so on for all 64 images in this collection!

I sincerely hope that La Vecchia had some automated process that he used to produce the thousands of lines of configuration that he needed, and he didn’t write his files by hand!

We don’t have access to those configuration files, but we can infer what hit areas they might have. If each hit area is 72×89 pixels, we can hit the centre of the top-left one at 36×44 and then just keep adding on 72 and 89 pixels to permute the centrepoints of all the hit areas.

In pseudocode, what we’d need to do is:

For each library from 1 to 113,
- For each X coordinate from the set {36, 108, 180, 252, 324, 396, 468, 540}
  - For each Y coordinate from the set {44, 133, 222, 311, 400, 489, 578, 667}
    1. Generate a URL of the form:
      https://www.ibiblio.org/iconbin/imagemap/icon{library}?{x},{y}
    2. Make a HTTP HEAD request to that URL
    3. If you get a HTTP 302 (redirect) response code, record the resulting Location:

That gets us the URL of every one of the thousands of GIFs on the service. Next, we can use wget to download each of them. Sorted!

But we can do one better: once we’ve got all the icons, we can present them in a new website. One without server-side image maps, and with a working search. So that’s what I did. I hacked together a very basic static site generator using Ruby and ERB templates, that produces a gallery with pagination (mirroring the page numbers from the original), plus client-side search. And of course the whole repository can be cloned if you just want a copy of the icons for yourself:

I think my modernised version of this icon library is, while basic, a huge improvement upon the original… (thanks in large part to the advancement of Web technologies in the intervening years!).

Anyway: if you’d like to browse the library in its new form, it’s at ibiblio-icon-archive.danq.dev. It… looks its age, but at least now it’s accessible to the world and able to be archived for posterity.

Letting Games Die

Letting code (and games) die

Mike Cook wrote a provocative blog post this weekend; an anti-preservationist argument for video games. The essence of his arguments seem to boil down to:

Emphasising creation over preservation is liberating, as demonstrated by the imagination in the livecoding community.
Archiving without intensive curation is building an emotional or intellectual safety net you never expect to be used.
Digital preservation is a lossy process: effort spent on accurately preserving some media is at the expense of other media, whose lossy preservation paints in inaccurate picture of what is lost.
Recreation, rather than strict preservation, ensures the continuity of the most culturally-important parts of games

He concludes to say:

60 games are released on Steam every day.

There are 294 game jams active on Steam as I write this.

Preserve nothing. Make more.

To make is to preserve.

Let games die.

Digital preservationism

Philosophically-speaking, there’s no doubt that I am a digital preservationist. I argue against unnecessary URI changes. I donate to The Internet Archive. Back at the Bodleian, I used to carve out free time from project work to spend time making sure the University’s “older” exhibition websites could be made to survive¹. My approach to running out of hard drive space is to buy more hard drives. Even my blog retains content going back into the last millennium²!

But I like this kind of conversation. For World Digital Preservation Day a few years back I re-implemented Pong as a modern application but using retro controllers. Within its micro-exhibition, I used this as an excuse to get people to discuss what does it mean to preserve a videogame?

Two participants play Pong on the Heritage Window in Blackwell Hall at the Bodleian Library. — My reimplementation of *Pong* had several distinct differences from the original… but to a layperson – for whom *Pong* are the target audience! – those differences are irrelevant. To what level fidelity matters depends on many factors, and the biggest problem is that we don’t know what those factors are until it’s time to retrieve these historical media.

Similarly, back in 2021 I reverse-engineered and re-implemented “lost” piece of advertainment Axe Feather, mostly because I felt that a slightly-modernised version belonged in the “commons”.

A woman lies on a bed with her legs crossed, playfully wagging her finger. The mouse cursor is shaped like a feather. — This screenshot isn’t from the original site but from my homage to it. More on that later.

This makes it seem like I’m very much on the side of recreation, rather than preservation, but that’s not the case. In both of these projects I started by disassembling the original works.

That I chose to make them accessible to a modern audience by reimplementation rather than by emulation was an artistic choice. I opted for lower fidelity by making something mildly-transformative. I chose to appeal to the widest possible audience, at the expense of presenting an experience that was totally in-keeping with the original.

But I couldn’t have done that without access to the originals. Had I recreated Pong from memory rather than from re-playing it, I’d have doubtless introduced inconsistencies that would have “felt wrong” to people whose memories of the game, while fundamentally accurate, differed from mine. Had I recreated Axe Feather without first coming up with a mechanism to extract and reformat the video clips in the original I’d have failed to tap into the specific nostalgia of some of its users, which was tied to the specific actor who performed in it³.

So I guess it’s important to me that somebody is preserving these things. So that I can use them to create new things. I stand for preservation for culture’s sake, so that I personally can enjoy the benefits for nostalgia’s sake.

But I get what Mike’s saying

For all that I feel like I’m making the case for “preserve everything; work out what’s important later”, Mike’s argument gives me an uncomfortable cognitive dissonance. Because I’ve also come to discover a joy in the ephemeral, too.

Screenshot from M1 on ARCC, showing the high-score table with DAN50 holding the top score of 14963 (second place scored 12204). — I don’t know who’ll preserve ARCC, with its permanently-capped 500-playerbase limit, but I’m happy that I’ll probably *always* hold the highscore on driving/racing minigame M1.

Increasingly, I’m okay with just taking the experience of something with me. It bothers me that my memory is fallible and that I can’t necessarily recreate a digital experience whose technology has been lost to time, but I am, for the most part, okay with it.

Some of the best gaming experiences I’ve ever had are impossible to “capture” in an archive anyway. They were conversations over the tabletop roleplaying table, or moments of tension resulting from a videogame’s emergent gameplay, or random occurrences unlikely to be replicated. Those get preserved in my memory alone, retold as stories with gradually-decreasing accuracy as new memories take their place.

That said…

Who decides what games get preserved?

I feel like the decision about what to preserve and how should be in the hands of the audience of a piece of art, not its creators. If a videogame (or film, or book, or whatever) is culturally-significant enough to warrant a high-fidelity preservation, it ought to be ultimately up to the members of that culture to make that decision!

Transport Tycoon Deluxe met that bar, and it’s possible to play both faithful recreations or modern reimplementations (the latter having excellent new features) courtesy of the OpenTTD project⁴.

But modern videogames are, perhaps, getting harder to preserve. Always-online features, insidious DRM, digital distribution, live updates, and games-as-a-service streaming all shift the balance of power more-firmly into the hands of publishers⁵ rather than players. It’s already hard to play a randomly-selected thirty-year-old videogame today; I reckon it’ll be almost impossible to do the same thirty years hence.

Saying “let games die” feels a bit like giving up to that inevitability. Like saying to the slimier publishers “it’s okay, we didn’t care about keeping that anyway” when they shut down servers or remotely kill games. I know that’s not what Mike’s saying, but it could be wilfully misinterpreted that way.

Anyway: I don’t have a nice conclusion to any of this. Just a lot of mixed-up feelings.

Footnotes

¹ A policy which, since my departure, does not seem to have continued.

² Even where those writings don’t really represent me well any more.

³ It turns out that, for a significant number of folks who are mostly younger-than-me, this advertisement represented a kind of sexual awakening, based on some of the comments and emails I’ve received about it!

⁴ Which I’ve also donated too. Turns out I’m happy to invest in both pure preservation and in spiritual-successor reimplementation!

⁵ Supposing that Sonic Rumble Party somehow wasn’t a catastrophic pay-to-win nightmare and somehow was deemed culturally-significant… how would you go about archiving it? Without Sega/Sonic Team’s consent, you’d be totally out of luck.

Out of the Trees

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

After “Monty Python’s Flying Circus” ended, Graham Chapman worked with an up-and-coming young writer named Douglas Adams on a new sketch comedy show for the BBC. It was called “Out of the Trees,” and it bombed. Only one episode was made, and that aired only once, on January 10, 1976.

Once the Beeb gave up on “Out of the Trees,” they did to it what they did to so many other programs of that era: they erased it.

…

Chapman had recorded the show on one of the very earliest home videotape formats… it took two years to build a compatible player.

Jason Lefkowitz

It’s neither Chapman nor Adams best work, and you can see how it got canned after only a pilot episode. But it’s not terrible.

But the lesson here is one about the challenge of archiving non-print media. Anything that needs a device to “play” it, whether it’s as simple as a vinyl record or as complex as a videogame, is at greater risk of being lost forever. And the faster the pace of technology moves, the more stuff gets left behind as technology moves on. Is a digital dark age looming? Are we already in it, but that won’t be known until some future date?

Running Code Over Time

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

I’m posting this on the last day of 2019. As I write it, the second post I ever made on meyerweb says it was published “20 years, 6 days ago”. It was published on the second-to-last day of 1999, which was 20 years and one day ago.

What I realized, once the discrepancy was pointed out to me (hat tip: Eric Portis), is the five-day error is there because in the two decades since I posted it, there have been five leap days. When I wrote the code to construct those relative-time strings, I either didn’t think about leap days, or if I did, I decided a day or two here and there wouldn’t matter all that much.

Which is to say, I failed to think about the consequences of my code running over long periods of time. Maybe a day or two of error isn’t all that big a deal, in human-friendly relative-time output. If a post was six years and two days ago but the code says 6 and 1, well, nobody will really care that much even if they notice. But five days is noticeable, and what’s more, it’s a little human-unfriendly. It’s noticeable. It jars.

…

Eric Meyer

As I mentioned in my comments on a repost last week, I work to try to make the things I publish to this site last. But that’s not to say that problems can’t creep in, either because of fundamental bugs left unnoticed until later on (such as the image recompression problem that’s recently lead to some of my older images going wonky; I’m working on it) or else because because of environmental changes e.g. in the technologies that are supported and the ways in which they’re used. The latter are helped by standards and by an adherence to them, but the former will trip over Web developers time and time again, and it’s possible that there’s nothing we can do about it.

No system is perfect, and we don’t have time to engineer every system, every site, every page in a way that near-guarantees its longevity; not by a long shot. I tripped myself over just the year before last when I added Content-Security-Policy headers to my site and promptly broke every embedded YouTube or Vimeo video because they didn’t fit the newly (and retroactively) enforced pattern of allowable content. Such problems are easy to create when you’re maintaining a long-running system with a lot of data. I’m only talking about my blog, but larger, older and/or more-complex systems (of which I’ve worked on a few!) come with their own multitudinous challenges.

That said, the Web has demonstrated a resilience that surpasses most of what is expected in consumer computing. If you want to run a video game from 1994 or even 2001 on a modern computer, you’re likely to find that you have to put in considerably more work than you would have on the day it was released! But even some of the oldest webpages still-existing remain usable today.

Occasionally, though: a “hip” modern technology without the backing of widespread browser standards comes along and creates a dark age. Flash created such a dark age; now there are millions of Flash-dependent web pages that simply don’t work any longer. Java created another. And I worry that the unnecessary overuse of front-end rendering technologies are creating a third that we’re living through right now, oblivious to the data we’re creating and losing.

Let’s think about longevity.

This Page is Designed to Last: A Manifesto for Preserving Content on the Web

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

…

How do we make web content that can last and be maintained for at least 10 years? As someone studying human-computer interaction, I naturally think of the stakeholders we aren’t supporting. Right now putting up web content is optimized for either the professional web developer (who use the latest frameworks and workflows) or the non-tech savvy user (who use a platform).

But I think we should consider both 1) the casual web content “maintainer”, someone who doesn’t constantly stay up to date with the latest web technologies, which means the website needs to have low maintenance needs; 2) and the crawlers who preserve the content and personal archivers, the “archiver”, which means the website should be easy to save and interpret.

So my proposal is seven unconventional guidelines in how we handle websites designed to be informative, to make them easy to maintain and preserve. The guiding intention is that the maintainer will try to keep the website up for at least 10 years, maybe even 20 or 30 years. These are not controversial views necessarily, but are aspirations that are not mainstream—a manifesto for a long-lasting website.

…

Jeff Huang

This page is designed to last, too. In fact, virtually every post of any type I’ve made to this blog (since 2003, older content may vary) has been designed with the intention that it ought to be accessible without dependence on CSS, Javascript, nor any proprietary technology, that the code should be as human-readable as posssible, and that the site itself should be as “archivable” as possible, just as a matter of course.

But that’s only 15 years of dedicated effort to longevity and I’ve still not achieved 100% success! For example, consider my blog post of 14 December 2003, describing the preceeding Troma Night, whose content was lost during the great server failure of July 2004 and for which the backups were unable to completely describe. I’m more-careful now, with more redundancies and backups, but it’s still always going to be the case that a sufficiently-devastating set of simultaneous failures could take this content away. All information has fragility and we can work to mitigate it but we can never completely solve it.

The large number of dead outbound links on the older parts of my site is both worrying – that most others don’t seem to have the same level of commitment to the retention of articles on the Web – and reassuring – that I’m doing significantly better than the average. So next, I guess, I need to focus my attention – like Jeff is – on how we can make such efforts usable by other people, too. The Web belongs to all of us, after all.

Pong

I’ve recently been reimplementing retro arcade classic Pong to show off during a celebration of World Digital Preservation Day 2018 yesterday at the Bodleian Libraries. Here’s how that went down.

The team responsible for digital archiving had plans to spend World Digital Preservation Day running a stand in Blackwell Hall for some time before I got involved. They’d asked my department about using the Heritage Window – the Bodleian’s 15-screen video wall – to show a carousel of slides with relevant content over the course of the day. Or, they added, half-jokingly, “perhaps we could have Pong up there as it’ll be its 46th birthday?”

But I didn’t take it as a joke. I took it as a challenge.

Emulating Pong is pretty easy. Emulating Pong perfectly is pretty hard. Indeed, a lot of the challenge in the preservation of (especially digital) archives in general is in finding the best possible compromise in situations where perfect preservation is not possible. If these 8″ disks are degrading, is is acceptable to copy them onto a different medium? If this video file is unreadable in modern devices, is it acceptable to re-encode it in a contemporary format? These are the kinds of questions that digital preservation specialists have to ask themselves all the damn time.

Emulating Pong in a way that would work on the Heritage Window but be true to the original raised all kinds of complications. (Original) Pong’s aspect ratio doesn’t fit nicely on a 16:9 widescreen, much less on a 27:80 ultrawide. Like most games of its era, the speed is tied to the clock rate of the processor. And of course, it should be controlled using a “dial”.

By the time I realised that there was no way that I could thoroughly replicate the experience of the original game, I decided to take a different track. Instead, I opted to reimplement Pong. A reimplementation could stay true to the idea of Pong but serve as a jumping-off point for discussion about how the experience of playing the game may be superficially “like Pong” but that this still wasn’t an example of digital preservation.

Two participants play Pong on the Heritage Window — Bip… boop… boop… bip… boop… bip…

Here’s the skinny:

A web page, displayed full-screen, contains both a <canvas> (for the game, sized appropriately for a 3 × 3 section of the video wall) and a <div> full of “slides” of static content to carousel alongside (filling a 2 × 3 section).
Javascript writes to the canvas, simulates the movement of the ball and paddles, and accepts input from the JS Gamepad API (which is awesome, by the way). If there’s only one player, a (tough! – only three people managed to beat it over the course of the day!) AI plays the other paddle.
A pair of SNES controllers adapted for use as USB controllers which I happened to own already.

I felt that the day, event, and game were a success. A few dozen people played Pong and explored the other technology on display. Some got nostalgic about punch tape, huge floppy disks, and even mechanical calculators. Many more talked to the digital archives folks and I about the challenges and importance of digital archiving. And a good time was had by all.

I’ve open-sourced the entire thing with a super-permissive license so you can deploy it yourself (you know, on your ultrawide video wall) or adapt it as you see fit. Or if you’d just like to see it for yourself on your own computer, you can (but unless you’re using a 4K monitor you’ll probably need to use your browser’s mobile/responsive design simulator set to 3200 × 1080 to make it fit your screen). If you don’t have controllers attached, use W/S to control player 1 and the cursor keys for player 2 in a 2-player game.

Happy 46th birthday, Pong.