Moving Three Rings’ Servers

Yesterday, I fulfilled the primary Three Rings objective I set for myself when I kicked off my sabbatical twelve weeks ago and migrated the entire application to a new hosting provider (making a stack of related improvements along the way).

Network diagram but with entities having faces and named Chungus, Atul, Summer, Gwen, Alice, Astrid, and Demmy.
If you ignore the smiley faces and names my 10-year-old annotated it with, this diagram’s a reasonably-accurate representation of what each of our three production server clusters look like.

I did some work on this project during my Three Rings-focussed International Volunteer Day last week, but it feels like I’ve been working on it for much longer than that. And it feels like it… because I have been.

Months prior, I was comparing different providers and their relative merits, making sure that our (quirky and specific) needs could be met. Weeks beforehand, I was running a “dry run” every four or five days, streamlining the process of moving the ~450GB1 of live data while minimising downtime. Days before the event felt like the countdown for a rocket launch, with final preparations underway: reducing DNS time-to-lives, ensuring users knew about our downtime window, and generally fitting in a little time to panic.

Terminal screenshot showing a directory listing of a logs directory with several gzipped logfiles with different date-stamped suffixes, and the contents of the logrotate configuration file that produced them.
I made reference on International Volunteer Day to how we needed to configure logrotate. When you’re building architecture for a system as gnarly as Three Rings, there’s about a billion tools that need such careful tweaking2.
The whole operation was amazingly successful. We’d announced an at-risk period of up to six hours and I was anticipating it taking three… but the whole thing was completed within a downtime window of just two and a half hours. And I fully credit all of the preparation time. It turns out that “measure twice, cut once” is a sensible strategy3.

It’s challenging to pull off a “big”, intensive operation like this in an entirely voluntary operation. I’m not saying I couldn’t have done it were I not on sabbatical, but it’d certainly have been harder and riskier.

But then, I also couldn’t have done it without the kickass team of volunteers I’ve surrounded myself with. I guess the real success story here is in the power of a well-aligned team and in volunteer effort.

Footnotes

1 Three Rings‘ user data is represented by a little under 70GB of MariaDB databases plus about 380GB of organisational storage: volunteer photos, files, email attachments, and the like. Certainly not massive by comparison to, say, social media sites, search engines, and larger eCommerce platforms… but large enough that moving it takes a little planning!

2 Okay, a billion tools to configure? That’s an exaggeration. Especially now: since the architectural changes I’ve put in place this week, for example, production app server builds of Three Rings no require a custom-compiled build of Nginx (yes, this really was something we used to need).

3 Which you’d think I’d have realised with my more-successful recent second attempt at secret-cabinet-making.

× ×

International Volunteer Day

Duration

Podcast Version

This post is also available as a podcast. Listen here, download for later, or subscribe wherever you consume podcasts.

Today is International Volunteer Day. And because I’m in the middle of my (magical) sabbatical, I’ve had no difficulty dedicating what would have been the entire workday to a variety of volunteer activities for the benefit of Three Rings, the nonprofit I founded 22 years ago for the purpose of making volunteer management, and therefore volunteering, easier.

Liveblogging my day

Dan, a white man with a goatee-style beard and long hair tied back, wearing a black t-shirt with an abstract pride rainbow, holds a mug of coffee in one hand and gives a thumbs-up with the other. Behind him, three computer screens show a mixture of code, source control, chat, email, and web browser windows. His desk is cluttered.
Step one in a highly productive day of tech volunteering is, as you might have expected: coffee.

I’m pretty sure that most folks don’t know what my voluntary work at Three Rings involves1, and so I decided I’d celebrate this year’s International Volunteer Day by live-blogging what I got up to in a series of notes throughout the day (1, 2, 3, 4, 5, 6, 7, 8)2.

Maybe, I figured, doing so might provide more of an insight into what a developer/devops role at Three Rings looks like.

Regression-testing a fix

My first task for the day related to a bugfix that we’re looking to deploy. Right now, there’s a problem which sometimes stops the “mail merge” fields in emails sent by Three Rings3. We have a candidate fix, but because it’s proposed as a hotfix (i.e. for deployment directly to production), it requires a more-thorough review process involving more volunteer developers than code which will be made available for beta testing first.

Screenshot showing web application Three Rings viewed on a local development machine, being used to compose an email about International Volunteer Day. A merge field called 'first_name' is included in the email, and it's hand-annotated with the question 'working?'. Alongside, GitHub Desktop shows that we've rolled-back to a revision from two months ago.
My aim with this task was to roll-back to an earlier version of the software, before the bug was first introduced (by fixing something different!), and ensure that the functionality remained the same as it always had been.

It turned out that everything was alright, so I reported back to my fellow reviewers about how I’d tested and what my results had been. Once some more eyes have hit the tests and the new/changed code, that’ll hopefully be ready-to-launch.

Slack screenshot showing a discussion between Dan Q and another volunteer about whether it's reasonable for a volunteer to be able to send another volunteer an email containing private information accessible only to the second volunteer, so long as it's embedded via a merge field and the actual value is never shown to the first volunteer. Ultimately, Dan suggests that we stick to the established behaviour, which turns out after testing to be that volunteers can't send such private information to one another.
Three Rings volunteers primarily communicate via Slack: it helps us to work asynchronously, which supports the fact that our volunteers all have different schedules and preferences for how they plan their volunteering: some spend whole days, some just a few hours now and then. We’re the ultimate “armchair volunteering” opportunity!

Other code reviews

There were also a couple of other code reviews that demanded my attention, so I moved on to those next.

GitHub screenshot showing a suggested change to a pull request. Dan-Q is suggesting that the new sentence 'These roles belong to <%= h @org.name %>.' should not have a full stop at the end, for consistency with other similar headings.
We’re all about collaboration, discussion, learning from one another, and volunteer-empowerment, so my suggestions in this case were non-blocking: I trust my fellow volunteer to either accept my suggestion (if it’s right), reject it (if it’s wrong), or solicit more reviews or bring it to Slack or our fortnightly dev meeting (if it requires discussion).

Preparing new infrastructure

Next week we’re scheduled to do a big migration of server infrastructure to help provide more future growing-room: it’s exciting, but also a little scary4!

Terminal screenshot showing a directory listing of a logs directory with several gzipped logfiles with different date-stamped suffixes, and the contents of the logrotate configuration file that produced them.
For now, though, all I needed to do was to tweak our logrotate and backup configurations in response to testing of our new systems.

Not all of our developer volunteers also wear a “devops” hat, but a few of us do5. It’s quite a satisfying role – devops can feel like tidying and organising, and just as a physical space can feel clean, simple, and functional when it’s carefully and minimalistically laid out, a well-organised cluster of servers humming along in exactly the way they should can be a highly-satisfying thing to be responsible for, too.

Sorting the post

It’s not all techy work, though. And while it’s true that Three Rings has a good group of less-nerdy6 people to handle many of the non-programming tasks that you need to run a voluntary organisation like ours, it’s also true that many of us wear multiple hats and pull our weight in several different roles.

In front of a letterbox, an hand holds an open envelope and a (censored) Danske Bank cheque for £219.
Our time as volunteers may be free, but our servers aren’t, so the larger and richer charities that use our services help contribute to our hosting costs. Most send money digitally, but some use dual-signatory accounts that require they send cheques.

Today, for example, I pulled the recent dead-tree-type post out of our intray and filtered it. Somebody’s got to do these admin tasks, and today I figured it should be me.

Improving our contact form

When people are stuck with Three Rings, or considering using it, or have feature suggestions, or anything else, we encourage them to fill in our contact form. The results of that make their way into our ticketing system where Support Team volunteers help people with whatever it is they need7. They asked if they could have a Slack notification when the form was filled, to grab their attention all the quicker if they were already online, so I obliged and added one.

A filled contact form alongside a Slack message and a resulting ticketing system message.
Adding the Slack notifications meant writing some WordPress PHP code, which feels closer to my “day job” than my Three Rings volunteering!

That quick improvement done, it was time to move on to a task both bigger and more-exciting:

Wrapping up a new feature

I’ve recently been working on an upcoming new feature for Three Rings. Inspired by only occasional user requests, this idea’s been sitting in our (long!) backlog for a while now8: a way to edit the details of volunteers in your organisation in bulk, as though they were in a spreadsheet.

Screenshot showing a web page, 'Three Rings Directory - Edit All': a spreadsheet-like view of multiple volunteers, some of whom have had personal details like First Name, Address, or Telephone changed, resulting in these being highlighted and a prompt appearing to allow the changes to be saved/committed.
The feature’s been almost-complete for weeks now, but I had a few last tweaks to make, based on early feedback and show-and-tell sessions.

I’ve been working on this feature by a series of iterations since the end of October, periodically demonstrating it to other Three Rings volunteers and getting early feedback. In the last round of demonstrations (plus a little user testing, at an in-person Three Rings event) I solicited opinions on how the new feature should be presented, and who it should be made available to9.

So this afternoon I was working on bits of the user interface and, delightfully, got the feature to a point where I only need to update the test instructions and it’ll be ready for a full review and consideration for inclusion in our next milestone release, early next year. Hurrah!

GitHub screenshot showing successful checks passing on a PR.
It’s only a draft PR for now, but at least the initial checks look positive.

Reflecting on my day

I don’t normally get this much Three Rings work done in a day. Even since my sabbatical started, I don’t always get so productive a day10, and when I’m working normally I’d probably only get maybe one or two of these achievements done in a typical week.

So I’m hugely appreciative that my employer encourages staff to take a three-month sabbatical every five years. Because it gives me the opportunity have days like this International Volunteer Day, where I can spend the whole day throwing myself headlong at some valuable volunteering efforts and come out the other side with the satisfaction that I gave my time to make the world a very slightly better place.

If you’re not already volunteering somewhere, I’d highly recommend that you consider it. Volunteering can be purposeful, enriching, and hugely satisfying. Happy International Volunteer Day!

Footnotes

1 In fact, most of the charities who use Three Rings’ services are surprised to discover that we’re a voluntary organisation at all, because we provide the kinds of uptime guarantee, tech support response times, software quality etc. that they might have come to expect from much-richer organisations with a much larger – and paid! – staff. That we’re a voluntary organisation helping voluntary organisations is so unusual that sometimes people have been unsure how to handle us: one time, for example, a helpline charity that was considering making use of us declared us “unsustainable” without a commercial model. At some point in the last decade or two they saw that we’ve outlasted many other services of our type, most of them commercial, and realised: yeah, okay, it turns out we’re in this for the long haul.

2 Gosh: a nine-post day is gonna keep throwing the stats of my recent streak all over the place, isn’t it?

3 My use of italics for Three Rings isn’t arbitrary, but I’ll admit it is confusing. “Three Rings” is the name of our nonprofit. “Three Rings” is the name of the software service we provide. Branding is hard when your company name and product name are the same. And it’s even harder when all of your users insist on abbreviating both to “3R”. 🙄

4 With around 60,000 volunteers depending on Three Rings to coordinate their efforts, the pressure is always on to minimise downtime. I’ve spent many hours over the last few weeks running and re-running through practice runs of the migration strategy before I take the lead on it next week.

5 That said, one of the big things I’ve been pushing for in our new infrastructure is new tools to make it easier for our developers to do “server stuff” like deploying new releases, in an effort to bring us closer to the dream of a continuous integration pipeline. Some day!

6 Or “normal people”, as they might call themselves.

7 The Support Team are a wonderful and hard-working group of volunteers, who aim to reply to every contact within 24 hours, 365 days a year, and  often manage a lot faster than that. They’re at the front-line of what makes Three Rings a brilliant system.

8 While we curate a backlog of user requests and prioritise them based on the optimisation ratio of amount-of-good-done to the amount-of-effort-expected, our developer volunteers enjoy a huge amount of autonomy about what tasks they choose to pick up. It’s not unknown for developers who also volunteer at other organisations (that might be users of Three Rings) to spend a disproportionate amount of time on features that their organisation would benefit from, and that’s fine… so long as the new feature will also benefit at least a large minority of the other organisations that depend on Three Rings. Also, crucially: we try to ensure that new features never inconvenience existing users and the ways in which they work. That’s increasingly challenging in our 22-year-old software tool, but it’s important to us that we’re not like your favourite eCommerce or social networking service that dramatically change their user interface every other year or drop features without warning nor consideration for who might depend upon them.

9 The new feature’s secured such that it works for everybody: if you accessed it as a volunteer with low privileges, you might be able to see virtually nothing about most of the other volunteers and be able to edit only a few details about yourself, for example. But that’d be a pretty-confusing interface, so we concluded that it probably didn’t need to be made available to all volunteers but only those with certain levels of access. We can always revisit later.

10 After all, sometimes I’ve been too busy adding Nex support to CapsulePress, coming up with server naming schemes that don’t suck, teaching myself to like a new flavour or crisps, wearing old clothes, getting thrown off by the combination of timezone moves during daylight savings changes, implementing a 2000-year old algorithm in JavaScript, visiting Spain including PortAventura (and writing more JavaScript for some reason), trying to persuade the children to go to bed, fretting over my sad car, eating quesapizzas, XPath-scraping a friend’s blog, teaching the kids to play Arimaa, enumerating my domain names, playing OpenTTD, not writing a story, considering the twelfth and twentieth amendments, walking the dog in the snow, learning about horse-powered locomotives, teaching my mum to geohash, unlocking graticules, getting caught out by floods, trying to summon a train, geohashing in forests, enjoying gorgeous views, crawling under bridges, getting muddy, boating, in hospital after injuring my arm, or just drinking.

× × × × × × × × ×

Note #25356

For my final bit of Three Rings volunteering this International Volunteer Day, I’m working on improving the UI of a new upcoming feature: a spreadsheet-like page that makes it easy for administrators to edit the details of many volunteers simultaneously (all backed by the usual level of customisation and view/edit permissions that Three Rings is known for).

Screenshot showing a web page, 'Three Rings Directory - Edit All': a spreadsheet-like view of multiple volunteers, some of whom have had personal details like First Name, Address, or Telephone changed, resulting in these being highlighted and a prompt appearing to allow the changes to be saved/committed.

It’s a moderately-popular request, and I can see it being helpful to volunteer coordinators at many different types of voluntary organisation.

Don’t have time to write up the test instructions today, though, because I’ve got to wrap up my volunteering and go do some childwrangling, so that’ll do for now!

×

Note #25347

Even when it’s technical, not all of my International Volunteer Day work for Three Rings has been spent using our key technologies (LNMR [Linux, Nginx, MariaDB, Ruby] stacks).

Today, I wrote some extra PHP for our WordPress-powered contact form to notify our Support Team volunteers via Slack when messages are sent. We already aim to respond to every message within 24 hours, 365 days a year, and are often faster than that… but this might help us to be even more-responsive to the needs of the charities who we help look after.

A filled contact form alongside a Slack message and a resulting ticketing system message.

×

Note #25345

My Three Rings volunteering this International Volunteer Day isn’t all technical work. It’s also time to process the incoming postal mail.

In front of a letterbox, an hand holds an open envelope and a (censored) Danske Bank cheque for £219.

Our time as volunteers may be free, but our servers aren’t, so the larger and richer charities that use our services help contribute to our hosting costs. Most send money digitally, but some use dual-signatory accounts that require they send cheques.

×

Note #25343

As well as the programming tasks I’m working on for Three Rings this International Volunteer Day, I’m also doing a little devops. We’ve got a new server architecture rolling out next week, and I’m tasked with ensuring that the logging on them meets our security standards.

Terminal screenshot showing a directory listing of a logs directory with several gzipped logfiles with different date-stamped suffixes, and the contents of the logrotate configuration file that produced them.

Each server’s on-device logs are retained in date-stamped files for 14 days, but they’re also backed-up offsite daily.

Those bits all seem to be working, so next I need to work out a way to add a notification to our monitoring platform if any server doesn’t successfully push a log to the offsite backup in a timely manner.

×

Note #25341

Not every code review is fireworks, but most Three Rings changes come from the actual needs of the voluntary organisations that Three Rings supports. Some of our users were confused by the way the Admin > Roles page was laid out, so one of our volunteers wrote an improved version.

And because we’re all about collaboration, discussion, learning from one another, and volunteer-empowerment… I’ve added a minor suggestion… but approved their change “with or without” it. I trust my fellow volunteer to either accept my suggestion (if it’s right), reject it (if it’s wrong), or solicit more reviews or bring it to Slack or our fortnightly dev meeting (if it requires discussion).

GitHub screenshot showing a suggested change to a pull request. Dan-Q is suggesting that the new sentence 'These roles belong to .' should not have a full stop at the end, for consistency with other similar headings.

×

Note #25339

Good news! It turns out that the new code to fix the mail merge fields in Three Rings doesn’t introduce an inconsistency with established behaviour. It was important to check, but it turns out all is well.

I touched bases with a fellow volunteer on Slack. Three Rings volunteers primarily communicate via Slack: it helps us to work asynchronously, which supports the fact that our volunteers all have different schedules and preferences. Some might do a couple of evenings a week, others might do the odd weekend, others still might do an occasional intense solid week of volunteering with us and then nothing for months! A communication model that works both synchronously and asynchronously is really important to make that volunteering model work, and Slack fits the bill.

Slack screenshot showing a discussion between Dan Q and another volunteer about whether it's reasonable for a volunteer to be able to send another volunteer an email containing private information accessible only to the second volunteer, so long as it's embedded via a merge field and the actual value is never shown to the first volunteer. Ultimately, Dan suggests that we stick to the established behaviour, which turns out after testing to be that volunteers can't send such private information to one another.

We get together in person sometimes, and we meet on Zoom from time to time too, but Slack is king of communication at Three Rings

×

Note #25337

My first task this International Volunteer Day is to test a pull request that aims to fix a bug with Three Rings’ mail merge fields functionality. As it’s planned to be a hotfix (direct into production) we require extra rigor and more reviewers than code that just goes into the main branch for later testing on our beta environment.

My concern is that fixing this bug might lead to a regression not described by our automated tests, so I’m rolling back to a version from a couple of months ago to compare the behaviour of the affected tool then and now. Sometimes you just need some hands-on testing!

Screenshot showing web application Three Rings viewed on a local development machine, being used to compose an email about International Volunteer Day. A merge field called 'first_name' is included in the email, and it's hand-annotated with the question 'working?'. Alongside, GitHub Desktop shows that we've rolled-back to a revision from two months ago.

×

Note #25334

Happy International Volunteer Day! Volunteering rocks.

As I’m on sabbatical, I’m in the lucky position of being able to spend most of the day on a volunteer project very close to my heart: Three Rings. Three Rings is a 22-year-old web-based service produced by volunteers, for volunteers. The software service we produce supports the efforts of around 60,000 volunteers working at charities and other voluntary organisations around the globe.

Dan, a white man with a goatee-style beard and long hair tied back, wearing a black t-shirt with an abstract pride rainbow, holds a mug of coffee in one hand and gives a thumbs-up with the other. Behind him, three computer screens show a mixture of code, source control, chat, email, and web browser windows. His desk is cluttered.

I’ll be posting throughout the day about some of the different tasks I take on. My volunteer role with Three Rings is primarily a developer/devops one, but it takes all sorts to make a project like this work (even if my posts look biased towards the technical stuff)!

But first: coffee.

×

Stuck in a Lift

Duration

Podcast Version

This post is also available as a podcast. Listen here, download for later, or subscribe wherever you consume podcasts.

I was a small child the first time I got stuck in an elevator. I was always excited by lifts and the opportunity for button-pushing that they provided1, and so I’d run ahead of my mum to get into a lift, at which point the doors closed behind me. The call button on the outside didn’t work for some reason, and I wasn’t tall enough to reach the “open doors” button on the inside. As a result, I was trapped within the elevator until it was called from another floor.

Dan, aged ~4, stands on a railway station platform alongside his mother, yawning, on a bench. It is overcast and drizzly.
The lift I got stuck in as a child wasn’t here at Liskeard Station in Cornwall2. This photo is just to provide a sense of scale about how small I once was.
That time as a small child is, I think, the only time I’ve been stuck in a lift as a result of my own incapability. But my most-memorable getting-stuck-in-a-lift was without a doubt a result of my own stupidity.

How to brake break a lift

Y’see: it turns out that in some lifts, the emergency brakes are sensitive enough that even a little bit of a bounce can cause them to engage. And once they’re locked-on, the lift won’t move – at all – until the brakes are manually released by an engineer.

As I discovered, way back in March 2004.

Screengrab from the third episode of Russian Doll, showing Alan (Charlie Barnett) and Nadia (Natasha Lyonne) standing in a lift whose red emergency lights have come on.
Contrary to what TV and movies will teach you, it’s actually incredibly difficult to make a lift “drop” down its shaft.

On behalf of Three Rings, I was speaking at the 2004 Nightline Association conference. While there, I’d bumped into my friend Fiona, who was also attending the conference3 The conference was taking place on the upper floor of the Manchester University Students Union building, and as the pair of us got into a lift down to the ground floor, I noticed something strange.

“Woah! This lift is really spongy, isn’t it?” I asked, noticing how much the cabin seemed to bounce and sag as we stepped into it.

“Yeah,” said Fiona, shifting her weight to give it an experimental jiggle.

The elevator started to descend, and as it did so we both gave it another gentle bump, mostly (in my case at least) with an experimental mindset: did it only wobble so much when it was stopped at a floor, or did it do it at all times?

It turns out it did so at all times. Except when it bounced between floors, as we were now, the emergency brakes detected this as a problem and locked on. The lift jerked to an immediate halt. We were stuck.

Touchscreen interface for operating a smart lift, housed in the lobby.
I was reminded of my 2004 capture-by-a-lift in a dream the other night, which in turn was probably inspired by Ruth sharing with me her recent experience of using a “smart” lift she found in Dublin.

We shouted for help from people passing on a nearby floor, and they were able to summon assistance from the lift’s maintenance company. Unfortunately, we were told, because it was a weekend we’d likely have to wait around four hours before anybody could get to us, so we’d have to amuse ourselves in the meantime.

The first thing I learned about Fiona that day

That’s when I made the first of two discoveries that I would make, this day, about Fiona. I learned… that she’s mildly claustrophobic. Not enough to stop her from going into a lift, but enough that when she knows she can’t get out of a lift, it’s likely to cause her a problem. I realised that I should try to find a way to distract her from our situation, so I suggested a game.

“How about I-Spy?” I asked, half-jokingly, knowing that this game could surely not occupy us for long within the confines of a small metal box.

“Sure,” she agreed, “You go first.”

Three-storey building on a city street.
The Manchester University Student’s Union building. Image courtesy Peter McDermott, used under a CC-By-SA license.

“I spy with my little eye… something beginning with… N!” I said. If we were going to be stuck here playing I-Spy for several hours, I might as well pick something deviously tricky. Embedded into the corners of the floor were four recessed hexagonal nuts: my word was nut. That’d keep her occupied for a while.

I forget what she guessed and when, but she eventually guessed correctly. It probably took less than 5 minutes. Now it was her turn.

The second thing I learned about Fiona that day

Fiona thought for a little while, looking around our tiny prison for inspiration. Eventually, she’d found something:

“I spy with my little eye,” she said. Then, after a pause: “Something beginning with… S.”

“Screw?” I asked, assuming immediately that she’d have chosen something as devious as I’d thought mine was, and noticing that the button panel was secured with a quartet of recessed flat-head screws. Nope, Fiona indicated.

“Shoes? Oh! Soles?” I suggested, pointing to the bottoms of my shoes, which were visible as I sat on the floor of the lift. Nope.

“Shirt? Socks?” I glanced at myself. I wasn’t sure there was much inside the lift that wasn’t me or Fiona, so it seemed likely that the thing I was looking for was on, or part of, one of us.

“Step?” I gambled, indicating the metal strip that ran underneath the closed doors. No luck.

“Umm… shaft? Can you see part of the lift shaft somehow?” A smirk and an eye roll. I was getting further from the right answer.

Finger pressing a lift button.
It turns out there’s not much to I-Spy in a stopped elevator. “Six? Seven? No… wait… there aren’t that many floors in this building…”

“Ssss….sliding doors?” “Slit?” “Slot?” Still nothing.

This continued for… three… hours4. Fiona sat, self-satisfied, smugly enjoying my increasing frustration right up until the point at which the lift engineer arrived and began levering open the doors on one of the two floors we were between to allow us to wriggle our way out. I must’ve inspected every square centimetre of that tiny space, of myself, and of my gaming companion. Clearly I was alongside the world grandmaster of I-Spy and hadn’t even known it.

“Okay, I give up,” I said, at last. “What the hell was it?”

Soon, I would make the second of the two discoveries I would make about Fiona that day. That she’s quite profoundly dyslexic.

“Circle,” she said, pointing at the lit ring around the alarm button, which we’d pressed some hours before.

Dan, touching his temples, a pained look on his face.
I don’t think it’s possible for a person to spontaneously explode. Because if it were, I’d have done so.

I’d like to think that when Fiona got stuck in a lift a second time that same Spring, it was karma.

Footnotes

1 My obsession with button-pushing as a child also meant that it was hard to snap a photo of me, because I always wanted to be the one to press the shutter button. I’ve written about this previously, if you’d like to see examples of a photos I took as a toddler.

2 The photo is, specifically, Platform 3 of Liskeard Station, which is distinctly separate from the other two platforms, requiring that you leave the main station and cross the road. This is a quirky consequence of the way this section of the Liskeard to Looe branch line was constructed, which necessitated entering Liskeard at right angles to the rest of the station.

3 If I remember rightly, I first met Fiona on a bulletin board when she volunteered to help test Three Rings. She later visited Aberystwyth where she and Kit – who was also helping with the project back in those days – fell in love. It was very sweet.

4 I’d love to say that the three hours flew by, but they didn’t. But it was still infinitely preferable to being stuck in there alone. And, in fact, there are plenty of people for whom I’d have rather been stuck alone than stuck with.

× × × × × ×

Quesapizza Lunch

After a morning of optimising a nonprofit’s reverse proxy configuration, I feel like I’ve earned my lunch! Four cheese, mushroom and jalapeño quesapizzas, mmm…

Gas stovetop.a frying pan contains a tortilla wrap topped with tomato sauce, cheese, mushrooms, and jalapeños. Beside its a plate containing a completed quesapizza: two crispy tortilla wraps sandwiching their contents.

×

Sabbatical Magic

A couple of weeks ago, I kicked off my first sabbatical since starting at Automattic a little over five years ago1.

Dan sits in front of two laptops (one of which shows a photo of an echidna for some reason), in a meeting room full of casually-dressed volunteers.
The first weekend of my sabbatical might have set the tone for a lot of the charity hacking that will follow, being dominated by a Three Rings volunteering weekend.

The first fortnight of my sabbatical has consisted of:

  1. Three Rings CIC’s AGM weekend and lots of planning for the future of the organisation and how we make it a better place to volunteer, and better value for our charity users,
  2. building a first draft of Three Rings’ new server architecture, which turns out to mostly work but still needs some energy thrown at it,
  3. a geohashing expedition with the dog, and
  4. a family holiday to Catalonia, Spain.
Dan, Ruth, and JTA with their children and a tour guide called Julie, enjoying churros in a Barcelona cafe.
You’d be amazed how many churros these children can put away.

The trip to Spain followed a model for European family breaks that we first tried in Paris last year2, but was extended to give us a feel for more of the region than a simple city break would. Ultimately, we ended up in three separate locations:

  1. Barcelona, where we stayed in a wonderful skyscraper hotel with fantastic breakfasts and, after I was able to get enough sleep, explored the obvious touristy bits of the city (e.g. la Sagrada Família3 and other Gaudían architecture, the chocolate museum, the fort at Montjuic, and because it’s me, of course, a widely varied handful of geocaches).
  2. The PortAventura World theme park, whose accommodation was certainly a gear shift after the 5-star hotel we’d come from4 but whose rides kept us and the kids delighted for a couple of days (Shambhala was a particular hit with the eldest kid and me).
  3. A villa in el Vilosell – a village of only 190 people – at which the kids mostly played in the outdoor pool (despite the sometimes pouring rain) but we did get the chance to explore the local area a little. Also, of course, some geocaching: some local caches are 1-2 years old and yet had so few finds that I was able to be only the tenth or even just the third person to sign the logbooks!
Dan and the kids atop the remains of a castle tower.
All that remains of the Castell del Vilosell is part of a single tower, but it affords excellent views over the rest of the village as well as being home to a wonderfully-placed geocache.

I’d known – planned – that my sabbatical would involve a little travel. But it wasn’t until we began to approach the end of this holiday that I noticed a difference that a holiday on sabbatical introduces, compared to any other holiday I’ve taken during my adult life…

Perhaps because of the roles I’ve been appointed to – or maybe as a result of my personality – I’ve typically found that my enjoyment of the last day or two of a week-long trip are marred somewhat by intrusive thoughts of the work week to follow.

Dan sits at a laptop in a hotel bar, a view of Barcelona out of the window behind him, a beer bottle alongside him.
I’m not saying that I didn’t write code while on holiday. I totally did, and I open-sourced it too. But programming feels different when your paycheque doesn’t depend on it.

If I’m back to my normal day job on Monday, then by Saturday I’m already thinking about what I’ll need to be working on (in my case, it’s usually whatever I left unfinished right before I left), contemplating logging-in to work to check my email or Slack, and so on5.

But this weekend, that wasn’t even an option. I’ve consciously and deliberately cut myself off from my usual channels of work communication, and I’ve been very disciplined about not turning any of them back on. And even if I did… my team aren’t expecting me to sign into work for about another 11 weeks anyway!

Dan, standing in an airport departure lounge, mimes "mind blown" to the camera.
🤯🤯🤯

Monday and Tuesday are going to mostly be split between looking after the children, and voluntary work for Three Rings (gotta fix that new server architecture!). Probably. Wednesday? Who knows.

That’s my first taste of the magic of a sabbatical, I think. The observation that it’s possible to unplug from my work life and, y’know, not start thinking about it right away again.

Maybe I can use this as a vehicle to a more healthy work/life balance next year.

Footnotes

1 A sabbatical is a perk offered to Automatticians giving them three months off (with full pay and benefits) after each five years of work. Mine coincidentally came hot on the tail of my last meetup and soon after a whole lot of drama and a major shake-up, so it was a very welcome time to take a break… although of course it’s been impossible to completely detach from bits of the drama that have spilled out onto the open Web!

2 I didn’t get around to writing about Paris, but I did write about how the hotel we stayed at introduced our eldest, and by proxy re-introduced me, to Wonder Boy, ultimately leading to me building an arcade cabinet on which I finally, beat the game, 35 years after first playing it.

3 Whose construction has come on a lot since the last time I toured inside it.

4 Although alcohol helped with that.

5 I’m fully aware that this is a symptom of poor work/life balance, but I’ve got two decades of ingrained bad habits working against me now; don’t expect me to change overnight!

× × × × ×

Bluesky and enshittification

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Any system where users can leave without pain is a system whose owners have high switching costs and whose users have none. An owner who makes a bad call – like removing the block function say, or opting every user into AI training – will lose a lot of users. Not just those users who price these downgrades highly enough that they outweigh the costs of leaving the service. If leaving the service is free, then tormenting your users in this way will visit in swift and devastating pain upon you.

There’s a name for this dynamic, from the world of behavioral economics. It’s called a “Ulysses Pact.” It’s named for the ancient hacker Ulysses, who ignored the normal protocol for sailing through the sirens’ sea. While normie sailors resisted the sirens’ song by filling their ears with wax, Ulysses instead had himself lashed to the mast, so that he could hear the sirens’ song, but could not be tempted into leaping into the sea, to be drowned by the sirens.

Whenever you take a measure during a moment of strength that guards against your own future self’s weakness, you enter into a Ulysses Pact – think throwing away the Oreos when you start your diet.

Wise words from Cory about why he isn’t on Bluesky, which somewhat echo my own experience. If you’ve had the experience in recent memory of abandoning an enshittified Twitter (and if you didn’t yet… why the fuck not?), TikTok, or let’s face it Reddit… and you’ve looked instead to services like Bluesky or arguably Threads… then you haven’t learned your lesson at all.

Freedom to exit is fundamental, and I’m a big fan of systems with a built-in Ulysses Pact. In non-social or unidirectionally-social software it’s sufficient for the tools to be open source: this allows me to host a copy myself if a hosted version falls to enshittification. But for bidirectional social networks, it’s also necessary for them to be federated, so that I’m not disadvantaged by choosing to drop any particular provider in favour of another or my own.

Bluesky keeps promising a proper federation model, but it’s not there yet. And I’m steering clear until it is.

I suppose I also enjoyed this post of Cory’s because it helped remind me of where I myself am failing to apply the Ulysses Pact. Right now, Three Rings is highly-centralised, and while I and everybody else involved with it know our exit strategy should the project have to fold (open source it, help charities migrate to their own instances, etc.) right now that plan is less “tie ourselves to the mast” than it is “trust one another to grab us if we go chasing sirens”. We probably ought to fix that.

Bad Names for Servers

Six or seven years ago our eldest child, then a preschooler, drew me a picture of the Internet1. I framed it and I keep it on the landing outside my bedroom – y’know, in case I get lost on the Internet and need a map:

Framed child-drawn picture showing multiple circles, connected by lines, each with a number 0 or 1 in it.
Lots of circles, all connected to one another, passing zeroes and ones around. Around this time she’d observed that I wrote my number zeroes “programmer-style” (crossed) and clearly tried to emulate this, too.

I found myself reminded of this piece of childhood art when she once again helped me with a network map, this weekend.

As I kick off my Automattic sabbatical I’m aiming to spend some of this and next month building a new server architecture for Three Rings. To share my plans, this weekend, I’d been drawing network diagrams showing my fellow volunteers what I was planning to implement. Later, our eldest swooped in and decided to “enhance” the picture with faces and names for each server:

Network diagram but with entities having faces and named Chungus, Atul, Summer, Gwen, Alice, Astrid, and Demmy.
I don’t think she intended it, but she’s made the primary application servers look the grumpiest. This might well fit with my experience of those servers, too.

I noted that she named the read-replica database server Demmy2, after our dog.

French Bulldog with her tongue sticking out.
You might have come across our dog before, if you followed me through Bleptember.

It’s a cute name for a server, but I don’t think I’m going to follow it. The last thing I want is for her to overhear me complaining about some possible future server problem and misinterpret what I’m saying. “Demmy is a bit slow; can you give her a kick,” could easily cause distress, not to mention “Demmy’s dying; can we spin up a replacement?”

I’ll stick to more-conventional server names for this new cluster, I think.

Footnotes

1 She spelled it “the Itnet”, but still: max props to her for thinking “what would he like a picture of… oh; he likes the Internet! I’ll draw him that!”

2 She also drew ears and a snout on the Demmy-server, in case the identity wasn’t clear to me!

× × ×