Yesterday, I fulfilled the primary Three Rings objective I set for myself when I kicked off my sabbatical
twelve weeks ago and migrated the entire application to a new hosting provider (making a stack of related improvements along the way).
Months prior, I was comparing different providers and their relative merits, making sure that our (quirky and specific) needs could be met. Weeks beforehand, I was running a “dry run”
every four or five days, streamlining the process of moving the ~450GB1
of live data while minimising downtime. Days before the event felt like the countdown for a rocket launch, with final preparations underway: reducing DNS time-to-lives, ensuring users
knew about our downtime window, and generally fitting in a little time to panic.
I made reference on International Volunteer Day to how we needed to configure logrotate. When you’re building architecture for a system as gnarly as Three Rings, there’s
about a billion tools that need such careful tweaking2.
The whole operation was amazingly successful. We’d announced an at-risk period of up to six hours and I was anticipating it taking three… but the whole thing was completed within
a downtime window of just two and a half hours. And I fully credit all of the preparation time. It turns out that “measure twice, cut once” is a sensible strategy3.
It’s challenging to pull off a “big”, intensive operation like this in an entirely voluntary operation. I’m not saying I couldn’t have done it were I not on sabbatical, but
it’d certainly have been harder and riskier.
1Three Rings‘ user data is represented by a little under 70GB of MariaDB
databases plus about 380GB of organisational storage: volunteer photos, files, email attachments, and the like. Certainly not massive by comparison to, say, social media sites, search
engines, and larger eCommerce platforms… but large enough that moving it takes a little planning!
2 Okay, a billion tools to configure? That’s an exaggeration. Especially
now: since the architectural changes I’ve put in place this week, for example, production app server builds of Three Rings no require a custom-compiled build of Nginx (yes,
this really was something we used to need).
Today is International Volunteer Day. And because I’m in the middle of my (magical) sabbatical, I’ve had no difficulty dedicating what would have been the entire workday to a variety
of volunteer activities for the benefit of Three Rings, the nonprofit I founded 22 years ago for the purpose of making volunteer management,
and therefore volunteering, easier.
Liveblogging my day
Step one in a highly productive day of tech volunteering is, as you might have expected: coffee.
I’m pretty sure that most folks don’t know what my voluntary work at Three Rings involves1,
and so I decided I’d celebrate this year’s International Volunteer Day by live-blogging what I got up to in a series of notes throughout the day (1, 2, 3, 4, 5, 6, 7, 8)2.
Maybe, I figured, doing so might provide more of an insight into what a developer/devops role at Three Rings looks like.
Regression-testing a fix
My first task for the day related to a bugfix that we’re looking to deploy. Right now, there’s a problem which sometimes stops the “mail merge”
fields in emails sent by Three Rings3.
We have a candidate fix, but because it’s proposed as a hotfix (i.e. for deployment directly to production), it requires a more-thorough review process involving more volunteer
developers than code which will be made available for beta testing first.
My aim with this task was to roll-back to an earlier version of the software, before the bug was first introduced (by fixing something different!), and ensure that the functionality
remained the same as it always had been.
It turned out that everything was alright, so I reported back to my fellow reviewers about how I’d tested and what my results had been. Once some
more eyes have hit the tests and the new/changed code, that’ll hopefully be ready-to-launch.
Three Rings volunteers primarily communicate via Slack: it helps us to work asynchronously, which supports the fact that our volunteers all have different schedules and preferences
for how they plan their volunteering: some spend whole days, some just a few hours now and then. We’re the ultimate “armchair volunteering” opportunity!
We’re all about collaboration, discussion, learning from one another, and volunteer-empowerment, so my suggestions in this case were non-blocking: I trust my fellow volunteer to
either accept my suggestion (if it’s right), reject it (if it’s wrong), or solicit more reviews or bring it to Slack or our fortnightly dev meeting (if it requires discussion).
Preparing new infrastructure
Next week we’re scheduled to do a big migration of server infrastructure to help provide more future growing-room: it’s exciting, but also a little scary4!
For now, though, all I needed to do was to tweak our logrotate and backup configurations in response to testing of our new systems.
Not all of our developer volunteers also wear a “devops” hat, but a few of us do5.
It’s quite a satisfying role – devops can feel like tidying and organising, and just as a physical space can feel clean, simple, and functional when it’s carefully and minimalistically
laid out, a well-organised cluster of servers humming along in exactly the way they should can be a highly-satisfying thing to be responsible for, too.
Sorting the post
It’s not all techy work, though. And while it’s true that Three Rings has a good group of less-nerdy6 people to handle many of the non-programming tasks that you need
to run a voluntary organisation like ours, it’s also true that many of us wear multiple hats and pull our weight in several different roles.
Our time as volunteers may be free, but our servers aren’t, so the larger and richer charities that use our services help contribute to our hosting costs. Most send money digitally,
but some use dual-signatory accounts that require they send cheques.
When people are stuck with Three Rings, or considering using it, or have feature suggestions, or anything else, we encourage them to fill in our contact form. The results of that make their way into our ticketing system where Support Team volunteers help people with whatever it is
they need7.
They asked if they could have a Slack notification when the form was filled, to grab their attention all the quicker if they were already online, so I obliged and added one.
Adding the Slack notifications meant writing some WordPress PHP code, which feels closer to my “day job” than my Three Rings volunteering!
That quick improvement done, it was time to move on to a task both bigger and more-exciting:
Wrapping up a new feature
I’ve recently been working on an upcoming new feature for Three Rings. Inspired by only occasional user requests, this idea’s been sitting in our (long!) backlog for a while
now8:
a way to edit the details of volunteers in your organisation in bulk, as though they were in a spreadsheet.
The feature’s been almost-complete for weeks now, but I had a few last tweaks to make, based on early feedback and show-and-tell sessions.
I’ve been working on this feature by a series of iterations since the end of October, periodically demonstrating it to other Three Rings volunteers and getting early feedback. In the
last round of demonstrations (plus a little user testing, at an in-person Three Rings event) I solicited opinions on how the new feature should be presented, and who it should be made
available to9.
So this afternoon I was working on bits of the user interface and, delightfully, got the feature to a point where I only need to update the test instructions and it’ll be ready for a
full review and consideration for inclusion in our next milestone release, early next year. Hurrah!
It’s only a draft PR for now, but at least the initial checks look positive.
Reflecting on my day
I don’t normally get this much Three Rings work done in a day. Even since my sabbatical started, I don’t always get so productive a day10,
and when I’m working normally I’d probably only get maybe one or two of these achievements done in a typical week.
So I’m hugely appreciative that my employer encourages staff to take a three-month sabbatical every five years. Because it
gives me the opportunity have days like this International Volunteer Day, where I can spend the whole day throwing myself headlong at some valuable volunteering efforts and come out the
other side with the satisfaction that I gave my time to make the world a very slightly better place.
If you’re not already volunteering somewhere, I’d highly recommend that you consider it. Volunteering can be purposeful, enriching, and hugely satisfying. Happy International Volunteer
Day!
Footnotes
1 In fact, most of the charities who use Three Rings’ services are surprised to discover
that we’re a voluntary organisation at all, because we provide the kinds of uptime guarantee, tech support response times, software quality etc. that they might have come to
expect from much-richer organisations with a much larger – and paid! – staff. That we’re a voluntary organisation helping voluntary organisations is so unusual that sometimes people
have been unsure how to handle us: one time, for example, a helpline charity that was considering making use of us declared us “unsustainable” without a commercial model. At some
point in the last decade or two they saw that we’ve outlasted many other services of our type, most of them commercial, and realised: yeah, okay, it turns out we’re in this for the
long haul.
2 Gosh: a nine-post day is gonna keep throwing the stats of my recent streak all over the place, isn’t it?
3 My use of italics for Three Rings isn’t arbitrary, but I’ll admit it is
confusing. “Three Rings” is the name of our nonprofit. “Three Rings” is the name of the software service we provide. Branding is hard when your company name and product name
are the same. And it’s even harder when all of your users insist on abbreviating both to “3R”. 🙄
4 With around 60,000 volunteers depending on Three Rings to coordinate their
efforts, the pressure is always on to minimise downtime. I’ve spent many hours over the last few weeks running and re-running through practice runs of the migration strategy before I
take the lead on it next week.
5 That said, one of the big things I’ve been pushing for in our new infrastructure is new
tools to make it easier for our developers to do “server stuff” like deploying new releases, in an effort to bring us closer to the dream of a continuous integration pipeline. Some
day!
6 Or “normal people”, as they might call themselves.
7 The Support Team are a wonderful and hard-working group of volunteers, who aim to reply
to every contact within 24 hours, 365 days a year, and often manage a lot faster than that. They’re at the front-line of what makes Three Rings a brilliant
system.
8 While we curate a backlog of user requests and prioritise them based on the optimisation
ratio of amount-of-good-done to the amount-of-effort-expected, our developer volunteers enjoy a huge amount of autonomy about what tasks they choose to pick up. It’s not unknown for
developers who also volunteer at other organisations (that might be users of Three Rings) to spend a disproportionate amount of time on features that their
organisation would benefit from, and that’s fine… so long as the new feature will also benefit at least a large minority of the other organisations that depend on
Three Rings. Also, crucially: we try to ensure that new features never inconvenience existing users and the ways in which they work. That’s increasingly challenging in our
22-year-old software tool, but it’s important to us that we’re not like your favourite eCommerce or social networking service that dramatically change their user interface every other
year or drop features without warning nor consideration for who might depend upon them.
9 The new feature’s secured such that it works for everybody: if you accessed it
as a volunteer with low privileges, you might be able to see virtually nothing about most of the other volunteers and be able to edit only a few details about yourself, for example.
But that’d be a pretty-confusing interface, so we concluded that it probably didn’t need to be made available to all volunteers but only those with certain levels of
access. We can always revisit later.
As I’m on sabbatical, I’m in the lucky position of being able to spend most of the day on a volunteer project very close to my heart: Three
Rings. Three Rings is a 22-year-old web-based service produced by volunteers, for volunteers. The software service we produce supports the efforts of around 60,000 volunteers
working at charities and other voluntary organisations around the globe.
I’ll be posting throughout the day about some of the different tasks I take on. My volunteer role with Three Rings is primarily a developer/devops one, but it takes all sorts to make a
project like this work (even if my posts look biased towards the technical stuff)!
It’s been a long day of driving around Ireland, scrambling through forests, navigating to a hashpoint, exploring a medieval castle, dodging the rain, finding a series of geocaches,
getting lost up a hill in the dark, and generally having a kickass time with one of my very favourite people on this earth: my mum.
And now it’s time for a long soak in a hot bath with a pint of the black stuff and my RSS reader for company. A perfect finish.
When my mother proposed that we take a holiday together somewhere, and that I could choose the destination, I started by looking at the Geohashing Expeditions Map.
Where, I wondered, could I find a cluster of mostly-land graticules (“square” degree of latitude and longitude) in which nobody had ever logged a successful expedition?
I’ve been geohashing for ten years now and I’ve never yet scored a “Graticule Unlocked” achievement for being the first to reach any hashpoint in a given graticule.
Over the next week, if the fluctuations of the Dow Jones and the variable Irish weather allow, I’ll be changing that.
Maybe it’s just that my sabbatical is making me pay more attention then usual, but it feels like I’m getting very lucky with nearby geohashpoints lately. Tomorrow’s hashpoint in my
graticule might be achievable!
This is a good omen, perhaps, for next week. Next week my mother and I are going to hop over to the West coast of Ireland where there are several contiguous mostly-land graticules that
have never seen a successful expedition. We could be the first! 🤞
The first weekend of my sabbatical might have set the tone for a lot of the charity hacking that will follow, being dominated by a Three Rings volunteering weekend.
The first fortnight of my sabbatical has consisted of:
Three Rings CIC’s AGM weekend and lots of planning for the future of the organisation and how we make it a better place to volunteer, and better value for our charity users,
You’d be amazed how many churros these children can put away.
The trip to Spain followed a model for European family breaks that we first tried in Paris last year2,
but was extended to give us a feel for more of the region than a simple city break would. Ultimately, we ended up in three separate locations:
The PortAventura World theme park, whose accommodation was certainly a gear shift after the 5-star hotel we’d come from4 but whose rides kept us and the kids delighted for a
couple of days (Shambhala was a particular hit with the eldest kid and me).
A villa in el Vilosell – a village of only 190 people – at which the kids mostly played in the outdoor pool (despite the
sometimes pouring rain) but we did get the chance to explore the local area a little. Also, of course, some geocaching: some local caches are 1-2 years old and yet had so few finds that
I was able to be only the tenth or even just the third person to sign the logbooks!
I’d known – planned – that my sabbatical would involve a little travel. But it wasn’t until we began to approach the end of this holiday that I noticed a difference that a holiday
on sabbatical introduces, compared to any other holiday I’ve taken during my adult life…
Perhaps because of the roles I’ve been appointed to – or maybe as a result of my personality – I’ve typically found that my enjoyment of the last day or two of a week-long trip are
marred somewhat by intrusive thoughts of the work week to follow.
I’m not saying that I didn’t write code while on holiday. I totally did, and I open-sourced it too.
But programming feels different when your paycheque doesn’t depend on it.
If I’m back to my normal day job on Monday, then by Saturday I’m already thinking about what I’ll need to be working on (in my case, it’s usually whatever I left unfinished right before
I left), contemplating logging-in to work to check my email or Slack, and so on5.
But this weekend, that wasn’t even an option. I’ve consciously and deliberately cut myself off from my usual channels of work communication, and I’ve been very disciplined about not
turning any of them back on. And even if I did… my team aren’t expecting me to sign into work for about another 11 weeks anyway!
🤯🤯🤯
Monday and Tuesday are going to mostly be split between looking after the children, and voluntary work for Three Rings (gotta fix that new server architecture!). Probably. Wednesday?
Who knows.
That’s my first taste of the magic of a sabbatical, I think. The observation that it’s possible to unplug from my work life and, y’know, not start thinking about it right away
again.
Maybe I can use this as a vehicle to a more healthy work/life balance next year.
Footnotes
1A sabbatical is a perk offered to
Automatticians giving them three months off (with full pay and benefits) after each five years of work. Mine coincidentally came hot on the tail of my last meetup and soon after a whole lot of drama and a major
shake-up, so it was a very welcome time to take a break… although of course it’s been impossible to completely detach from bits of the drama that have spilled out onto the open
Web!
5 I’m fully aware that this is a symptom of poor work/life balance, but I’ve got two
decades of ingrained bad habits working against me now; don’t expect me to change overnight!
I’m on sabbatical from work right now, so I’m hoping to be able to get out to this hashpoint while the kids are at school.
Expedition
After dropping the kids off at school, the geopup/hashhound and I set out for the hashpoint. Coming up the “short side” of the bridleway from Botley would be a shorter walk, but we
opted to park in Cumnor and come up the “long side” of Harcourt Hill to avoid Oxford’s traffic (and the inevitable fee for parking on the city’s side of the hill).
Harcourt Hill (like my village of Stanton Harcourt) doubtless gets its name from the Harcourt Family, who supported William the Conqueror during his conquest of Great Britain back in
1066 and were ultimately granted huge swathes of land around this part of the world in recognition of their loyalty. To this day, you find “Harcourt” in a lot of place names in this
neck of the woods.
The hashpoint was so easy to find, we almost walked right over it: it’s right in the centre of the footpath/bridleway. Even my dog, who often doesn’t like long walks or muddy paths,
didn’t get a chance to complain before we got there. We arrived at 09:35 and took the requisite photos, which can be found below. We also kept a GPS tracklog and vlogged our experience,
all of which you can see below.
I’ve not properly hashed in a long while, so it was great to get back out there!
What’s the point of this crossing? Do rabbits pass very fast through this junction?
View East from the hashpoint (plus dog).
View West from the hashpoint.
Right in the middle of the circle of uncertainty.
Silly grin/silly tongue-sticking-out.
Six or seven years ago our eldest child, then a preschooler, drew me a picture of the Internet1. I framed it and I
keep it on the landing outside my bedroom – y’know, in case I get lost on the Internet and need a map:
Lots of circles, all connected to one another, passing zeroes and ones around. Around this time she’d observed that I wrote my number zeroes “programmer-style” (crossed) and clearly
tried to emulate this, too.
I found myself reminded of this piece of childhood art when she once again helped me with a network map, this weekend.
As I kick off my Automattic sabbatical I’m aiming to spend some of this and next month building a new server architecture for Three Rings. To share my plans, this weekend, I’d
been drawing network diagrams showing my fellow volunteers what I was planning to implement. Later, our eldest swooped in and decided to “enhance” the picture with faces and names for
each server:
I don’t think she intended it, but she’s made the primary application servers look the grumpiest. This might well fit with my experience of those servers, too.
I noted that she named the read-replica database server Demmy2, after our dog.
You might have come across our dog before, if you followed me through Bleptember.
It’s a cute name for a server, but I don’t think I’m going to follow it. The last thing I want is for her to overhear me complaining about some possible future server problem and
misinterpret what I’m saying. “Demmy is a bit slow; can you give her a kick,” could easily cause distress, not to mention “Demmy’s dying; can we spin up a replacement?”
I’ll stick to more-conventional server names for this new cluster, I think.
Footnotes
1 She spelled it “the Itnet”, but still: max props to her for thinking “what would
he like a picture of… oh; he likes the Internet! I’ll draw him that!”
2 She also drew ears and a snout on the Demmy-server, in case the identity wasn’t clear to
me!
Day #2 of my sabbatical had a morning in which I’ve mostly been roped into some charity-related digital forensics… until I got distracted by dndle.app,
which apparently I accidentally broke yesterday! Move Fast and Fix Things!
Kicking off day #1 of my three-month sabbatical from work at a hotel in Reading, at
a meeting with fellow Three Rings volunteers to discuss our organisational culture and values.
tl;dr: I’m tidying up and consolidating my personal hosting; I’ve made a little progress, but I’ve got a way to go – fortunately I’ve got a sabbatical coming up at
work!
At the weekend, I kicked-off what will doubtless be a multi-week process of gradually tidying and consolidating some of the disparate digital things I run, around the Internet.
I’ve a long-standing habit of having an idea (e.g. gamebook-making tool Twinebook, lockpicking puzzle game Break Into Us, my Cheating Hangman game, and even FreeDeedPoll.org.uk!),
deploying it to one of several servers I run, and then finding it a huge headache when I inevitably need to upgrade or move said server because there’s such an insane diversity of
different things that need testing!
DNDle, my Wordle-clone where you have to guess the Dungeons & Dragons 5e monster’s stat block, is now hosted by GitHub Pages. Also, I
fixed an issue reported a month ago that meant that I was reporting Giant Scorpions as having a WIS of 19 instead of 9.
Abnib, which mostly reminds people of upcoming birthdays and serves as a dumping ground for any Abnib-related shit I produce, is now hosted by
GitHub Pages.
RockMonkey.org.uk, which doesn’t really do much any more, is now hosted by GitHub Pages.
Sour Grapes, the single-page promo for a (remote) murder mystery party I hosted during a COVID lockdown, is now hosted by GitHub
Pages.
A convenience-page for giving lost people directions to my house is now hosted by GitHub Pages.
Dan Q’s Things is now automatically built on a schedule and hosted by GitHub Pages.
Robin’s Improbable Blog, which spun out from 52 Reflect, wasn’t getting enough traffic to justify
“proper” hosting so now it sits in a Docker container on my NAS.
My μlogger server, which records my location based on pings from my phone, has also moved to my NAS. This has broken
Find Dan Q, but I’m not sure if I’ll continue with that in its current form anyway.
All of my various domain/subdomain redirects have been consolidated on, or are in the process of moving to, to a tinyLinode/Akamai
instance. It’s a super simple plain Nginx server that does virtually nothing except redirect people – this is where I’ll park the domains I register but haven’t found a use for yet, in
future.
I was pretty proud of EGXchange.org, but I’ll be first to admit that it’s among the stupider of my throwaway domains.
It turns out GitHub pages is a fine place to host simple, static websites that were open-source already. I’ve been working on improving my understanding of GitHub Actions
anyway as part of what I’ve been doing while wearing my work, volunteering, and personal hats, so switching some static build processes like DNDle’s to GitHub
Actions was a useful exercise.
Stuff I’m still to tidy…
There’s still a few things I need to tidy up to bring my personal hosting situation under control:
DanQ.me
You’re looking at it. But later this year, you might be looking at it… elsewhere?
This is the big one, because it’s not just a WordPress blog: it’s also a Gemini, Spartan, and Gopher server (thanks CapsulePress!), a Finger server, a general-purpose host to a stack of complex stuff only some of which is powered by Bloq (my WordPress/PHP integrations): e.g.
code to generate the maps that appear on my geopositioned posts, code to integrate with the Fediverse, a whole stack of configuration to make my caching work the way I want, etc.
FreeDeedPoll.org.uk
Right now this is a Ruby/Sinatra application, but I’ve got a (long-running) development branch that will make it run completely in the browser, which will further improve privacy, allow
it to run entirely-offline (with a service worker), and provide a basis for new features I’d like to provide down the line. I’m hoping to get to finishing this during my Automattic
sabbatical this winter.
The website’s basically unchanged for most of a decade and a half, and… umm… it looks it!
A secondary benefit of it becoming browser-based, of course, is that it can be hosted as a static site, which will allow me to move it to GitHub Pages too.
When I took over running the world’s geohashing hub from xkcd‘s Randall Munroe (and davean), I flung the site together on whatever hosting I had sitting
around at the time, but that’s given me some headaches. The outbound email transfer agent is a pain, for example, and it’s a hard host on which to apply upgrades. So I want to get that
moved somewhere better this winter too. It’s actually the last site left running on its current host, so it’ll save me a little money to get it moved, too!
Geohashing’s one of the strangest communities I’m honoured to be a part of. So it’d be nice to treat their primary website to a little more respect and attention.
Right now I run this on my NAS, but that turns out to be a pain sometimes because it means that if my home Internet goes down (e.g. thanks to a power cut, which we have from time to time), I lose access to the first and last place I
go on the Internet! So I’d quite like to move that to somewhere on the open Internet. Haven’t worked out where yet.
Next steps
It’s felt good so far to consolidate and tidy-up my personal web hosting (and to rediscover some old projects I’d forgotten about). There’s work still to do, but I’m expecting to spend
a few months not-doing-my-day-job very soon, so I’m hoping to find the opportunity to finish it then!
RSS
Spotify
Pocket Casts
Apple Podcasts
YouTube
