Good news! It turns out that the new code to fix the mail merge fields in Three Rings doesn’t introduce an inconsistency with established behaviour. It was important to check, but it
turns out all is well.
I touched bases with a fellow volunteer on Slack. Three Rings volunteers primarily communicate via Slack: it helps us to work asynchronously, which supports the fact that our volunteers
all have different schedules and preferences. Some might do a couple of evenings a week, others might do the odd weekend, others still might do an occasional intense solid week of
volunteering with us and then nothing for months! A communication model that works both synchronously and asynchronously is really important to make that volunteering model work, and
Slack fits the bill.
We get together in person sometimes, and we meet on Zoom from time to time too, but Slack is king of communication at Three Rings
My first task this International Volunteer Day is to test a pull request that aims to fix a bug with Three Rings’ mail merge fields functionality. As it’s planned to be a hotfix (direct
into production) we require extra rigor and more reviewers than code that just goes into the main branch for later testing on our beta environment.
My concern is that fixing this bug might lead to a regression not described by our automated tests, so I’m rolling back to a version from a couple of months ago to compare the behaviour
of the affected tool then and now. Sometimes you just need some hands-on testing!
As I’m on sabbatical, I’m in the lucky position of being able to spend most of the day on a volunteer project very close to my heart: Three
Rings. Three Rings is a 22-year-old web-based service produced by volunteers, for volunteers. The software service we produce supports the efforts of around 60,000 volunteers
working at charities and other voluntary organisations around the globe.
I’ll be posting throughout the day about some of the different tasks I take on. My volunteer role with Three Rings is primarily a developer/devops one, but it takes all sorts to make a
project like this work (even if my posts look biased towards the technical stuff)!
I was a small child the first time I got stuck in an elevator. I was always excited by lifts and the opportunity for button-pushing that they provided1,
and so I’d run ahead of my mum to get into a lift, at which point the doors closed behind me. The call button on the outside didn’t work for some reason, and I wasn’t tall enough to
reach the “open doors” button on the inside. As a result, I was trapped within the elevator until it was called from another floor.
The lift I got stuck in as a child wasn’t here at Liskeard Station in Cornwall2.
This photo is just to provide a sense of scale about how small I once was.
That time as a small child is, I think, the only time I’ve been stuck in a lift as a result of my own incapability. But my most-memorable getting-stuck-in-a-lift was
without a doubt a result of my own stupidity.
How to brake break a lift
Y’see: it turns out that in some lifts, the emergency brakes are sensitive enough that even a little bit of a bounce can cause them to engage. And once they’re locked-on, the lift won’t
move – at all – until the brakes are manually released by an engineer.
As I discovered, way back in March 2004.
Contrary to what TV and movies will teach you, it’s actually incredibly difficult to make a lift “drop” down its shaft.
On behalf of Three Rings, I was speaking at the 2004 Nightline Association conference. While there,
I’d bumped into my friend Fiona, who was also attending the
conference3
The conference was taking place on the upper floor of the Manchester University Students Union building, and as the pair of us got into a lift down to the ground floor, I noticed
something strange.
“Woah! This lift is really spongy, isn’t it?” I asked, noticing how much the cabin seemed to bounce and sag as we stepped into it.
“Yeah,” said Fiona, shifting her weight to give it an experimental jiggle.
The elevator started to descend, and as it did so we both gave it another gentle bump, mostly (in my case at least) with an experimental mindset: did it only wobble so much when it was
stopped at a floor, or did it do it at all times?
It turns out it did so at all times. Except when it bounced between floors, as we were now, the emergency brakes detected this as a problem and locked on. The lift jerked to an
immediate halt. We were stuck.
I was reminded of my 2004 capture-by-a-lift in a dream the other night, which in turn was probably inspired by Ruth sharing with me her
recent experience of using a “smart” lift she found in Dublin.
We shouted for help from people passing on a nearby floor, and they were able to summon assistance from the lift’s maintenance company. Unfortunately, we were told, because it was a
weekend we’d likely have to wait around four hours before anybody could get to us, so we’d have to amuse ourselves in the meantime.
The first thing I learned about Fiona that day
That’s when I made the first of two discoveries that I would make, this day, about Fiona. I learned… that she’s mildly claustrophobic. Not enough to stop her from going into a lift, but
enough that when she knows she can’t get out of a lift, it’s likely to cause her a problem. I realised that I should try to find a way to distract her from our situation, so I
suggested a game.
“How about I-Spy?” I asked, half-jokingly, knowing that this game could surely not occupy us for long within the confines of a small metal box.
“Sure,” she agreed, “You go first.”
The Manchester University Student’s Union building. Image courtesy Peter
McDermott, used under a CC-By-SA license.
“I spy with my little eye… something beginning with… N!” I said. If we were going to be stuck here playing I-Spy for several hours, I might as well pick something deviously tricky.
Embedded into the corners of the floor were four recessed hexagonal nuts: my word was nut. That’d keep her occupied for a while.
I forget what she guessed and when, but she eventually guessed correctly. It probably took less than 5 minutes. Now it was her turn.
The second thing I learned about Fiona that day
Fiona thought for a little while, looking around our tiny prison for inspiration. Eventually, she’d found something:
“I spy with my little eye,” she said. Then, after a pause: “Something beginning with… S.”
“Screw?” I asked, assuming immediately that she’d have chosen something as devious as I’d thought mine was, and noticing that the button panel was secured with a quartet of recessed
flat-head screws. Nope, Fiona indicated.
“Shoes? Oh! Soles?” I suggested, pointing to the bottoms of my shoes, which were visible as I sat on the floor of the lift. Nope.
“Shirt? Socks?” I glanced at myself. I wasn’t sure there was much inside the lift that wasn’t me or Fiona, so it seemed likely that the thing I was looking for was on, or part of,
one of us.
“Step?” I gambled, indicating the metal strip that ran underneath the closed doors. No luck.
“Umm… shaft? Can you see part of the lift shaft somehow?” A smirk and an eye roll. I was getting further from the right answer.
It turns out there’s not much to I-Spy in a stopped elevator. “Six? Seven? No… wait… there aren’t that many floors in this building…”
“Ssss….sliding doors?” “Slit?” “Slot?” Still nothing.
This continued for… three… hours4.
Fiona sat, self-satisfied, smugly enjoying my increasing frustration right up until the point at which the lift engineer arrived and began levering open the doors on one of the two
floors we were between to allow us to wriggle our way out. I must’ve inspected every square centimetre of that tiny space, of myself, and of my gaming companion. Clearly I was alongside
the world grandmaster of I-Spy and hadn’t even known it.
“Okay, I give up,” I said, at last. “What the hell was it?”
Soon, I would make the second of the two discoveries I would make about Fiona that day. That she’s quite profoundly dyslexic.
“Circle,” she said, pointing at the lit ring around the alarm button, which we’d pressed some hours before.
I don’t think it’s possible for a person to spontaneously explode. Because if it were, I’d have done so.
1 My obsession with button-pushing as a child also meant that it was hard to snap a photo
of me, because I always wanted to be the one to press the shutter button. I’ve written about this previously, if you’d like to see
examples of a photos I took as a toddler.
2 The photo is, specifically, Platform 3 of Liskeard Station, which is distinctly separate
from the other two platforms, requiring that you leave the main station and cross the road. This is a quirky consequence of the way this section of the Liskeard to Looe branch line was constructed, which necessitated entering Liskeard at
right angles to the rest of the station.
3 If I remember rightly, I first met Fiona on a bulletin board when she volunteered to
help test Three Rings. She later visited Aberystwyth where she and Kit – who was also helping with the project back in those days – fell in love. It was very sweet.
4 I’d love to say that the three hours flew by, but they didn’t. But it was still
infinitely preferable to being stuck in there alone. And, in fact, there are plenty of people for whom I’d have rather been stuck alone than stuck with.
After a morning of optimising a nonprofit’s reverse proxy configuration, I feel like I’ve earned my lunch! Four cheese, mushroom and jalapeño quesapizzas, mmm…
The first weekend of my sabbatical might have set the tone for a lot of the charity hacking that will follow, being dominated by a Three Rings volunteering weekend.
The first fortnight of my sabbatical has consisted of:
Three Rings CIC’s AGM weekend and lots of planning for the future of the organisation and how we make it a better place to volunteer, and better value for our charity users,
You’d be amazed how many churros these children can put away.
The trip to Spain followed a model for European family breaks that we first tried in Paris last year2,
but was extended to give us a feel for more of the region than a simple city break would. Ultimately, we ended up in three separate locations:
The PortAventura World theme park, whose accommodation was certainly a gear shift after the 5-star hotel we’d come from4 but whose rides kept us and the kids delighted for a
couple of days (Shambhala was a particular hit with the eldest kid and me).
A villa in el Vilosell – a village of only 190 people – at which the kids mostly played in the outdoor pool (despite the
sometimes pouring rain) but we did get the chance to explore the local area a little. Also, of course, some geocaching: some local caches are 1-2 years old and yet had so few finds that
I was able to be only the tenth or even just the third person to sign the logbooks!
I’d known – planned – that my sabbatical would involve a little travel. But it wasn’t until we began to approach the end of this holiday that I noticed a difference that a holiday
on sabbatical introduces, compared to any other holiday I’ve taken during my adult life…
Perhaps because of the roles I’ve been appointed to – or maybe as a result of my personality – I’ve typically found that my enjoyment of the last day or two of a week-long trip are
marred somewhat by intrusive thoughts of the work week to follow.
I’m not saying that I didn’t write code while on holiday. I totally did, and I open-sourced it too.
But programming feels different when your paycheque doesn’t depend on it.
If I’m back to my normal day job on Monday, then by Saturday I’m already thinking about what I’ll need to be working on (in my case, it’s usually whatever I left unfinished right before
I left), contemplating logging-in to work to check my email or Slack, and so on5.
But this weekend, that wasn’t even an option. I’ve consciously and deliberately cut myself off from my usual channels of work communication, and I’ve been very disciplined about not
turning any of them back on. And even if I did… my team aren’t expecting me to sign into work for about another 11 weeks anyway!
🤯🤯🤯
Monday and Tuesday are going to mostly be split between looking after the children, and voluntary work for Three Rings (gotta fix that new server architecture!). Probably. Wednesday?
Who knows.
That’s my first taste of the magic of a sabbatical, I think. The observation that it’s possible to unplug from my work life and, y’know, not start thinking about it right away
again.
Maybe I can use this as a vehicle to a more healthy work/life balance next year.
Footnotes
1A sabbatical is a perk offered to
Automatticians giving them three months off (with full pay and benefits) after each five years of work. Mine coincidentally came hot on the tail of my last meetup and soon after a whole lot of drama and a major
shake-up, so it was a very welcome time to take a break… although of course it’s been impossible to completely detach from bits of the drama that have spilled out onto the open
Web!
5 I’m fully aware that this is a symptom of poor work/life balance, but I’ve got two
decades of ingrained bad habits working against me now; don’t expect me to change overnight!
Any system where users can leave without pain is a system whose owners have high switching costs and whose users have none. An owner who makes a bad call – like
removing the block function say, or opting every user into AI training – will lose a lot of users. Not just those users who price these downgrades highly enough that they
outweigh the costs of leaving the service. If leaving the service is free, then tormenting your users in this way will visit in swift and devastating pain upon you.
…
There’s a name for this dynamic, from the world of behavioral economics. It’s called a “Ulysses Pact.” It’s named for the ancient hacker Ulysses, who ignored the normal protocol for
sailing through the sirens’ sea. While normie sailors resisted the sirens’ song by filling their ears with wax, Ulysses instead had himself lashed to the mast, so that he could hear
the sirens’ song, but could not be tempted into leaping into the sea, to be drowned by the sirens.
Whenever you take a measure during a moment of strength that guards against your own future self’s weakness, you enter into a Ulysses Pact – think throwing away the Oreos when you
start your diet.
…
Wise words from Cory about why he isn’t on Bluesky, which somewhat echo my own experience. If you’ve had the experience in recent memory of abandoning an enshittified Twitter (and if
you didn’t yet… why the fuck not?), TikTok, or let’s face it Reddit… and you’ve looked instead to services like Bluesky or arguably Threads… then you haven’t learned your lesson at all.
Freedom to exit is fundamental, and I’m a big fan of systems with a built-in Ulysses Pact. In non-social or unidirectionally-social software it’s sufficient for the tools to be open
source: this allows me to host a copy myself if a hosted version falls to enshittification. But for bidirectional social networks, it’s also necessary for them to be federated,
so that I’m not disadvantaged by choosing to drop any particular provider in favour of another or my own.
Bluesky keeps promising a proper federation model, but it’s not there yet. And I’m steering clear until it is.
I suppose I also enjoyed this post of Cory’s because it helped remind me of where I myself am failing to apply the Ulysses Pact. Right now, Three Rings is highly-centralised, and while I and everybody else involved with it know our exit strategy should the project have to fold (open
source it, help charities migrate to their own instances, etc.) right now that plan is less “tie ourselves to the mast” than it is “trust one another to grab us if we go chasing
sirens”. We probably ought to fix that.
Six or seven years ago our eldest child, then a preschooler, drew me a picture of the Internet1. I framed it and I
keep it on the landing outside my bedroom – y’know, in case I get lost on the Internet and need a map:
Lots of circles, all connected to one another, passing zeroes and ones around. Around this time she’d observed that I wrote my number zeroes “programmer-style” (crossed) and clearly
tried to emulate this, too.
I found myself reminded of this piece of childhood art when she once again helped me with a network map, this weekend.
As I kick off my Automattic sabbatical I’m aiming to spend some of this and next month building a new server architecture for Three Rings. To share my plans, this weekend, I’d
been drawing network diagrams showing my fellow volunteers what I was planning to implement. Later, our eldest swooped in and decided to “enhance” the picture with faces and names for
each server:
I don’t think she intended it, but she’s made the primary application servers look the grumpiest. This might well fit with my experience of those servers, too.
I noted that she named the read-replica database server Demmy2, after our dog.
You might have come across our dog before, if you followed me through Bleptember.
It’s a cute name for a server, but I don’t think I’m going to follow it. The last thing I want is for her to overhear me complaining about some possible future server problem and
misinterpret what I’m saying. “Demmy is a bit slow; can you give her a kick,” could easily cause distress, not to mention “Demmy’s dying; can we spin up a replacement?”
I’ll stick to more-conventional server names for this new cluster, I think.
Footnotes
1 She spelled it “the Itnet”, but still: max props to her for thinking “what would
he like a picture of… oh; he likes the Internet! I’ll draw him that!”
2 She also drew ears and a snout on the Demmy-server, in case the identity wasn’t clear to
me!
Kicking off day #1 of my three-month sabbatical from work at a hotel in Reading, at
a meeting with fellow Three Rings volunteers to discuss our organisational culture and values.
This evening I pushed against my illness-addled brain to try to sit in on the fortnightly Zoom call with the Three Rings dev team.
Unfortunately it seems like the primary symptom of my cold is an inability to string words together.
At one point, I apologised to by colleague “Beff” (I meant “Bev”, but I had just been talking about “Geoff”) that I couldn’t work out how to stop “scaring my screen” (well, I suppose
Halloween is coming up…). Then, realising my mistake, explained that it was a bit of a “ting-twister”.
Three Rings operates a Web contact form to help people get in touch with us: the idea
is that it provides a quick and easy way to reach out if you’re a charity who might be able to make use of the system, a user who’s having difficulty with the features of the software,
or maybe a potential new volunteer willing to give your time to the project.
But then the volume of spam it received increased dramatically. We don’t want our support team volunteers to spend all
their time categorising spam: even if it doesn’t take long, it’s demoralising. So what could we do?
It’s clearly spam, but if it takes you 2 seconds to categorise it and there are 30 in your Inbox, that’s still a drag.
Our conventional antispam tools are configured pretty liberally: we don’t want to reject a contact from a legitimate user just because their message hits lots of scammy keywords (e.g.
if a user’s having difficulty logging in and has copy-pasted all of the error messages they received, that can look a lot like a password reset spoofing scam to a spam filter). And we
don’t want to add a CAPTCHA, because not only do those create a barrier to humans – while not necessarily reducing spam very much, nowadays – they’re often terrible for accessibility,
privacy, or both.
But it didn’t take much analysis to spot some patterns unique to our contact form and the questions it asks that might provide an opportunity. For example, we discovered that
spam messages would more-often-than-average:
Fill in both the “name” and (optional) “Three Rings username” field with the same value. While it’s cetainly possible for Three Rings users to have
a login username that’s identical to their name, it’s very rare. But automated form-fillers seem to disproportionately pair-up these two fields.
Fill the phone number field with a known-fake phone number or a non-internationalised phone number from a country in which we currently support no charities.
Legitimate non-UK contacts tend to put international-format phone numbers into this optional field, if they fill it at all. Spammers often put NANP (North American Numbering Plan)
numbers.
Include many links in the body of the message. A few links, especially if they’re to our services (e.g. when people are asking for help) is not-uncommon in legitimate
messages. Many links, few of which point to our servers, almost certainly means spam.
Choose the first option for the choose -one question “how can we help you?” Of course real humans sometimes pick this option too, but spammers almost always
choose it.
None of these characteristics alone, or any of the half dozen or so others we analysed (including invisible checks like honeypots and IP-based geofencing), are reason to
suspect a message of being spam. But taken together, they’re almost a sure thing.
To begin with, we assigned scores to each characteristic and automated the tagging of messages in our ticketing system with these scores. At this point, we didn’t do anything to block
such messages: we were just collecting data. Over time, this allowed us to find a safe “threshold” score above which a message was certainly spam.
Even when a message fails our customised spam checks, we only ‘soft-block’ it: telling the user their message was rejected and providing suggestions on working around that or emailing
us conventionally. Our experience shows that the spammers aren’t willing to work to overcome this additional hurdle, but on the very rare ocassion a human hits them, they are.
Once we’d found our threshold we were able to engage a soft-block of submissions that exceeded it, and immediately the volume of spam making it to the ticketing system dropped
considerably. Under 70 lines of PHP code (which sadly I can’t share with you) and we reduced our spam rate by over 80% while having, as far as we can see, no impact on the
false-positive rate.
Where conventional antispam solutions weren’t quite cutting it, implementing a few rules specific to our particular use-case made all the difference. Sometimes you’ve just got to roll
your sleeves up and look at the actual data you do/don’t want, and adapt your filters accordingly.
The second of two caches found on a morning walk from the nearby Cambridge Belfry Hotel, where some fellow volunteers and I met yesterday for a meeting. This cache looked so close, but
being on the other side of the A428 meant that my route to get from one to the other side of the trunk road necessitated a long and circuitous route around half a dozen (ill-maintained)
pegasus crossings around the perimeter of two large roundabouts! Thankfully traffic was quiet at this point if a Saturday morning.
Cache itself was worth the effort though. Feels like it’s increasingly rare to find a large, appropriately-camouflaged, well looked-after cache in a nice location, so FP awarded. TFTC!
Even early on a Saturday morning, after a volunteering event the previous day at the hotel across the road, this highly-exposed GZ made me
feel vulnerable! It’s not as though anybody were actually watching me as I stood around nonchalantly at the GZ waiting for an opportunity
to make a search: a couple of shop workers setting up, maybe, and a handful of drivers going past… but what got me was that every time I looked up from my rummaging I spotted, in the
corner of my eye, a police officer standing to attention just on the other side of the car park, staring intently in my direction!
The copper in question, of course, was nothing more than a cardboard cut-out designed to spook shoplifters, but man that’s a chilling thing to spot in your peripheral vision when you’re
rooting around in the bushes for a concealed container in a quiet car park!
Signed the log and took a selfie with my law enforcement friend (attached) before getting back to my day. TFTC!
The elder child and I are staying nearby and couldn’t resist coming to a nearby cache with so many FPs. The name gave us a bit of a clue what we would be looking for but nothing could
have prepared us for for this imaginative and unusual container! FP awarded. Attached is very non-spoiler photo of us with our very own Incy
Wincies. Greetings from Oxfordshire!
RSS
Spotify
Pocket Casts
Apple Podcasts
YouTube
