Ahead of schedule on work project. Invited to 2nd COVID jab next week. Spent half of day working on laptop in sunny garden. Parcel arrived from @LEGO_group with Everyone Is Awesome model (pictured).
Today was a Good Day.
Dan Q
Just learned Tuca & Bertie is getting a 2nd season (thanks Adult Swim, no thanks Netflix). Super excited. In anticipation I’m rewatching season 1 of this sort-of “upbeat feminist Bojack Horseman” and (re)loving it.
Among Twitter’s growing list of faults over the years are various examples of its increasing divergence from open Web standards and developer-friendly endpoints. Do you remember when you used to be able to subscribe to somebody’s feed by RSS? When you could see who follows somebody without first logging in? When they were still committed to progressive enhancement and didn’t make your browser download ~5MB of Javascript or else not show any content whatsoever? Feels like a long time ago, now.
But those complaints aside, the thing that bugged me most this week was how much harder they’ve made it to programatically get access to things that are publicly accessible via web pages. Like avatars, for example!
If you’re a human and you want to see the avatar image associated with a given username, you can go to twitter.com/that-username
and – after you’ve waited
a bit for all of the mandatory JavaScript to download and run (I hope you’re not on a metered connection!) – you’ll see a picture of the user, assuming they’ve uploaded one and not made
their profile private. Easy.
If you’re a computer and you want to get the avatar image, it used to be just as easy; just go to
twitter.com/api/users/profile_image/that-username
and you’d get the image. This was great if you wanted to e.g. show a Facebook-style facepile of images of people who’d retweeted your content.
But then Twitter removed that endpoint and required that computers log in to Twitter, so a clever developer made
a service that fetched avatars for you if you went to e.g. twivatar.glitch.com/that-username
.
But then Twitter killed that, too. Because despite what they claimed 5½ years ago, Twitter still clearly hates developers.
Recently, I needed a one-off program to get the avatars associated with a few dozen Twitter usernames.
First, I tried the easy way: find a service that does the work for me. I’d used avatars.io
before but it’s died, presumably because (as I soon discovered) Twitter had made
things unnecessarily hard for them.
Second, I started looking at the Twitter API documentation but it took me in the region of 30-60 seconds before I said “fuck that noise” and decided that the set-up overhead in doing things the official way simply wasn’t justified for my simple use case.
So I decided to just screen-scrape around the problem. If a human can just go to the web page and see the image, a computer pretending to be a human can do exactly the same. Let’s do this:
Obviously, using this code would violate Twitter’s terms of use for automation, so… don’t, I guess?
Given that I only needed to run it once, on a finite list of accounts, I maintain that my approach was probably kinder on their servers than just manually going to every page and saving the avatar from it. But if you set up a service that uses this approach then you’ll certainly piss off somebody at Twitter and history shows that they’ll take their displeasure out on you without warning.
But it works. It was fast and easy and I got what I was looking for.
And the moral of the story is: if you make an API and it’s terrible, don’t be surprised if people screen-scape your service instead. (You can’t spell “scraping” without “API”, amirite?)
This is a repost promoting content originally published elsewhere. See more things Dan's reposted.
I have this exact same ride-on mower, but mine doesn’t do this. I feel cheated.
This weekend, while investigating a bug in some code that generates iCalendar (ICS) feeds, I learned about a weird quirk in the Republic of Ireland’s timezone. It’s such a strange thing (and has so little impact on everyday life) that I imagine that even most Irish people don’t even know about it, but it’s important enough that it can easily introduce bugs into the way that computer calendars communicate:
Most of Europe put their clocks forward in Summer, but the Republic of Ireland instead put their clocks backward in Winter.
If that sounds to you like the same thing said two different ways – or the set-up to a joke! – read on:
After high-speed (rail) travel made mean solar timekeeping problematic, Great Britain in 1880 standardised on Greenwich Mean Time (UTC+0) as the time throughout the island, and Ireland standardised on Dublin Mean Time (UTC-00:25:21). If you took a ferry from Liverpool to Dublin towards the end of the 19th century you’d have to put your watch back by about 25 minutes. With air travel not yet being a thing, countries didn’t yet feel the need to fixate on nice round offsets in the region of one-hour (today, only a handful of regions retain UTC-offsets of half or quarter hours).
That’s all fine in peacetime, but by the First World War and especially following the Easter Rising, the British government decided that it was getting too tricky for their telegraph operators (many of whom operated out of Ireland, which provided an important junction for transatlantic traffic) to be on a different time to London.
So the Time (Ireland) Act 1916 was passed, putting Ireland on Greenwich Mean Time. Ireland put her clocks back by 35 minutes and synched-up with the rest of the British Isles. And from then on, everything was simple and because nothing ever went wrong in Ireland as a result of the way it was governed by by Britain, nobody ever had to think about the question of timezones on the island again.Ah. Hmm.
Following Irish independence, the keeping of time carried on in much the same way for a long while, which will doubtless have been convenient for families spread across the Northern Irish border. But then came the Second World War.
Summers in the 1940s saw Churchill introduce Double Summer Time which he believed would give the UK more daylight, saving energy that might otherwise be used for lighting and increasing production of war materiel.
Ireland considered using the emergency powers they’d put in place to do the same, as a fuel saving measure… but ultimately didn’t. This was possibly because aligning her time with Britain might be seen as undermining her neutrality, but was more likely because the government saw that such a measure wouldn’t actually have much impact on fuel use (it certainly didn’t in Britain). Whatever the reason, though, Britain and Northern Ireland were again out-of-sync with one another until the war ended.
From 1968 to 1971 Britain experimented with “British Standard Time” – putting the clocks forward in Summer once, to UTC+1, and then leaving them there for three years. This worked pretty well except if you were Scottish in which case you’ll have found winter mornings to be even gloomier than you were used to, which was already pretty gloomy. Conveniently: during much of this period Ireland was also on UTC+1, but in their case it was part of a different experiment. Ireland were working on joining the European Economic Community, and aligning themselves with “Paris time” year-round was an unnecessary concession but an interesting idea.
But here’s where the quirk appears: the Standard Time Act 1968, which made UTC+1 the “standard” timezone for the Republic of Ireland, was not repealed and is still in effect. Ireland could have started over in 1971 with a new rule that made UTC+0 the standard and added a “Summer Time” alternative during which the clocks are put forward… but instead the Standard Time (Amendment) Act 1971 left UTC+1 as Ireland’s standard timezone and added a “Winter Time” alternative during which the clocks are put back.
(For a deeper look at the legal history of time in the UK and Ireland, see this timeline. Certainly don’t get all your history lessons from me.)
You might rightly be thinking: so what! Having a standard time of UTC+0 and going forward for the Summer (like the UK), is functionally-equivalent to having a standard time of UTC+1 and going backwards in the Winter, like Ireland, right? It’s certainly true that, at any given moment, a clock in London and a clock in Dublin should show the same time. So why would anybody care?
But declaring which is “standard” is important when you’re dealing with computers. If, for example, you run a volunteer rota management system that supports a helpline charity that has branches in both the UK and Ireland, then it might really matter that the computer systems involved know what each other mean when they talk about specific times.
The author of an iCalendar file can choose to embed timezone information to explain what, in that file, a particular timezone means. That timezone information might say, for example, “When I say ‘Europe/Dublin’, I mean UTC+1, or UTC+0 in the winter.” Or it might say – like the code above! – “When I say ‘Europe/Dublin’, I mean UTC+0, or UTC+1 in the summer.” Both of these declarations would be technically-valid and could be made to work, although only the first one would be strictly correct in accordance with the law.
But if you don’t include timezone information in your iCalendar file, you’re relying on the feed subscriber’s computer (e.g. their calendar software) to make a sensible interpretation.. And that’s where you run into trouble. Because in cases like Ireland, for which the standard is one thing but is commonly-understood to be something different, there’s a real risk that the way your system interprets and encodes time won’t necessarily be the same as the way somebody else’s does.
If I say I’ll meet you at 12:00 on 1 January, in Ireland, you rightly need to know whether I’m talking about 12:00 in Irish “standard” time (i.e. 11:00, because daylight savings are in effect) or 12:00 in local-time-at-the-time-of-the-meeting (i.e. 12:00). Humans usually mean the latter because we think in terms of local time, but when your international computer system needs to make sure that people are on a shift at the same time, but in different timezones, it needs to be very clear what exactly it means!
And when your daylight savings works “backwards” compared to everybody else’s… that’s sure to make a developer somewhere cry. And, possibly, blog about your weird legislation.
This checkin to GC8W7QW Forgotten Bridge reflects a geocaching.com log entry. See more of Dan's cache logs.
The rain finally stopped this afternoon so I figured I’d take my next Zoom meeting outdoors with me and stretch my legs while I talked to work colleagues. And so I went, selfie-stick ahead of me and chatting to teammates in New York and Florence (what a world we live in!), out for a ramble and soon remembered that I was carrying Duck Race / Mustache Pink, a travel bug I’d picked up near Lands End this weekend. So I diverted my walk to come and check up on this cache (it’s looking fine!) and drop off the TB for the next leg of its journey!
This checkin to GC1VQ7G Pooh Sticks Bridge reflects a geocaching.com log entry. See more of Dan's cache logs.
Coordinates brought me exactly to a tree that would match the hint… except for the fact that it’s been recently felled! (Picture attached.) No sign of cache, and anything else nearby that would fit the hint is on very-clearly-private land, so I’m concerned this cache might have vanished. :-(
Cheered myself up with a quick game of pooh sticks. I won, but that’s to be expected when you play solo.
This checkin to GC5J6JR Would you believe another Almost Motorway Mayhem? reflects a geocaching.com log entry. See more of Dan's cache logs.
Stopped at the nearby services on a long journey from dropping my partner’s brother and his boat off Lands End (I can just about see my car from near the GZ: photo attached) for a hot drink and to remotely participate in a work meeting. Meeting’s not starting yet so I walked out the services’ staff exit and came up here to find this cache. Easy find, TFTC.
This checkin to GC4NTRC Motorway Mayhem M5 Michaelwood Northbound reflects a geocaching.com log entry. See more of Dan's cache logs.
Stopped at the services on my way back to Oxford from Lands End, where I was dropping my partner’s brother and his skiff into the sea to begin his attempt to row the length of the UK! The boat trailer is wobbling in a curious way so I’ve been driving extra carefully, so it’s been a long journey so far (and I’ve still got the A40 to tackle!) so the opportunity for a break is a welcome one.
Cache was easy to sight – with the hint – and stealthing around the nearby truckers wasn’t hard, but prickly plants made retrieving the container a little challenging. Wish I’d brought gloves! SL, TFTC.
This checkin to GCYK2M Got A Light Boy ? Longships Lighthouse reflects a geocaching.com log entry. See more of Dan's cache logs.
I just launched my partner’s brother – shown in free attached picture – out in his rowboat to begin his attempt to row from Land’s End to John O’ Groats. Naturally this first involves rowing South, around the headland and past the lighthouse, to get to Land’s End! So I came up the hill to watch him get started. And while I was at it, I figured I’d find this cache! Took travel bug. TFTC!
This checkin to GC6VG1N "BOSISTOW FARM" reflects a geocaching.com log entry. See more of Dan's cache logs.
I arrived yesterday at nearby Raftra Farm for a weekend, mostly to launch my partner’s brother into the sea to begin his attempt to row from Land’s End to John O’ Groats (making use of inland waterways as much as possible). After a bit of a lie-in this morning, I came out for a brief walk and to find this geocache. Probably this’ll be a highlight of my day, as much of the rest of it will be dominated by catching up on the work I didn’t get done yesterday (during the drive down here from Oxford), at least until the afternoon tide turns which is when we’re doing the first launch!
Easy to find cache hidden in the most likely location – I maintain one just like this near my old house North of Oxford! TFTC.
This checkin to GC5VXQ3 Motorway Mayhem - M5 - Gloucester Services (South) reflects a geocaching.com log entry. See more of Dan's cache logs.
A quick and easy find (though I was glad of the hint when I approached the obstacle at the GZ) while travelling from Oxford to Cornwall to dump my partner’s brother in the sea for the start of his personal challenge to row the length of the UK. (Photo of our boat in tow attached!)
At the weekend, Ruth‘s brother Robin begins his attempt to row the length of the UK (by inland waterways where possible). I just received this invitation to see him off…
This checkin to GC90RH3 Tiny Log Book reflects a geocaching.com log entry. See more of Dan's cache logs.
I was in the area anyway so, following a recent DNF, I checked up on this cache. It’s in perfect condition (though I did have to empty a woodlouse out of the outer cache container) and ready to find (previous logger was probably looking in the wrong place: there’s no risk of touching a stinging plant to get this cache!).