Note #18760

Everyone Is Awesome LEGO mode showing BIPOC/LGBTQ+ colours.

Ahead of schedule on work project. Invited to 2nd COVID jab next week. Spent half of day working on laptop in sunny garden. Parcel arrived from @LEGO_group with Everyone Is Awesome model (pictured).

Today was a Good Day.

×

Getting Twitter Avatars (without the Twitter API)

Among Twitter’s growing list of faults over the years are various examples of its increasing divergence from open Web standards and developer-friendly endpoints. Do you remember when you used to be able to subscribe to somebody’s feed by RSS? When you could see who follows somebody without first logging in? When they were still committed to progressive enhancement and didn’t make your browser download ~5MB of Javascript or else not show any content whatsoever? Feels like a long time ago, now.

Lighthouse Performance score for Twitter's Twitter account page on mobile, scoring 50%.
For one of the most-popular 50 websites in the world, this score is frankly shameful.

But those complaints aside, the thing that bugged me most this week was how much harder they’ve made it to programatically get access to things that are publicly accessible via web pages. Like avatars, for example!

If you’re a human and you want to see the avatar image associated with a given username, you can go to twitter.com/that-username and – after you’ve waited a bit for all of the mandatory JavaScript to download and run (I hope you’re not on a metered connection!) – you’ll see a picture of the user, assuming they’ve uploaded one and not made their profile private. Easy.

If you’re a computer and you want to get the avatar image, it used to be just as easy; just go to twitter.com/api/users/profile_image/that-username and you’d get the image. This was great if you wanted to e.g. show a Facebook-style facepile of images of people who’d retweeted your content.

But then Twitter removed that endpoint and required that computers log in to Twitter, so a clever developer made a service that fetched avatars for you if you went to e.g. twivatar.glitch.com/that-username.

But then Twitter killed that, too. Because despite what they claimed 5½ years ago, Twitter still clearly hates developers.

Dan Q's Twitter profile header showing his avatar image.
You want to that image? Well you’ll need a Twitter account, a developer account, an OAuth token set, a stack of code…

Recently, I needed a one-off program to get the avatars associated with a few dozen Twitter usernames.

First, I tried the easy way: find a service that does the work for me. I’d used avatars.io before but it’s died, presumably because (as I soon discovered) Twitter had made things unnecessarily hard for them.

Second, I started looking at the Twitter API documentation but it took me in the region of 30-60 seconds before I said “fuck that noise” and decided that the set-up overhead in doing things the official way simply wasn’t justified for my simple use case.

So I decided to just screen-scrape around the problem. If a human can just go to the web page and see the image, a computer pretending to be a human can do exactly the same. Let’s do this:

/* Copyright (c) 2021 Dan Q; released under the MIT License. */

const Puppeteer = require('puppeteer');

getAvatar = async (twitterUsername) => {
  const browser = await Puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox']});
  const page = await browser.newPage();
  await page.goto(`https://twitter.com/${twitterUsername}`);
  await page.waitForSelector('a[href$="/photo"] img[src]');
  const url = await page.evaluate(()=>document.querySelector('a[href$="/photo"] img').src);
  await browser.close();
  console.log(`${twitterUsername}: ${url}`);
};

process.argv.slice(2).forEach( twitterUsername => getAvatar( twitterUsername.toLowerCase() ) );
The code is ludicrously simple. It took less time, energy, and code to write this than to follow Twitter’s “approved” procedure. You can download the code via Gist.

Obviously, using this code would violate Twitter’s terms of use for automation, so… don’t, I guess?

Given that I only needed to run it once, on a finite list of accounts, I maintain that my approach was probably kinder on their servers than just manually going to every page and saving the avatar from it. But if you set up a service that uses this approach then you’ll certainly piss off somebody at Twitter and history shows that they’ll take their displeasure out on you without warning.

$ node get-twitter-avatar.js alexsdutton richove geohashing TailsteakAD LilFierce1 ninjanails
alexsdutton: https://pbs.twimg.com/profile_images/740505937039986688/F9gUV0eK_200x200.jpg
lilfierce1: https://pbs.twimg.com/profile_images/1189417235313561600/AZ2eLjAg_200x200.jpg
richove: https://pbs.twimg.com/profile_images/1576438972/2011_My_picture4_200x200.jpeg
geohashing: https://pbs.twimg.com/profile_images/877137707939581952/POzWWV2d_200x200.jpg
ninjanails: https://pbs.twimg.com/profile_images/1146364466801577985/TvCfb49a_200x200.jpg
tailsteakad: https://pbs.twimg.com/profile_images/1118738807019278337/y5WWkLbF_200x200.jpg
This output shows the avatar URLs of a half a dozen Twitter accounts. It took minutes to write the code and takes seconds to run, but if I’d have done it the “right” way I’d still be unnecessarily wading through Twitter’s sprawling documentation.

But it works. It was fast and easy and I got what I was looking for.

And the moral of the story is: if you make an API and it’s terrible, don’t be surprised if people screen-scape your service instead. (You can’t spell “scraping” without “API”, amirite?)

× ×

Note #18572

Hey @LloydsBank! 2009 called and asked if you’re done sending your customers links to unencrypted HTTP endpoints yet. How do you feel about switching this to a HTTPS link rather than relying on an interceptable/injectable HTTP request?

Text message: "Follow this link to download your free Lloyds Bank Mobile Banking app. http://www.lloydsbank.com/mobileapp"

×

Ireland and the UK Aren’t In The Same Timezone!

This weekend, while investigating a bug in some code that generates iCalendar (ICS) feeds, I learned about a weird quirk in the Republic of Ireland’s timezone. It’s such a strange thing (and has so little impact on everyday life) that I imagine that even most Irish people don’t even know about it, but it’s important enough that it can easily introduce bugs into the way that computer calendars communicate:

Most of Europe put their clocks forward in Summer, but the Republic of Ireland instead put their clocks backward in Winter.

If that sounds to you like the same thing said two different ways – or the set-up to a joke! – read on:

Map showing timezones of Europe. The UK and Ireland are grouped (along with Iceland) in a zone labelled as being UTC+0.
The timezones of Europe look pretty simple compared to some parts of the world, but the illustration of the British Isles hides an interesting eccentricity.

A Brief History of Time (in Ireland)

Poster titled "Time (Ireland) Act 1916", advising that "On and after Sunday 1st October 1916 Western European Time will be ovserved throughout Ireland" asking people to set their clocks and watches back 35 minutes.
Spring forward, fall back… just a little bit back, though. Not too much.

After high-speed (rail) travel made mean solar timekeeping problematic, Great Britain in 1880 standardised on Greenwich Mean Time (UTC+0) as the time throughout the island, and Ireland standardised on Dublin Mean Time (UTC-00:25:21). If you took a ferry from Liverpool to Dublin towards the end of the 19th century you’d have to put your watch back by about 25 minutes. With air travel not yet being a thing, countries didn’t yet feel the need to fixate on nice round offsets in the region of one-hour (today, only a handful of regions retain UTC-offsets of half or quarter hours).

That’s all fine in peacetime, but by the First World War and especially following the Easter Rising, the British government decided that it was getting too tricky for their telegraph operators (many of whom operated out of Ireland, which provided an important junction for transatlantic traffic) to be on a different time to London.

1885 GPO telegraph instrument from the Porthcurno Telegraph Museum, which Dan almost visited the other week but it was closed.
It’s widely believed that the world’s first “U UP? [STOP]” message never got a response as a direct result of Anglo-Irish timezone confusion.
So the Time (Ireland) Act 1916 was passed, putting Ireland on Greenwich Mean Time. Ireland put her clocks back by 35 minutes and synched-up with the rest of the British Isles. And from then on, everything was simple and because nothing ever went wrong in Ireland as a result of the way it was governed by by Britain, nobody ever had to think about the question of timezones on the island again.

Ah. Hmm.

December 1920 photograph showing St Patrick's Street, Cork, following the burning of the city by British forces.
“Those Irish people want to govern their own country, do they? After we so kindly shared our king with them? Right-ho: let’s set fire to their cities and see how they feel then.”

Following Irish independence, the keeping of time carried on in much the same way for a long while, which will doubtless have been convenient for families spread across the Northern Irish border. But then came the Second World War.

Summers in the 1940s saw Churchill introduce Double Summer Time which he believed would give the UK more daylight, saving energy that might otherwise be used for lighting and increasing production of war materiel.

Ireland considered using the emergency powers they’d put in place to do the same, as a fuel saving measure… but ultimately didn’t. This was possibly because aligning her time with Britain might be seen as undermining her neutrality, but was more likely because the government saw that such a measure wouldn’t actually have much impact on fuel use (it certainly didn’t in Britain). Whatever the reason, though, Britain and Northern Ireland were again out-of-sync with one another until the war ended.

Newspaper clipping advising that "Double Summer Time comes to an end on Saturday night, August 8-9, when all clocks and watches should be put back one hour, thus reverting to British Summer Time, which will probably be maintained throughout the winter."
I like to imagine that the development of powerful computers by the folks at Bletchley Park was a result of needing to keep track of timezones across the British Isles.

From 1968 to 1971 Britain experimented with “British Standard Time” – putting the clocks forward in Summer once, to UTC+1, and then leaving them there for three years. This worked pretty well except if you were Scottish in which case you’ll have found winter mornings to be even gloomier than you were used to, which was already pretty gloomy. Conveniently: during much of this period Ireland was also on UTC+1, but in their case it was part of a different experiment. Ireland were working on joining the European Economic Community, and aligning themselves with “Paris time” year-round was an unnecessary concession but an interesting idea.

But here’s where the quirk appears: the Standard Time Act 1968, which made UTC+1 the “standard” timezone for the Republic of Ireland, was not repealed and is still in effect. Ireland could have started over in 1971 with a new rule that made UTC+0 the standard and added a “Summer Time” alternative during which the clocks are put forward… but instead the Standard Time (Amendment) Act 1971 left UTC+1 as Ireland’s standard timezone and added a “Winter Time” alternative during which the clocks are put back.

Two clocks, both showing the same time. One has a sign reading "LONDON", the other "DUBLIN, I GUESS?"
It all seems so simple until you actually think about it.

(For a deeper look at the legal history of time in the UK and Ireland, see this timeline. Certainly don’t get all your history lessons from me.)

So what?

You might rightly be thinking: so what! Having a standard time of UTC+0 and going forward for the Summer (like the UK), is functionally-equivalent to having a standard time of UTC+1 and going backwards in the Winter, like Ireland, right? It’s certainly true that, at any given moment, a clock in London and a clock in Dublin should show the same time. So why would anybody care?

Perl Data::ICal::TimeZone implementation of Dublin timezone, incorrectly showing summer DST at +1 rather than winter DST of -1.
This code for Europe/Dublin, from the Perl module Data::ICal::TimeZone, is technically-incorrect because it states that the winter time is the standard and daylight savings of +1 hour apply in the summer, rather than the opposite.

But declaring which is “standard” is important when you’re dealing with computers. If, for example, you run a volunteer rota management system that supports a helpline charity that has branches in both the UK and Ireland, then it might really matter that the computer systems involved know what each other mean when they talk about specific times.

The author of an iCalendar file can choose to embed timezone information to explain what, in that file, a particular timezone means. That timezone information might say, for example, “When I say ‘Europe/Dublin’, I mean UTC+1, or UTC+0 in the winter.” Or it might say – like the code above! – “When I say ‘Europe/Dublin’, I mean UTC+0, or UTC+1 in the summer.” Both of these declarations would be technically-valid and could be made to work, although only the first one would be strictly correct in accordance with the law.

Stressed programmer hunched over a MacBook. Photo by Anna Shvets from Pexels.
Clients who need solid timezone support represent 50% of a programmer’s production of stress hormones. See also Falsehoods Programmers Believe About Time.

But if you don’t include timezone information in your iCalendar file, you’re relying  on the feed subscriber’s computer (e.g. their calendar software) to make a sensible interpretation.. And that’s where you run into trouble. Because in cases like Ireland, for which the standard is one thing but is commonly-understood to be something different, there’s a real risk that the way your system interprets and encodes time won’t necessarily be the same as the way somebody else’s does.

If I say I’ll meet you at 12:00 on 1 January, in Ireland, you rightly need to know whether I’m talking about 12:00 in Irish “standard” time (i.e. 11:00, because daylight savings are in effect) or 12:00 in local-time-at-the-time-of-the-meeting (i.e. 12:00). Humans usually mean the latter because we think in terms of local time, but when your international computer system needs to make sure that people are on a shift at the same time, but in different timezones, it needs to be very clear what exactly it means!

And when your daylight savings works “backwards” compared to everybody else’s… that’s sure to make a developer somewhere cry. And, possibly, blog about your weird legislation.

× × × × × × × ×

Dan Q performed maintenance for GC8W7QW Forgotten Bridge

This checkin to GC8W7QW Forgotten Bridge reflects a geocaching.com log entry. See more of Dan's cache logs.

The rain finally stopped this afternoon so I figured I’d take my next Zoom meeting outdoors with me and stretch my legs while I talked to work colleagues. And so I went, selfie-stick ahead of me and chatting to teammates in New York and Florence (what a world we live in!), out for a ramble and soon remembered that I was carrying Duck Race / Mustache Pink, a travel bug I’d picked up near Lands End this weekend. So I diverted my walk to come and check up on this cache (it’s looking fine!) and drop off the TB for the next leg of its journey!

Geocacher with a selfie-stick

×

Dan Q couldn’t find GC1VQ7G Pooh Sticks Bridge

This checkin to GC1VQ7G Pooh Sticks Bridge reflects a geocaching.com log entry. See more of Dan's cache logs.

Coordinates brought me exactly to a tree that would match the hint… except for the fact that it’s been recently felled! (Picture attached.) No sign of cache, and anything else nearby that would fit the hint is on very-clearly-private land, so I’m concerned this cache might have vanished. :-(

Cheered myself up with a quick game of pooh sticks. I won, but that’s to be expected when you play solo.

×

Dan Q found GC5J6JR Would you believe another Almost Motorway Mayhem?

This checkin to GC5J6JR Would you believe another Almost Motorway Mayhem? reflects a geocaching.com log entry. See more of Dan's cache logs.

Stopped at the nearby services on a long journey from dropping my partner’s brother and his boat off Lands End (I can just about see my car from near the GZ: photo attached) for a hot drink and to remotely participate in a work meeting. Meeting’s not starting yet so I walked out the services’ staff exit and came up here to find this cache. Easy find, TFTC.

×

Dan Q found GC4NTRC Motorway Mayhem M5 Michaelwood Northbound

This checkin to GC4NTRC Motorway Mayhem M5 Michaelwood Northbound reflects a geocaching.com log entry. See more of Dan's cache logs.

Stopped at the services on my way back to Oxford from Lands End, where I was dropping my partner’s brother and his skiff into the sea to begin his attempt to row the length of the UK! The boat trailer is wobbling in a curious way so I’ve been driving extra carefully, so it’s been a long journey so far (and I’ve still got the A40 to tackle!) so the opportunity for a break is a welcome one.

Cache was easy to sight – with the hint – and stealthing around the nearby truckers wasn’t hard, but prickly plants made retrieving the container a little challenging. Wish I’d brought gloves! SL, TFTC.

Dan Q found GCYK2M Got A Light Boy ? Longships Lighthouse

This checkin to GCYK2M Got A Light Boy ? Longships Lighthouse reflects a geocaching.com log entry. See more of Dan's cache logs.

I just launched my partner’s brother – shown in free attached picture – out in his rowboat to begin his attempt to row from Land’s End to John O’ Groats. Naturally this first involves rowing South, around the headland and past the lighthouse, to get to Land’s End! So I came up the hill to watch him get started. And while I was at it, I figured I’d find this cache! Took travel bug. TFTC!

Robin rows away from Sennen Cove and towards St. Ives

×

Dan Q found GC6VG1N “BOSISTOW FARM”

This checkin to GC6VG1N "BOSISTOW FARM" reflects a geocaching.com log entry. See more of Dan's cache logs.

I arrived yesterday at nearby Raftra Farm for a weekend, mostly to launch my partner’s brother into the sea to begin his attempt to row from Land’s End to John O’ Groats (making use of inland waterways as much as possible). After a bit of a lie-in this morning, I came out for a brief walk and to find this geocache. Probably this’ll be a highlight of my day, as much of the rest of it will be dominated by catching up on the work I didn’t get done yesterday (during the drive down here from Oxford), at least until the afternoon tide turns which is when we’re doing the first launch!

Easy to find cache hidden in the most likely location – I maintain one just like this near my old house North of Oxford! TFTC.

Dan, on a country lane in Cornwall, in front of a bright blue sky, waves to the camera.

×

Dan Q found GC5VXQ3 Motorway Mayhem – M5 – Gloucester Services (South)

This checkin to GC5VXQ3 Motorway Mayhem - M5 - Gloucester Services (South) reflects a geocaching.com log entry. See more of Dan's cache logs.

A quick and easy find (though I was glad of the hint when I approached the obstacle at the GZ) while travelling from Oxford to Cornwall to dump my partner’s brother in the sea for the start of his personal challenge to row the length of the UK. (Photo of our boat in tow attached!)

Arthur (red car) and Lucy (rowboat in tow) parked at a service station alongside caravans and HGVs

×

Dan Q posted a note for GC90RH3 Tiny Log Book

This checkin to GC90RH3 Tiny Log Book reflects a geocaching.com log entry. See more of Dan's cache logs.

I was in the area anyway so, following a recent DNF, I checked up on this cache. It’s in perfect condition (though I did have to empty a woodlouse out of the outer cache container) and ready to find (previous logger was probably looking in the wrong place: there’s no risk of touching a stinging plant to get this cache!).