But those complaints aside, the thing that bugged me most this week was how much harder they’ve made it to programatically get access to things that are publicly accessible via web pages. Like avatars, for example!
If you’re a human and you want to see the avatar image associated with a given username, you can go to
If you’re a computer and you want to get the avatar image, it used to be just as easy; just go to
twitter.com/api/users/profile_image/that-username and you’d get the image. This was great if you wanted to e.g. show a Facebook-style facepile of images of people who’d retweeted your content.
But then Twitter removed that endpoint and required that computers log in to Twitter, so a clever developer made a service that fetched avatars for you if you went to e.g.
But then Twitter killed that, too. Because despite what they claimed 5½ years ago, Twitter still clearly hates developers.
Recently, I needed a one-off program to get the avatars associated with a few dozen Twitter usernames.
First, I tried the easy way: find a service that does the work for me. I’d used
avatars.io before but it’s died, presumably because (as I soon discovered) Twitter had made things unnecessarily hard for them.
Second, I started looking at the Twitter API documentation but it took me in the region of 30-60 seconds before I said “fuck that noise” and decided that the set-up overhead in doing things the official way simply wasn’t justified for my simple use case.
So I decided to just screen-scrape around the problem. If a human can just go to the web page and see the image, a computer pretending to be a human can do exactly the same. Let’s do this:
Given that I only needed to run it once, on a finite list of accounts, I maintain that my approach was probably kinder on their servers than just manually going to every page and saving the avatar from it. But if you set up a service that uses this approach then you’ll certainly piss off somebody at Twitter and history shows that they’ll take their displeasure out on you without warning.
But it works. It was fast and easy and I got what I was looking for.
And the moral of the story is: if you make an API and it’s terrible, don’t be surprised if people screen-scape your service instead. (You can’t spell “scraping” without “API”, amirite?)
5 replies to Getting Twitter Avatars (without the Twitter API)
Thanks for this, would this process still work
And how exactly would I use it? Does one need an app or can it be run from visual studio?
Probably! This will probably work until Twitter overhauls their UI, so it’s possible that it’s more-durable than their API too.
Just save the source code as eg get-twitter-avatars.js, install Node, npm install puppeteer (the only dependency), then run it using node(as shown), passing usernames to get photos of.
The only dependencies are Node and Puppeteer. I don’t know much about Visual Studio – I haven’t seriously used it in about 15+ years – but I don’t see any reason the same technique couldn’t be adapted to any programming language you like: download the profile page in something like a browser, find the relevant element, download the referenced photo!
Confirmed still working. @Adam: based on the fact that you’re talking about Visual Studio, I’ve assumed you’re on Windows, so I’ve just given it a go on a Windows 10 box running NodeJS 17 (installed using Chocolatey because I love it, but you do you):
But there’s no way to get an image directly, with an HTTP request? Without a headless browser.
Yes, you can do so using the Twitter API; that’s the approved way to do so.