Blog

More than you expected?

You're reading everything on Dan's blog - including notes, reposts, checkins, videos and comics.
That might be more than you wanted to see, if you're only interested in blog posts (articles) Dan has written.

XPath Scraping with FreshRSS

I’ve been spending a while running on reduced brain capacity lately so, to ease myself back into thinking like a programmer, I upgraded my preferred feed reader FreshRSS to version 1.20.0 – which was released a couple of weeks ago – and tried out what I believe is its killer new feature: HTML + XPath scraping.

Screenshot showing Beverley Newing's weblog; two articles are visible - Paperback copy of 'Disability Visibility', edited by Alice Wong, next to a cup of tea Setting up an Accessibility Book Club, published on 1 March 2022, and Reflecting on 2021, published on 1 January 2022.
I like to keep up-to-date with my friend Bev’s blog, but they don’t have an RSS feed.

I’ve been using RSS1 for about 20 years and I love it. It feels great to be able to curate my updates based on “what I care about”, and not on “what some social network thinks I should care about”, to keep things to read later, to prioritise effectively based on my own categorisation, to consume content offline and have my to-read list synchronise later, etc.

RSS never went away, of course (what do you think a podcast is?), but it got steamrollered out of the public eye by big companies who make their money out of keeping your eyes on their platforms and off the open Web. But it feels like it’s slowly coming back: even Substack – whose entire thing is that an email client is more-convenient than a feed reader for most people – launched an RSS reader this week!

A smartphone on a wooden surface. The screen shows the FeedMe app, showing the most-recent blog post from Beverley's blog.
My day usually starts in my feed reader, accessed via the FeedMe app from my mobile (although FreshRSS provides a reasonably good responsive interface out-of-the-box!)

I love RSS so much that I routinely retrofit other people’s websites with feeds just so I can subscribe to them: I even published the tool I use to do so! Whether filtering sports headlines out of BBC News, turning retro webcomics into “reading lists” so I can track my progress, or just working around sites that really should have feeds but refuse to, I just love sidestepping these “missing feeds”. My friend Beverley has a blog without any kind of feed, so I added one so I could subscribe to it. Magic.

But with FreshRSS 1.20.0, I no longer have to maintain my own tool to get this brilliant functionality, and I’m overjoyed. Let’s look at how it works by re-subscribing to Beverley’s blog but without a middleware tool.

Screenshot showing FetchRSS being used to graphically create a feed from Beverley's blog.
This post is about to get pretty technical. If you don’t want to learn some XPath but just want to make a feed out of a web page, use a graphical tool like FetchRSS.

In the latest version of FreshRSS, when you add a new feed to your reader, a new section “Type of feed source” is available. Unfold it, and you can change from the default (“RSS / Atom”) to the new option “HTML + XPath (Web scraping)”. Put a human-readable page address rather than a feed address into the “Feed URL” field and fill these fields to tell FreshRSS how to parse the page to get the content you want. Note that it doesn’t matter if the web page isn’t valid XML (e.g. missing closing tags) because it’s going to get run through PHP’s DOMDocument anyway which will “correct” for some really sloppy code if needed.

Browser debugger running document.evaluate('//li[@class="blog__post-preview"]', document).iterateNext() on Beverley's weblog and getting the first blog entry.
You can use your browser’s debugger to help check your XPath rules: here I’ve run  document.evaluate('//li[@class="blog__post-preview"]', document).iterateNext() and got back the first blog post on the page, so I know I’m on the right track.
You’ll need to use XPath to express how to find a “feed item” on the page. Here’s the rules I used for https://webdevbev.co.uk/blog.html (many of these fields were optional – I didn’t have to do this much work):

  • Feed title: //h1
    I override this anyway in FreshRSS, so I could just have used the a string, but I wanted the XPath practice. There’s only one <h1> on the page, and it can be considered the “title” of the feed.
  • Finding items: //li[@class="blog__post-preview"]
    Each “post” on the page is an <li class="blog__post-preview">.
  • Item titles: descendant::h2
    Each post has a <h2> which is the post title. The descendant:: selector scopes the search to each post as found above.
  • Item content: descendant::p[3]
    Beverley’s static site generator template puts the post summary in the third paragraph of the <li>, which we can select like this.
  • Item link: descendant::h2/a/@href
    This expects a URL, so we need the /@href to make sure we get the value of the <h2><a href="...">, rather than its contents.
  • Item thumbnail: descendant::img[@class="blog__image--preview"]/@src
    Again, this expects a URL, which we get from the <img src="...">.
  • Item author: "Beverley Newing"
    Beverley’s blog doesn’t host any guest posts, so I just use a string literal here.
  • Item date: substring-after(descendant::p[@class="blog__date-posted"], "Date posted: ")
    This is the only complicated one: the published dates on Beverley’s blog aren’t explicitly marked-up, but part of a string that begins with the words “Date posted: “, so I use XPath’s substring-after function to strtip this. The result gets passed to PHP’s strtotime(), which is pretty tolerant of different date formats (although not of the words “Date posted:” it turns out!).
Screenshot: Adding a "HTML + XPath (Web scraping)" feed via FreshRSS.
I’d love one day for FreshRSS to provide some kind of “preview” feature here so you can see what you’ll expect to get back, as you work. That, and support for different input types (JSON, perhaps?), perhaps other selectors (I find CSS-style selectors much simpler than XPath), and maybe even an option to execute Javascript on the page before scraping (I use this in my own toolchain, but that’s just because I want to have my cake and eat it too). But this is still all pretty awesome.

I hope that this is just the beginning for this new killer feature in FreshRSS: there’s so much more it can be and do. But for now, I’m still mighty impressed that I can begin to phase-out my use of my relatively resource-intensive feed-building middleware and use my feed reader to do more and more of the heavy lifting for which I love it so much.

I also love that this functionally adds h-feed support in by the back door. I’d still prefer there to be a “h-feed” option in the “Type of feed source” drop-down, but at least I can add such support manually, now!

Beverley's blog post "Setting up an Accessibility Book Club" in FreshRSS.
The finished result: Bev’s blog posts appear directly in my feed reader, even though they don’t have a feed, and now without going through the middleware I’d set up for that purpose.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

All the game descriptions from the sale

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Video game poster for "It's Probably Fine", showing a woman driving with red and blue lights behind her.

You’ve got 37 unpaid parking tickets. You just got pulled over for speeding. In your defense, you were texting your sister about how drunk you are. Plus there’s all that blood on your windshield. Obviously you know it’s deer blood, but the police officers walking toward your vehicle don’t. Still, in the time it takes them to figure that out, maybe you’ll sober up. Or escape on foot! Either way, it’ll probably be fine.

User Tags: Poor Choices / Story Rich / Multiple Endings / Parkour

Video game poster for Dead Seagull Zoo Magnate, showing dead seagulls in a cartoony style.

Collect dead seagulls and build a zoo to house them all. Beautify the zoo with artistic flair and deodorizing sprays. Design creative group promotions to stir up interest! Is that a customer? You’d better hope it’s not the owner of the live seagull zoo down the street, because he’s probably got some questions.

User Tags: Hard Work / Supply / Demand / Diseases & Parasites

Claire Hummel produced fake video game art for the Steam Summer Sale, which was already excellent, but when @g-a-y-g-o-y-l-e reblogged, asking for more context, Claire delivered and then some. Every single one of these “game descriptions” is a special kind of comedy gold… and yet somehow believable from the store that sells us Dream Daddy, IKEA VR Pancake Kitchen, Organ Trail, Oh… Sir!! The Insult Simulator, and Goat Simulator (all of which I own copies of). Go read the full list.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

Covid Brain

I managed to dodge infection for 922 days of the Covid pandemic1, but it caught up with me eventually.

Lateral flow test, with "DAN" written on it, showing a solid control line and a very clear solid test line: a clear positive result.
Well, shit.

Frankly, it’s surprising that it took this long. We’ve always been careful, in accordance with guidance at any given time, nd we all got our jabs and boosters as soon as we were able… but conversely: we’ve got school-age children who naturally seem to be the biggest disease vectors imaginable. Our youngest, in fact, already had Covid, but the rest of us managed to dodge it perhaps thanks to all these precautions.

Vials of Covid vaccine scattered across a red background. Photo courtesy Maksim Goncharenok.
The vaccine provide protection, but it’s not a magical force-field.

Luckily I’m not suffering too badly, probably thanks to the immunisation. It’s still not great, but I dread to think how it might have been without the benefit of the jab! A minor fever came and went, and then it’s just been a few days of coughing, exhaustion, and… the most-incredible level of brain-fog.

Dan with the dog, in the garden.
Today, for example, I completey blanked the word “toilet” and struggled for some time to express to the dog why I’d brought her into the garden, while she stared at me expectantly.

I’ve taken the week off work to recover, which was a wise choice. As well as getting rest, it’s meant that I’ve managed to avoid writing production code with my addled brain! Instead, I’ve spent a lot of time chilling in bed and watching all of the films that I’d been meaning to! This week, I’ve watched:

  • Peggy Sue Got Married (y’know, that other mid-1980s movie about time travel and being a teenager in the 1950s). It was okay; some bits of the direction were spectacular for its age, like the “through the mirror” filming.
  • Fall. I enjoyed this more than I expected to. It’s not great, but while I spent most of the time complaining about the lack of believability in the setting and the characters’ reactions, the acting was good and the tension “worked”: it was ocassionally pretty vertigo-inducing, and that’s not just because I’ve been having some Covid-related dizziness!
  • RRR. Oh my god this Tollywood action spectacle was an adventure. At one point it’s a bromantic buddy comedy, then later there’s a dance-off, then for a while there’s a wonderful “even language can’t divide us” romance, but then later a man picks up a motorcycle with one hand and uses it to beat up an entire army, and somehow it all feels like it belongs together. The symbolism’s so thick you can spread it (tl;dr: colonialism bad), but it’s still a riot of a film.
  • Cyrano, which I feel was under-rated but that could just be that I have a soft spot for the story… and a love of musical theatre.
  • Also, at times when I didn’t think my brain had the focus for something new, I re-watched Dude, Where’s My Car? because I figured a stoner comedy that re-replains the plot every 20 minutes or so was about as good as I could expect my brain to handle at the time, and Everything Everywhere All At Once which I’ve now seen three times and loved every single one: it’s one of my favourite films.
Dan lying in bed, giving a weak "thumbs up".
See, I’m fine! (Feel like I’ve spent a lot of time lying here, this week.)

Anyway: hopefully next week I’ll be feeling more normal and my poor Covid-struck brain can be trusted with code again. Until then: time to try to rest some more.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Dan Q found GCA002Z Busted out!

This checkin to GCA002Z Busted out! reflects a geocaching.com log entry. See more of Dan's cache logs.

Could have been expected to get the FTF for this one, given that it’s (a) literally 20 seconds walk from my front door and (b) the CO had indicated that one would be hidden around here, but unfortunately I contracted covid last weekend and any walk longer tab my garden was quickly leaving me exhausted. This evening I felt a little better and so the geohound and I (pictured) braved a couple of minutes in the rain to come and sign the logbook.

Note to future cachers planning to park and grab: the “layby” indicated is a working bus stop, albeit with an infrequent (every 2 hours, weekday daytimes) schedule, so remember to be a polite cacher and try not to park in it at times that it’ll be needed by the minibus!

I might need to find a new home for my replacement to GC90RH3, whose bridge hiding place is only 100m or so (less than the requisite 0.1 miles!) from this new cache! Ah well, that’ll teach me to be a slow CO!

TFTC, and for getting me out of the house for a walk for the first time since I got sick almost a week ago.

Dan with the dog, in the garden.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.758167,-1.40125

Dan Q couldn’t find GC33BCA WWW#7 – www.letterboxing.org

This checkin to GC33BCA WWW#7 – www.letterboxing.org reflects a geocaching.com log entry. See more of Dan's cache logs.

Based on the description and the hint this one should have been easy. Perhaps a longer search might have been fruitful but the geopup was anxious to keep moving.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.810633,-1.480233

Dan Q found GC33BBW WWW#6 – www.geochecker.com

This checkin to GC33BBW WWW#6 – www.geochecker.com reflects a geocaching.com log entry. See more of Dan's cache logs.

After I found the right hiding place, this one was pretty easy, though I was initially hesitant to put my hand into it after I mistook the cache’s unusual container for (a very large version of) something else that could be laid in a place like this. TFTC.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.8069,-1.479383

Dan Q found GC33BAJ WWW#4 – earthcache.org

This checkin to GC33BAJ WWW#4 – earthcache.org reflects a geocaching.com log entry. See more of Dan's cache logs.

Geopup and I found quite easily while out on a walk. The excitable doggo isn’t so keen on stopping and searching for caches when there are so many new and exciting smells just over her visual horizon, so today’s expedition might only give me a couple of minutes to hunt for each: we’ll have to see if that’s enough to log any further finds this morning.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.8034,-1.4795

Dan Q performed maintenance for GC9Z37K Friar’s Farm – Far-Flung Fence

This checkin to GC9Z37K Friar's Farm - Far-Flung Fence reflects a geocaching.com log entry. See more of Dan's cache logs.

Found left out in clear with lid open but otherwise not vandalised. Closed and re-hid deeper into hiding place. Will add a geocaching information card on my next visit to better explain its presence to any muggles passing by.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.765367,-1.415383

Dan Q performed maintenance for GC9EXXC Shady Seat on The Green

This checkin to GC9EXXC Shady Seat on The Green reflects a geocaching.com log entry. See more of Dan's cache logs.

Dropped by for a routine check during an autumnal shower this afternoon. Hiding position had drifted slightly, so moved back where it belongs. All well and ready to find!

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.749,-1.40175

Dan Q performed maintenance for GC9Z37H Friar’s Farm – Woodland Walk

This checkin to GC9Z37H Friar's Farm - Woodland Walk reflects a geocaching.com log entry. See more of Dan's cache logs.

It’s been here a week, so a maintenance visit was due to check it was still happy, healthy, and ready to find. And it is.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.7618,-1.410917

Dan Q performed maintenance for GC9Z37G Friar’s Farm – Decrepit Directions

This checkin to GC9Z37G Friar's Farm - Decrepit Directions reflects a geocaching.com log entry. See more of Dan's cache logs.

Initial maintenance checkin. All well here!

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.759783,-1.410017

Dan Q performed maintenance for GC8W7QW Forgotten Bridge

This checkin to GC8W7QW Forgotten Bridge reflects a geocaching.com log entry. See more of Dan's cache logs.

Routine maintenance visit. Looks like a “Reggie” comes by this way with his dogs and signs the logbook, sometimes! :-) Not such a forgotten footbridge after all?

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.7652,-1.390367

Dan Q found GC5PPFT Ssssshhhh it’s a …….

This checkin to GC5PPFT Ssssshhhh it's a ....... reflects a geocaching.com log entry. See more of Dan's cache logs.

I’ve been working in Witney one day every week or two lately, but somehow I’ve never managed to sync up my work times with the hours that this building is accessible! Or, when I do, I’m in a hurry and don’t have time to stop and hunt!

This morning, though, the stars aligned and I was able to get to the GZ. The cache was pretty much where I expected based on the coordinates and the hint, but still took a minute out two to lay hands on. Soon, though, I was quietly sitting and reading past log entries.

SL TFTC! FP awarded for sheer awesome.

Dan sits in a library, holding a small book and with his finger to his lips in a 'shush' gesture. The back cover of the book contains text identifying it as a geocache.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.786517,-1.487183

Dan Q found GC8V4TA Freeland to Hanborough 4

This checkin to GC8V4TA Freeland to Hanborough 4 reflects a geocaching.com log entry. See more of Dan's cache logs.

GPSr was all over the place, but the hint quickly sent me to the obvious hiding place. SL, TFTC.

Footnotes

1 When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, h-feed, etc. But I can’t get over the old-fashioned name, and I don’t care to try.

1 Based on the World Health Organisation’s declaration of the outbreak being a pandemic on 11 March 2020 and my positive test on 19 September 2022, I stayed uninfected for two years, six months, one week, and one day. But who’s counting?

Map of 51.828783,-1.41065