Heatmapping my Movements

As I mentioned last year, for several years I’ve collected pretty complete historic location data from GPSr devices I carry with me everywhere, which I collate in a personal μlogger server.

Going back further, I’ve got somewhat-spotty data going back a decade, thanks mostly to the fact that I didn’t get around to opting-out of Google’s location tracking until only a few years ago (this data is now also housed in μlogger). More-recently, I now also get tracklogs from my smartwatch, so I’m managing to collate more personal location data than ever before.

Inspired perhaps at least a little by Aaron Parecki, I thought I’d try to do something cool with it.

Heatmapping my movements

The last year

Heatmap showing Dan's movements around Oxford since moving house in 2020. There's a strong cluster around Stanton Harcourt with heavy tendrils around Witney and Eynsham and along the A40 to Summertown, and lighter tendrils around North and Central Oxford.
My movements over the last year have been relatively local, but there are some interesting hotspots and common routes.

What you’re looking at is a heatmap showing my location over the last year or so since I moved to The Green. Between the pandemic and switching a few months prior to a job that I do almost-entirely at home there’s not a lot of travel showing, but there’s some. Points of interest include:

  • The blob around my house, plus some of the most common routes I take to e.g. walk or cycle the children to school.
  • A handful of my favourite local walking and cycling routes, some of which stand out very well: e.g. the “loop” just below the big blob represents a walk around the lake at Dix Pit; the blob on its right is the Devils Quoits, a stone circle and henge that I thought were sufficiently interesting that I made a virtual geocache out of them.
  • The most common highways I spend time on: two roads into Witney, the road into and around Eynsham, and routes to places in Woodstock and North Oxford where the kids have often had classes/activities.
  • I’ve unsurprisingly spent very little time in Oxford City Centre, but when I have it’s most often been at the Westgate Shopping Centre, on the roof of which is one of the kids’ favourite restaurants (and which we’ve been able to go to again as Covid restrictions have lifted, not least thanks to their outdoor seating!).

One to eight years ago

Let’s go back to the 7 years prior, when I lived in Kidlington. This paints a different picture:

Heatmap showing Dan's movements around Kidlington, including a lot of time in the village and in Oxford City Centre, as well as hotspots at the hospital, parks, swimming pools, and places that Dan used to volunteer. Individual expeditions can also be identified.
For the seven years I lived in Kidlington I moved around a lot more than I have since: each hotspot tells a story, and some tell a few.

This heatmap highlights some of the ways in which my life was quite different. For example:

  • Most of my time was spent in my village, but it was a lot larger than the hamlet I live in now and this shows in the size of my local “blob”. It’s also possible to pick out common destinations like the kids’ nursery and (later) school, the parks, and the routes to e.g. ballet classes, music classes, and other kid-focussed hotspots.
  • I worked at the Bodleian from early 2011 until late in 2019, and so I spent a lot of time in Oxford City Centre and cycling up and down the roads connecting my home to my workplace: Banbury Road glows the brightest, but I spent some time on Woodstock Road too.
  • For some of this period I still volunteered with Samaritans in Oxford, and their branch – among other volunteering hotspots – show up among my movements. Even without zooming in it’s also possible to make out individual venues I visited: pubs, a cinema, woodland and riverside walks, swimming pools etc.
  • Less-happily, it’s also obvious from the map that I spent a significant amount of time at the John Radcliffe Hospital, an unpleasant reminder of some challenging times from that chapter of our lives.
  • The data’s visibly “spottier” here, mostly because I built the heatmap only out of the spatial data over the time period, and not over the full tracklogs (i.e. the map it doesn’t concern itself with the movement between two sampled points, even where that movement is very-guessable), and some of the data comes from less-frequently-sampled sources like Google.

Eight to ten years ago

Let’s go back further:

Heatmap showing Dan's movements around Oxford during the period he lived in Kennington. Again, it's dominated by time at home, in the city centre, and commuting between the two.
Back when I lived in Kennington I moved around a lot less than I would come to later on (although again, the spottiness of the data makes that look more-significant than it is).

Before 2011, and before we bought our first house, I spent a couple of years living in Kennington, to the South of Oxford. Looking at this heatmap, you’ll see:

  • I travelled a lot less. At the time, I didn’t have easy access to a car and – not having started my counselling qualification yet – I didn’t even rent one to drive around very often. You can see my commute up the cyclepath through Hinksey into the City Centre, and you can even make out the outline of Oxford’s Covered Market (where I’d often take my lunch) and a building in Osney Mead where I’d often deliver training courses.
  • Sometimes I’d commute along Abingdon Road, for a change; it’s a thinner line.
  • My volunteering at Samaritans stands out more-clearly, as do specific venues inside Oxford: bars, theatres, and cinemas – it’s the kind of heatmap that screams “this person doesn’t have kids; they can do whatever they like!”

Every map tells a story

I really love maps, and I love the fact that these heatmaps are capable of painting a picture of me and what my life was like in each of these three distinct chapters of my life over the last decade. I also really love that I’m able to collect and use all of the personal data that makes this possible, because it’s also proven useful in answering questions like “How many times did I visit Preston in 2012?”, “Where was this photo taken?”, or “What was the name of that place we had lunch when we got lost during our holiday in Devon?”.

There’s so much value in personal geodata (that’s why unscrupulous companies will try so hard to steal it from you!), but sometimes all you want to do is use it to draw pretty heatmaps. And that’s cool, too.

Heatmap showing Dan's movements around Great Britain for the last 10 years: with a focus on Oxford, tendrils stretch to hotspots in South Wales, London, Cambridge, York, Birmingham, Preston, Glasgow, Edinburgh, and beyond.

How these maps were generated

I have a μlogger instance with the relevant positional data in. I’ve automated my process, but the essence of it if you’d like to try it yourself is as follows:

First, write some SQL to extract all of the position data you need. I round off the latitude and longitude to 5 decimal places to help “cluster” dots for frequency-summing, and I raise the frequency to the power of 3 to help make a clear gradient in my heatmap by making hotspots exponentially-brighter the more popular they are:

SELECT ROUND(latitude, 5) lat, ROUND(longitude, 5) lng, POWER(COUNT(*), 3) `count`
FROM positions
WHERE `time` BETWEEN '2020-06-22' AND '2021-08-22'
GROUP BY ROUND(latitude, 5), ROUND(longitude, 5)

This data needs converting to JSON. I was using Ruby’s mysql2 gem to fetch the data, so I only needed a .to_json call to do the conversion – like this:

db = Mysql2::Client.new(host: ENV['DB_HOST'], username: ENV['DB_USERNAME'], password: ENV['DB_PASSWORD'], database: ENV['DB_DATABASE'])

Approximately following this guide and leveraging my Mapbox subscription for the base map, I then just needed to include leaflet.js, heatmap.js, and leaflet-heatmap.js before writing some JavaScript code like this:

body.innerHTML = '<div id="map"></div>';
let map = L.map('map').setView([51.76, -1.40], 10);
// add the base layer to the map
L.tileLayer('https://api.mapbox.com/styles/v1/{id}/tiles/{z}/{x}/{y}?access_token={accessToken}', {
  maxZoom: 18,
  id: 'itsdanq/ckslkmiid8q7j17ocziio7t46', // this is the style I defined for my map, using Mapbox
  tileSize: 512,
  zoomOffset: -1,
  accessToken: '...' // put your access token here if you need one!
// fetch the heatmap JSON and render the heatmap
  let heatmapLayer = new HeatmapOverlay({
    "radius": parseFloat(document.querySelector('#radius').value),
    "scaleRadius": true,
    "useLocalExtrema": true,
  heatmapLayer.setData({ data: json });

That’s basically all there is to it!

City Roads

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

Map of Kidlington's roads

Cute open source project that produces on-demand SVG and PNG maps, like the one above, based on the roads in OpenStreetMap data. It takes a somewhat liberal view of what a “road” is: I found it momentarily challenging to get my bearings in the map above, which includes where I live, because the towpath and cycle paths are included which I hadn’t expected. Still a beautiful bit of output and the source code could be adapted for any number of interesting cartographic projects.

Maps that Talk

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

A former colleague talks about some of the artefacts from the Bodleian’s collections that didn’t make it into the Talking Maps exhibition (one of the last exhibitions I got to work on during my time there; indeed, you’ll see plenty of pictures from it in my post about making digital interactives). I was particularly pleased by the Soviet map of Oxford, but everything Nick presents in this video is pretty awesome: it’s a great reminder that for every fantastic exhibition you see at a good museum, there’s always at least as much material “behind the scenes” that you’re missing out on!

How Much of My Graticule is Covered With Water?

I’m a moderately-keen geohasher, as you might be aware if you follow my geohashing logs or you saw that video of me going ‘hashing earlier this month.

For those that don’t know, the skinny version is this: in May 2008 an XKCD comic was published proposing (or at least joking about) a new game with a name reminiscient of geocaching. To play the game, participants use a mathematical hashing function on the current date and the most recent Dow Jones Industrial Average opening value to generate sets of random coordinates around the globe and then try to find their way to them, hopefully experiencing adventures along the way. The nature of stock markets and hashing functions means that the coordinates for any given day are effectively random and impossible to predict (far) in advance, so it’s sometimes described as a spontaneous adventure generator.

XKCD comic #426, "Geohashing"
The XKCD comic that started it all.

Recently, I found myself wondering about how much of a disadvantage players are at if they live in very “wet” graticules. Residents of the Channel Islands graticule (49 -2), for example, are confined to two land masses surrounded entirely by water. And while it’s true that water hashpoints can be visited if you’re determined enough, it’s still got to be considered to be playing at a disadvantage compared to those of us lucky ones in landlocked graticules like mine (51 -1).

And because I’m me and so can’t comfortably leave a question unanswered, I wrote a program to try to answer it! It’s among the hackiest, dirtiest software solutions I’ve ever written, so if it works for you then it’s a flipping miracle. What it does is:

  1. Determines which OpenStreetMap tiles (the image files served to your browser when you use OpenStreetMap) cover the graticule in question, and downloads them.
  2. Extracts information about the colour of each pixel in each tile.
  3. Counts the proportion of “water blue” pixels to other pixels (this isn’t perfect, because it trips over things like ferry lines on the map as being “not water”, especially at low zoom-levels).
Extreme zoom-in on Worcester College Lake, on OpenStreetMap.
Some parts of Worcester College Lake are identified as “not water” on account of the text overlay.

I mentioned it was hacky, right?

You can try it for yourself, if you’d like. You’ll need NodeJS, wget, wc, and ImageMagick – all pretty standard or easy-to-get things on a typical Linux box. Run with node geohash-pcwater.js 51 -1, where 51 -1 is the identifier for the graticule you’re interested in. And in case you’re interested – the Swindon graticule (where I live) is about 0.68% water, but the Channel Islands graticule is closer to 93.13% water. That’s no small disadvantage: sorry, Channel Islands geohashers!

Update 2018-08-22: discovered some prior art that takes a somewhat-similar approach.

Maps Showing California as an Island – The Public Domain Review

This article is a repost promoting content originally published elsewhere. See more things Dan's reposted.

http://publicdomainreview.org/collections/maps-showing-california-as-an-island/ (publicdomainreview.org)

If California were a country its economy would be the fifth largest in the world (just ahead of the UK). Yet the tech boom is not the starkest way California has ever stood apart from its neighbours. That would surely be the maps depicting it as an island, entire of itself. Below we have featured our pick of these glorious seventeenth- and eighteenth-century aberrations, from a collection of hundreds held at Stanford.

The intriguing story of how the maps came to be deserves a little mapping itself. In the 1530s Spanish explorers led by Hernán Cortés encountered the strip of land we now know as the Baja Peninsula. They mistook it for an island and called it California.


Recently, I learned that the roads in Great Britain are numbered in accordance with a scheme first imagined about ninety years ago, and, as it evolved, these road numbers were grouped into radial zones around London (except for Scotland, whose road numbering only joined the scheme later). I’d often noticed the “clusters” of similarly-numbered roads (living in Aberystwyth, you soon notice that all the A and B roads start with a 4, and I soon noticed that the very same A44 that starts in Aberystwyth seems to have followed me to my home here in Oxford).

Road Numbering Zones of the United Kingdom

Who’d have thought that there was such a plan to it. If you’re aware of any of the many roads which are in the “wrong” zone, you’d be forgiven for not seeing the pattern earlier, though. However, seeing all of this attempt at adding order to what was a chaotic system for the long period between the Romans leaving and the mid-20th century makes me wonder one thing: are there “roadspotters”?

There exist trainspotters, who pursue the more-than-a-little-bit-nerdy hobby of traveling around and looking at different locomotives, marking down their numbers in notepads and crossing them off in reference books. Does the same phenomena exist within road networks?

It turns out that it does; or some close approximation of it does, anyway. One gentleman, for example, writes about “recovering” road signs formerly of the A6144(M), which – until 2006 – was the UK’s only single-carriageway motorway. A site calling itself The Motorway Archive has a thoroughly-researched article on the construction history of the M74/A74(M) from Glasgow to Carlisle. Another website – and one that I’m embarrassed to admit that I’d visited on a number of previous occasions – reviews every motorway service area in Britain. And, perhaps geekiest of all, the Society for All British and Irish Road Enthusiasts (SABRE) maintains a club, meetups, and a thoroughly-researched wiki of everything you never wanted to know about the roads of the British Isles.

From what started as a quick question about British road numbering, I find myself learning about a hobby that’s perhaps even geekier than trainspotting. Thanks, Internet.