Dan Q

Tag: statistics

100 Days To Offload

The ever-excellent Kev Quirk in 2020 came up with this challenge: write a blog post on each of 100 consecutive days. He called it #100DaysToOffload, in nominal reference to the “100 days of code” challenge. I was reflecting upon this as I reach this, my 36th consecutive day of blogging and my longest ever “daily streak” (itself a spin-off of my attempt at Bloganuary this year), and my 48th post of the year so far.

Monochrome photograph showing sprinters at the starting line.
I guess I’ve always been more of a sprinter/hurdles blogger than a marathon runner.

Might I meet that challenge? Maybe. But it turns out it’s easier than I thought because Kev revised the rules to require only 100 posts in a calendar year (or any other 365-day period, but I’m not going to start thinking about the maths of that).

That’s not only much more-achievable… I’ve probably already achieved it! Let’s knock out some SQL to check how many posts I made each year: 

SELECT
  YEAR(wp_posts.post_date_gmt) yyyy,
  COUNT(wp_posts.ID) total
FROM
  wp_posts
WHERE
  wp_posts.post_status='publish'
  AND wp_posts.post_type='post'
GROUP BY yyyy
ORDER BY yyyy
My code’s actually a little more-complicated than this, because of some plot, but this covers the essentials.

A big question in some years is what counts as a post. Kev’s definition is quite liberal and includes basically-everything, but I wonder if mine shouldn’t perhaps be stricter. For example:

  • Should I count checkins, even though they’re not always born as blog posts but often start as logs on geocaching websites? (My gut says yes!)
  • Do reposts and bookmarks contribute, a significant minority of which are presented without any further interpretation by me? (My gut says no!)
  • Does a vlog version of a blog post count separately, or is it a continuation of the same content? (My gut says the volume is too low to matter!)
  • Can a retroactive achievement (i.e. from before the challenge was announced) count? Kev writes “there is no specific start date”, but it seems a little counter to the idea of it specifically being a challenge to claim it when you weren’t attempting the challenge at the time.
  • And so on…
Year Posts Success? Notes
1998 7 ❌ No Some posts are lost from 1998/1999. If they were recovered I might have made 100 posts in 1999, but probably not in 1998 as I only started blogging on 27 September 1998.
1999 66 ❌ No
2000 2 ❌ No
2001 11 ❌ No
2002 5 ❌ No
2003 189 🏆 Yes Achieved 1 September, with a post about an article on The Register about timewasting. Or, if we allow reposts, three days earlier with a repost about Claire's car being claimed by the sea.
2004 374 🏆 Yes An early win on 20 April, with a made-up Chez Geek card. Or if we allow reposts, two days earlier with thoughts on a confusing pro-life (???) website.
2005 381 🏆 Yes In a highly-productive year of blogging, achieved on 7 April with a post about enjoy curry and public information films with friends. If we allow bookmarks (I was highly-active on del.icio.us at the time!), achieved even earlier on 18 February with some links to curious websites.
2006 206 🏆 Yes On 21 July, I shared a personality test (which was actually my effort to repeat an experiment in using Barnum-Forer statements) - I didn't initially give away that I was the author of the "test". Non-pedants will agree I achieved the goal earlier, on 19 June, with my thoughts on a programming language for a hypothetical infinitely-fast computer.
2007 166 🏆 Yes Achieved on 2 July with thoughts on films I'd watched and board games I'd played recently. Or arguably 12 days earlier with Claire's birthday trip to Manchester.
2008 86 ❌ No
2009 79 ❌ No
2010 159
(84 for pedants) 		✅ Yes* A heartfelt post about saying goodbye to Aberystwyth as I moved to Oxford on 16 June was my 100th of the year. Pedants might argue that this year shouldn't count, but so long as you're willing to count checkins (and you should) then it would... and my qualifying post would have come only a couple of days later, with a post about the Headington Shark, which I had just moved-in near to.
2011 177 🏆 Yes Reached the goal on 28 October when I wrote about mild successes in my enquiries with the Office of National Statistics about ensuring that information about polyamorous households was accurately recorded. Or if we earlier on 9 June with a visual gag about REM lyrics if you accept all my geocache logs as posts too (and again: you should).
2012 129
(87 for pedants) 		✅ Yes* My 100th post of the year came on 28 August when I wrote about launching a bus named after my recently-deceased father. You have to be willing to accept both checkins and reposts as posts to allow this year to count.
2013 138
(59 for pedants) 		😓 Probably not I'm not convined this low-blogging year should count: a clear majority of the posts were geocaching logs, and they weren't always even that verbose (consider this candidate for 100th post of 2013, from 1 October).
2014 335
(22 for pedants) 		🙁 Not really Another geocache log heavy, conventional blogpost light year that I'm not convinced should count, evem if the obvious candidate for 100th post would be 18 May's cool article about geocaching like Batman!
2015 205
(18 for pedants) 		🙁 Not really Still no, for the same reasons as above.
2016 163
(37 for pedants) 		🙁 Not really
2017 301
(42 for pedants) 		🙁 Not really
2018 547
(87 for pedants) 		✅ Yes* I maintain that checkins should count, even when they're PESOS'd from geocaching sites, so long as they don't make up a majority of the qualifying posts in a year. In which case this year should qualify, with the 100th post being my visit to this well-hidden London pub while on my way to a conference.
2019 387
(86 for pedants) 		✅ Yes* Similarly this year, when on 15 August I visited a GNSS calibration point in the San Francisco Bay Area... on the way to another conference!
2020 221
(64 for pedants) 		✅ Yes* Barely made it this year (ignoring reposts, of which I did lots), with my 21 December article about a little-known (and under-supported) way to inject CSS using HTTP headers, which I later used to make a web page for which View Souce showed nothing.
2021 190
(57 for pedants) 		✅ Yes* A cycle to a nearby geocache was the checkin that made the 100th post of this year, on 27 August.
2022 168
(55 for pedants) 		✅ Yes* My efforts to check up on one of my own geocaches on 7 September scored the qualifying spot.
2023 164
(86 for pedants) 		✅ Yes* My blogging ramped up again this year, and on 24 August I shared a motivational poster with a funny twist, plus a pun at the intersection between my sexuality and my preferred mode of transport.
2024 350 🏆 Yes Writing at full-tilt, my hundredth post came when I found a geocache near Regents Canal, but pedants who disregard reposts and checkins might instead count my excitement at the Ladybird Web browser as the record-breaker. This year also saw me write my 5,000th post on this blog! Wowza!
Total 5,211 Total count of all the posts.
Doesn't add up? Not all posts feature in one of the years above!

* Pedants might claim this year was not a success for the reasons described above. Make your own mind up.

In any case, I’d argue that I clearly achieved the revised version of the challenge on certainly six, probably fourteen, arguably (depending on how you count posts) as many as nineteen different years since I started blogging in 1998. My least-controversial claims would be:

  1. September 2003, with Timewasting
  2. April 2004, with Chez Geek Card of the Day
  3. April 2005, with Curry with Alec and Suz
  4. July 2006, with Coolest Personality Test I’ve Ever Seen
  5. July 2007, with It’s All Fun and Games
  6. June 2010, with Saying Goodbye
  7. October 2011, with Poly and the Census – Success! (almost)
  8. August 2012, with A Bus Called Peter
  9. June 2018, with Dan Q found GLW6CMKQ 16th Century Pub (Central London) 
  10. August 2019, with Dan Q found GC6KR0H Bay Area Calibration Point #4 – New Technology
  11. December 2020, with The Fourth Way to Inject CSS
  12. August 2021, with Dan Q found GC531M9 Walk by the Firehouse #1
  13. October 2022, with Dan Q performed maintenance for GC9Z37H Friar’s Farm – Woodland Walk
  14. August 2023, with Inclusivity

Given all these unanswered questions, I’m not going to just go ahead and raise a PR against the Hall of Fame! Instead, I’ll leave it to Kev to decide whether I’m (a) eligible to claim a 14-time award, (b) merely eligible for a 4-time award for the years following the challenge starting, or (c) ineligible to claim success until I intentionally post 100 times in a year (in, at current rates, another two months…). Over to you, Kev…

Update: Kev’s agreed that I can claim the most-recent four of them, so I raised a PR.

×

Article posted at UTC on .

2 comments

[Bloganuary] My Biggest Challenge

This post is part of my attempt at Bloganuary 2024. Today’s prompt is:

What are your biggest challenges?

The Challenge of Winter Motivation

Two years ago, I reflected in February that I’d made almost zero blog posts in the previous month. Last month, I implemented a dynamically-updating Blog Stats page and my “winter/early Spring dip” became more-visible than ever.

Chart showing number of articles on DanQ.me by month of year, with a pronounced dip starting in January and continuing through until a rebound in April.
I find winters are generally bad for my creativity and motivation, usually until I bounce back in the Spring.

In an attempt to keep me writing daily, I’m giving Bloganuary a go this year. It’s sort-of like the NaNoWriMo of blogging1. And for me, Bloganuary’s very purpose is to overcome the challenge of getting disconnected from blogging when the nights are long and inspiration’s hard to find2.

The Challenge of Staying On-Task

But outside of the winter, my biggest challenge is usually… staying on-task!

It’s easy to get my focus to wane and for me to drift into some other activity than whatever it is I should be spending my time on. It’s not even procrastination3 so much as it’s a fluctuating and changing field of interest. I’ll drift off of what I’m supposed to be working on and start on something that interests me more in that moment… and then potentially off that too, in turn. The net result is that both my personal and professional lives are awash with half-finished projects4, all waiting their turn for me to find the motivation to swing back around and pick them up on some subsequent orbit of my brain.

A person wearing a cardboard box on their head, labelled "BRAIN". Above, a hand reaches from out-of-frame to hold a sign labelled "IDEA" above them.
You know how sometimes a stock image says exactly what you need it to? This isn’t one of those times.

It’s the kind of productivity antipattern I’d bring up with my coach, except that I already know exactly how she’d respond. First, she’d challenge the need to change; require that I justify it first. Second, she’d insist that before I can change, I need to accept and come to terms with who I am, intrinsically: if this flitting-about is authentically “me”, who am I to change it?

Finally, after weeks or months of exercises to fulfil these two tasks, she’d point out that I’ve now reached a place where I’m still just as liable to change lanes in the middle of a project as I was to begin with, but now I’m more comfortable with that fact. I won’t have externally changed, I’ll “just” have found some kind of happy-clappy inner peace. And she’ll have been right that that’s what I’d actually needed all along.

Maybe it’s not such a challenge, after all.

Footnotes

1 Except that would be NaBloPoMo, of course. But it’s a similar thing.

2 Also, perhaps, to help me focus on writing more-often, on more-topics, than I might otherwise in the course of my slow, verbose writing.

3 Except when it is.

4 Not to mention countless draft blog posts!

× ×

Article posted at UTC on .

1 comment

Blogging Stats

During a conversation with a colleague last week, I claimed that while I blog more-frequently than I did 5-10 years ago, it’s still with a much lower frequency than say 15-20 years ago.

Only later did I stop to think: is that actually true? It’s time for a graph!

I’ve previously graphed my blogging in an ad-hoc way, e.g. in 2016 I did a word-count and in 2021 I graphed posts-by-month-of-year, but I’ve never made an “eternal”, automatically-updating, interactive1 graph. Until now:

Generating a chart...
If this message doesn't go away, the JavaScript that makes this magic work probably isn't doing its job right: please tell Dan so he can fix it.

If you consider just articles (and optionally notes, which some older content might have been better classified-as, in retrospect) it looks like I’m right. Long gone are months like February 2005 when I posted an average of three times every two days! November 2018 was a bit of an anomaly as a I live-tweeted Challenge Robin II: my recent output’s mostly been comparable to the “quiet period” from 2008-20102.

Looking at number of posts by month of the year, it’s interesting to see a pronounced “dip” in all kinds of output roundabout March, less reposts in Summer and Autumn, and – perhaps unsurprisingly – more checkins (which often represent geocaching/geohashing logs) in the warmer months. Even on this scale, you can see the impact of the November “Challenge Robin spike” in the notes:

Generating a chart...
If this message doesn't go away, the JavaScript that makes this magic work probably isn't doing its job right: please tell Dan so he can fix it.

Anyway, now I’ve actually automated these kinds of stats its easier than ever for me to ask questions about how and when I write in my blog. I’ve put living copies of the charts plus additional treats (want to know when my longest “daily streak” was?) on a special page dedicated to that purpose. It’ll be interesting to see how it looks on this blog’s 25th anniversary, in a little under a year!

Footnotes

1 Try clicking on any of the post kinds in the legend to add/remove them, or click-and-drag a range across the chart to zoom in.

2 In hindsight, I was clearly depressed in and around 2009 and this doubtless impacted my ability to engage in “creative” pursuits.

Article posted at UTC on .

No comments

Hey ONS: This Is Not A Mistake

Hi, ONS! I know we haven’t really spoken since you ghosted me in 2011, but I just wanted to clear something up for you –

This is not a mistake (except for the missing last names):

(Specimen) 2021 census form on which Ruth declares that she cohabits with both a husband AND a partner.
It’s perfectly possible for somebody to live with multiple partners, even if they’re forbidden from marrying more than one.

Back in 2011 you thought it was a mistake, and this prevented my partner, her husband and I from filling out the digital version of the census. I’m sure it’s not common for somebody to have multiple cohabiting romantic relationships (though it’s possibly more common than some other things you track…), but surely an “Are you sure?” would be better than a “No you don’t!”

Clippy says "It looks like you've got a husband AND a partner. Is that right?" with possible answers "Yes, and it's awesome." or "No, but I can dream!"
For all I know, you already fixed it. If not: I mocked-up a UI for you.

We worked around it in 2011 by using the paper forms. Apparently this way you still end up “correcting” our relationship status for us (gee, thanks!) but at least – I gather – the originals are retained. So maybe in a more-enlightened time, future statisticians might be able ask about the demographics of domestic nonmonogamy and have at least some data to work with from the early 21st century.

I know you’re keen for as many people as possible to do the census digitally this year. But unless you’ve fixed your forms then my family and I – and thousands of others like us – will either have to use the paper copies you’re trying to phase out… or else knowingly lie on the digital versions. Which would you prefer?

× ×

Article posted at UTC on .

2 comments

MegaLoungeVII is the most-generous of the first 15 MegaLounges! (and other interesting statistics) [x-post /r/MegaMegaMonitor]

This self-post was originally posted to /r/MegaLoungeVII. See more things from Dan's Reddit account.

Thanks to the data that /r/MegaMegaMonitor collates, I’ve been able to throw together some fun statistics about the MegaLounge chain. Let me share some graphs with you:

This is all just something fun I threw together while I’ve been off work sick today. If there’s anything else that anybody would like to see extracted from MMM’s data, let me know: it’s all interesting stuff!

Repost posted at UTC on .

No comments

Some graphs about the MegaLounge chain

This self-post was originally posted to /r/MegaMegaMonitor. See more things from Dan's Reddit account.

Thanks to the data that MegaMegaMonitor collates, I’ve been able to throw together some fun statistics about the MegaLounge chain. Let me share some graphs with you:

This is all just something fun I threw together while I’ve been off work sick today. If there’s anything else that anybody would like to see extracted from MMM’s data, let me know: it’s all interesting stuff!

Repost posted at UTC on .

No comments

Poly and the Census – Success! (almost)

You may remember the long-running story of my letters to the Office of National Statistics, and the more-concentrated effort by another blogger, in regard to the automatic “correction” of supposedly-“erroneous” data in the 2011 census, like somebody having multiple partners or identifying as neither gender. You don’t? Well here’s a reminder: part one, part two, part three, part four.

Well: we’ve finally had some success. A response has been received from the ONS, including – at last – segments of business logic from their “correction” code.

It’s hard to tell for certain what the result of the correction will be, but one thing’s for sure – Ruth, JTA and I’s census data won’t have passed their validation! Their relationship validations BP2, BP2a, and BP2b state that it is logically-impossible for a person to have a spouse and a partner living with them in the same household.

I should invite them around for dinner sometime, and they can see for themselves that this isn’t true.

I also note that they consider it invalid for anybody to tick both or neither of the (two) gender option boxes, although again, it’s not clear from the data they’ve provided how the automatic correction occurs. Increasingly, I’m coming to suspect that this might actually be a manual process, in which case I’m wondering what guidelines there are for their operators?

One good piece of news from this FoI request, though: the ONS has confirmed that the original census data – the filled-in paper forms, which unlike the online version doesn’t enforce its validation upon you – is not adjusted. So in a hundred years time, people will be able to look back at the actual forms filled in by poly, trans, and other non-standard households around the UK, and generate actual statistics on the frequency with which these occur. It’s not much, but it’s something.

Article posted at UTC on .

2 comments

Poly and the Census – Part Four

Following up on my earlier blog posts about how data on polyamorous households is recorded in the census (see parts one, two, and three), as well as subsequent queries by Zoe O’Connell on this and related topics (how the census records data on other relationships, such as marriage between same-gender partners and civil partnerships between opposite-gender partners), there’s finally been some progress!

No; that’s a lie, I’m afraid. We’re still left wading around in the same muddy puddle. Zoe’s Freedom of Information Act request, which basically said “Okay, so you treat this kind of data as erroneous. How often does this happen?” got a response. And that response basically said, “We can’t tell you that, because we don’t have the information and it’d cost too much to work it out.” Back to square one.

Still: it looks like she’s not keen to be beaten, as she’s sent a fresh FoI request to instead ask “So what’s the algorithm you’re using to detect this erroneous data?” I was pleased to see that she went on to add, effectively, “I don’t need an explanation: send me the code if you need to,” which makes it harder for them to fall behind the “It’s too expensive!” excuse yet again.

Anyway: it’s one to watch. And needless to say, I’ll keep you all posted when anything changes…

Article posted at UTC on .

No comments

Poly and the Census – Part Three

Unimpressed with the slow response time that I and others were getting to my query to the Office of National Statistics (to which I still never received a response) the month before last, Zoe O’Connell decided to send a Freedom of Information Act request demanding a response to a couple of similar questions. After some hassling (I suppose they’ve been busy, with the census and all), they finally responded. The original request and the full response is online now, as is Zoe’s blog post about the response. But here’s the short version of the response:

Polygamous marriages are not legally recognised in the UK and therefore any data received from a questionnaire that appeared to show polygamous relationship in the manner that you suggest would be read as an error. It is recognised that the majority of respondents recording themselves as being in a polygamous relationship in a UK census do so erroneously, for example, ticking the wrong box for one household member on the relationships question.

Therefore, the data to be used for statistical purposes would be adjusted by changing one or more of these relationships, so that each respondent is in a relationship with no more than one person. This is consistent with all previous UK censuses, and others around the world.

A copy of the original questionnaire would be retained as part of the historical record which would show such relationships as they were recorded. We do not attempt to amend the original record.

Any mismatches between the indicated sex and marital status of respondents will be resolved using a probabilistic statistical system which will not necessarily deal with each case in the same way. The system will look at other responses for each person, including those for the Household relationships, and will alter one or more variables to make the response consistent. In the example that you propose, it would either change the sex of one individual, or change the marital status to “Same-sex civil partnership”, depending on which is considered statistically more likely to be correct.

Honestly, I’m not particularly impressed. They’ve committed to maintaining a historical record of the original, “uncorrected” data, so that future statisticians can get a true picture of the answers given, but this is about the only positive point in this response. Treating unusual data as erroneous is akin to pretending that a societal change doesn’t exist, and that this approach is “consistent with previous censuses” neglects to entertain the possibility that this data has value that it might not have had previously.

Yes, there will be erroneous data: people who accidentally said that they had two husbands when they only have one, for example. And yes, this can probably (although they don’t state how they know to recognise this) be assumed to be more common that genuine cases where somebody meant to put that on their census (although there will also be an error rate amongst these people, too). But taking the broad brush approach of assuming that every case can be treated as an error reeks of the same narrow-mindedness as the (alleged; almost-certainly an urban legend) statement by Queen Victoria that lesbianism “didn’t exist.”

“Fixing” the data using probabilities just results in a regression towards the mean: “Hmm; this couple of men say they’re married: they could be civil partners, or it could be a mistake… but they’re in a county with statistically-few few gay people, so we’ll assume the latter.” Really: what?

I’m not impressed, ONS.

Update: a second FoI request now aims to determine how many “corrections” have been made on censuses, historically. One to watch.

Poly and the Census – Part Two

No reply yet from the Office of National Statistics after the letter I sent the other week, but I imagine that they’ve been busy, what with the census and everything. Needless to say, I’ll keep you posted.

However, in the meantime somebody’s one-upped me and has put in a Freedom of Information request, which – of course – the law mandates that they respond to. I should’a thought of that. Anyway, you can read the request here, and there’s options to follow it by RSS and/or email if you want updates.

Update (27th April 2011): Still no word in response to the FoI request.

Article posted at UTC on .

1 mention

Poly and the Census

I’ve just sent a letter to the Office of National Statistics, about the 2011 census. In case you’re interested, or if you’d like to send a similar letter to ensure that your own living arrangements will be correctly recorded, you’re welcome to use it as a template. My letter reads:

Census Customer Services
ONS
Segensworth Road
Titchfield
Fareham
Hampshire
PO15 5RR

10th March 2011

Dear Sir or Madam,

Re: Households containing people with multiple romantic relationships in the 2011 census

I write to you to ensure that the data that will be provided by my household for the 2011 census will be properly recorded and processed. I am a supporter of the census and understand its importance, but I am concerned that my response, and the response of others in my position, is at risk of being misunderstood or misinterpreted as a mistake.

I live with my partner and her husband in a three-way committed relationship. We have attempted to express this on the census form: my partner has checked the “husband or wife” box in reference to her relationship to her husband, and she had checked the “partner” box in reference to her relationship with me. Like many people in this kind of relationship, our family is the victim of unfair discrimination, and it’s important to us that we can be counted so that future lawmakers, armed with the statistical evidence, can pass policy that is fair to all: including those who choose to be romantically-involved with multiple people at the same time.

Please give me your assurances that our data will be correctly recorded. If this is not possible, please advise me to whom I should write to put the case that this should be changed.

Yours sincerely,

Dan Q

If the census isn’t the time for a little Poly-activism, then I don’t know when is. I’ll keep you posted if I get a response.

Article posted at UTC on .

1 mention