python – Dan Q

I Wrote the Same Code Six Times!

I was updating my CV earlier this week in anticipation of applying for a handful of interesting-looking roles¹ and I was considering quite how many different tech stacks I claim significant experience in, nowadays.

There are languages I’ve been writing in every single week for the last 15+ years, of course, like PHP, Ruby, and JavaScript. And my underlying fundamentals are solid.

But is it really fair for me to be able to claim that I can code in Java, Go, or Python: languages that I’ve not used commercially within the last 5-10 years?

Obviously, I couldn’t just let that question lie². Let’s find out!

The Test

I fished around on Glassdoor for a bit to find a medium-sized single-sitting tech test, and found a couple of different briefs that I mashed together to create this:

In an object-oriented manner, implement an LRU (Least-Recently Used) cache:

The size of the cache is specified at instantiation.

Arbitrary objects can be put into the cache, along with a retrieval key in the form of a string. Using the same string, you can get the objects back.

If a put operation would increase the number of objects in the cache beyond the size limit, the cached object that was least-recently accessed (by either a put or get operation) is removed to make room for it.

putting a duplicate key into the cache should update the associated object (and make this item most-recently accessed).

Both the get and put operations should resolve within constant (O(1)) time.

Add automated tests to support the functionality.

My plan was to implement a solution to this challenge, in as many of the languages mentioned on my CV as possible in a single sitting.

But first, a little Data Structures & Algorithms theory:

The Theory

Simple case with O(n) complexity

The simplest way to implement such a cache might be as follows:

Use a linear data structure like an array or linked list to store cached items.
On get, iterate through the list to try to find the matching item.
- If found: move it to the head of the list, then return it.
On put, first check if it already exists in the list as with get:
- If it already exists, update it and move it to the head of the list.
- Otherwise, insert it as a new item at the head of the list.
  - If this would increase the size of the list beyond the permitted limit, pop and discard the item at the tail of the list.

Illustration of a simple Cache class with a linked list data structure containing three items, referenced from the class by its head item. It has 'get' and 'put' methods. — It’s simple, elegant and totally the kind of thing I’d accept if I were recruiting for a junior or graduate developer. But we can do better.

The problem with this approach is that it fails the requirement that the methods “should resolve within constant (O(1)) time”³.

Of particular concern is the fact that any operation which might need to re-sort the list to put the just-accessed item at the top ⁴. Let’s try another design:

Achieving O(1) time complexity

Here’s another way to implement the cache:

Retain cache items in a doubly-linked list, with a pointer to both the head and tail
Add a hash map (or similar language-specific structure) for fast lookups by cache key
On get, check the hash map to see if the item exists.
- If so, return it and promote it to the head (as described below).
On put, check the hash map to see if the item exists.
- If so, promote it to the head (as described below).
- If not, insert it at the head by:
  - Updating the prev of the current head item and then pointing the head to the new item (which will have the old head item as its next), and
  - Adding it to the hash map.
  - If the number of items in the hash map would exceed the limit, remove the tail item from the hash map, point the tail at the tail item’s prev, and unlink the expired tail item from the new tail item’s next.
To promote an item to the head of the list:
1. Follow the item’s prev and next to find its siblings and link them to one another (removes the item from the list).
2. Point the promoted item’s next to the current head, and the current head‘s prev to the promoted item.
3. Point the head of the list at the promoted item.

Illustration of a more-sophisticated Cache class with a doubly-linked list data structure containing three items, referenced from the class by its head and tail item. It has 'get' and 'put' methods. It also has a separate hash map indexing each cached item by its key. — Looking at a plate of pointer-spaghetti makes me strangely hungry.

It’s important to realise that this alternative implementation isn’t better. It’s just different: the “right” solution depends on the use-case⁵.

The Implementation

That’s enough analysis and design. Time to write some code.

Picking a handful of the more-useful languages on my CV⁶, I opted to implement in:

Ruby (with RSpec for testing and Rubocop for linting)
PHP (with PHPUnit for testing)
TypeScript (running on Node, with Jest for testing)
Java (with JUnit for testing)
Go (which isn’t really an object-oriented language but acts a bit like one, amirite?)
Python (probably my weakest language in this set, but which actually ended up with quite a tidy solution)

Naturally, I open-sourced everything if you’d like to see for yourself. It all works, although if you’re actually in need of such a cache for your project you’ll probably find an alternative that’s at least as good (and more-likely to be maintained!) in a third-party library somewhere!

What did I learn?

This was actually pretty fun! I might continue to expand my repo by doing the same challenge with a few of the other languages I’ve used professionally at some point or another⁷.

And there’s a few takeaways I got from this experience –

Lesson #1: programming more languages can make you better at all of them

As I went along, one language at a time, I ended up realising improvements that I could make to earlier iterations.

For example, when I came to the TypeScript implementation, I decided to use generics so that the developer can specify what kind of objects they want to store in the cache, rather than just a generic Object, and better benefit type-safety. That’s when I remembered that Java supports generics, too, so I went back and used them there as well.

In the same way as speaking multiple (human) languages or studying linguistics can help unlock new ways of thinking about your communication, being able to think in terms of multiple different programming languages helps you spot new opportunities. When in 2020 PHP 8 added nullsafe operators, union types, and named arguments, I remember feeling confident using them from day one because those features were already familiar to me from Ruby⁸, TypeScript⁹, and Python¹⁰, respectively.

Lesson #2: even when I’m rusty, I can rely on my fundamentals

I’ve applied for a handful of jobs now, but if one of them had invited me to a pairing session on a language I’m rusty on (like Java!) I might’ve felt intimidated.

But it turns out I shouldn’t need to be! With my solid fundamentals and a handful of other languages under my belt, I understand when I need to step away from the code editor and hit the API documentation. Turns out, I’m in a good position to demo any of my language skills.

I remember when I was first learning Go, I wanted to make use of a particular language feature that I didn’t know whether it had. But because I’d used that feature in Ruby, I knew what to search for in Go’s documentation to see if it was supported (it wasn’t) and if so, what the syntax was¹¹.

Lesson #3: structural rules are harder to gearshift than syntactic ones

Switching between six different languages while writing the same application was occasionally challenging, but not in the ways I expected.

I’ve had plenty of experience switching programming languages mid-train-of-thought before. Sometimes you just have to flit between the frontend and backend of your application!

But this time around I discovered: changes in structure are apparently harder for my brain than changes in syntax. E.g.:

Switching in and out of Python’s indentation caught me out at least once (might’ve been better if I took the time to install the language’s tools into my text editor first!).
Switching from a language without enforced semicolon line ends (e.g. Ruby, Go) to one with them (e.g. Java, PHP) had me make the compiler sad several times.
This gets even tougher when not writing the language but writing about the language: my first pass at the documentation for the Go version somehow ended up with Ruby/Python-style #-comments instead of Go/Java/TypeScript-style //-comments; whoops!

I’m guessing that the part of my memory that looks after a language’s keywords, how a method header is structured, and which equals sign to use for assignment versus comparison… are stored in a different part of my brain than the bit that keeps track of how a language is laid-out?¹²

Okay, time for a new job

I reckon it’s time I got back into work, so I’m going to have a look around and see if there’s any roles out there that look exciting to me.

If you know anybody who’s looking for a UK-based, remote-first, senior+, full-stack web developer with 25+ years experience and more languages than you can shake a stick at… point them at my CV, would you?

Footnotes

¹ I suspect that when most software engineers look for a new job, they filter to the languages, frameworks, they feel they’re strongest at. I do a little of that, I suppose, but I’m far more-motivated by culture, sector, product and environment than I am by the shape of your stack, and I’m versatile enough that technology specifics can almost come second. So long as you’re not asking me to write VB.NET.

² It’s sort-of a parallel to how I decided to check the other week that my Gutenberg experience was sufficiently strong that I could write standard ReactJS, too.

³ I was pleased to find a tech test that actually called for an understanding of algorithm growth/scaling rates, so I could steal this requirement for my own experiment! I fear that sometimes, in their drive to be pragmatic and representative of “real work”, the value of a comprehension of computer science fundamentals is overlooked by recruiters.

⁴ Even if an algorithm takes the approach of creating a new list with the inserted/modified item at the top, that’s still just a very-specific case of insertion sort when you think about it, right?

⁵ The second design will be slower at writing but faster at reading, and will scale better as the cache gets larger. That sounds great for a read-often/write-rarely cache, but your situation may differ.

⁶ Okay, my language selection was pretty arbitrary. But if I’d have also come up with implementations in Perl, and C#, and Elixir, and whatever else… I’d have been writing code all day!

⁷ So long as I’m willing to be flexible about the “object-oriented” requirement, there are even more options available to me. Probably the language that I last wrote longest ago would be Pascal: I wonder how much of that I remember?

⁸ Ruby’s safe navigation/”lonely” operator did the same thing as PHP’s nullsafe operator since 2015.

⁹ TypeScript got union types back in 2015, and apart from them being more-strictly-enforced they’re basically identical to PHP’s.

¹⁰ Did you know that Python had keyword arguments since its very first public release way back in 1994! How did it take so many other interpreted languages so long to catch up?

¹¹ The feature was the three-way comparison or “spaceship operator”, in case you were wondering.

¹² I wonder if anybody’s ever laid a programmer in an MRI machine while they code? I’d be really interested to see if different bits of the brain light up when coding in functional programming languages than in procedural ones, for example!

Building a Secret Cabinet

We’ve recently had the attics of our house converted, and I moved my bedroom up to one of the newly-constructed rooms.

To make the space my own, I did a little light carpentry up there: starting with a necessary reshaping of the doors, then moving on to shelving and eventually… a secret cabinet!

I’d love to tell you about how I built it: but first, a disclaimer! I am a software engineer, and with good reason. Letting me near a soldering iron is ill-advised. Letting me use a table saw is tempting fate.

Letting me teach you anything about how you should use a soldering iron or a table saw is, frankly, asking for trouble.

A spirit level on an unfinished shelf, under a window and beneath an uncarpeted floor. — Knowing that I’d been short on shelf space in my old bedroom, I started work on fitting shelves for my new bedroom before the carpet had even arrived.

Building a secret cabinet wasn’t part of my plan, but came about naturally after I got started. I’d bought a stack of pine planks and – making use of Ruth’s table saw – cut them to squarely fit beneath each of the two dormer windows¹. While sanding and oiling the wood I realised that I had quite a selection of similarly-sized offcuts and found myself wondering if I could find a use for them.

Dan drinks a 0% alcohol beer in front of three upright planks of wood. — The hardest part of sanding and oiling wood on the hottest day of the year is all the beer breaks you have to take. Such a drag.

I figured I had enough lumber to insert a small cabinet into one of the bookshelves, and that got me thinking… what about if it were a secret cabinet, disguised as books unless you knew where to look. Or to go one step further: what if it had some kind of electronic locking mechanism that could be triggered from somewhere else in the room².

Magpie decal 'perched' on a light switch. — There are other ways in which I’ve made my new room distinctly-“mine” – like the pair of magpies – but probably the secret cabinet is the most-distinctive.

Not wanting to destroy a stack of real books, which is the traditional way to get a collection of book spines for the purpose of decorating a “fake bookshelf” panel³, I looked online and discovered the company that made the fake book spines used at the shop of my former employer. They looked ideal: carefully shaped and painted panels with either an old-school or contemporary look.

Buuut, they don’t seem to be well-equipped for short runs and are doubtless pricey, so I looked elsewhere and found the eBay presence of Beatty Lockey Antiques in Loewstof. They’d acquired a stack of them second-hand from the set of Netflix’s The School for Good and Evil.⁴

(By the way: at time of writing they’ve still got a few panels left, if you want to make your own…)

I absolutely must sing the praises of Brad at Beatty Lockey Antiques who, after the first delivery of fake book fronts was partially-damaged in transit, was super quick about helping me find the closest-available equivalent (I’d already measured-up based on the one I’d thought I was getting) and sent a replacement.

The cabinet is just a few bits of wood glued together and reinforced with L-shaped corner braces, with a trio of thin strips – made from leftover architrave board – attached using small brass hinges. The fake book fronts are stuck to the strips using double-sided mounting tape left over from installing a bathroom mirror. A simple magnetic clasp holds the door shut when pushed closed⁵, and the hinges are inclined to “want” the door to stand half-open, which means it only needs a gentle push away from the magnetic catch to swing it open.

I mounted a Raspberry Pi Zero W into a rear corner inside the cabinet⁶, and wired it up via a relay to what was sold to me as a “large push-pull solenoid”, then began experimenting with the position in which I’d need to mount it to allow it to “kick” open the door, against the force of the magnetic clasp⁷.

This was, amazingly, the hardest part of the whole project! Putting the solenoid too close to the door didn’t work: it couldn’t “push” it from a standing start. Too far away, and the natural give of the door took the strain without pushing it open. Just the right distance, and the latch had picked up enough momentum that its weight “kicked” the door away from the magnet and followed-through to ensure that it kept moving.

A second solenoid, mounted inside the top of the cabinet, slides into the “loop” part of a large bolt fitting, allowing the cabinet to be electronically “locked”.

A Raspberry Pi Zero, relay, and solenoid assembly on the bottom outside edge of the inside of the cabinet. — I seriously must’ve spent about an hour getting the position of that little “kicker” in the bottom right *just right*.

Next up came the software. I started with a very simple Python program⁸ that would run a webserver and, on particular requests, open the lock solenoid and push with the “kicker” solenoid.

#!/usr/bin/python
#
# a basic sample implementation of a web interface for a secret cabinet
#
# setup:
#   sudo apt install -y python3-flask
#   wget https://github.com/sbcshop/Zero-Relay/blob/master/pizero_2relay.py
#
# running:
#   sudo flask --app web run --host=0.0.0.0 --port 80

from flask import Flask, redirect, url_for
import pizero_2relay as pizero
from time import sleep

# set up pizero_2relay with the two relays attached to this Pi Zero:
r1 = pizero.relay("R1") # The "kicker" relay
r2 = pizero.relay("R2") # The "locking bolt" relay

app = Flask(__name__)

# GET / - nothing here
@app.route("/")
def index():
  return "Nothing to see here."

# GET /relay - show a page with "open" and "lock" links
@app.route("/relay")
def relay():
  return "<html><head><meta name='viewport' content='width=device-width, initial-scale=1'></head><body><ul><li><a href='/relay/open'>Open</a></li><li><a href='/relay/lock'>Lock</a></li></ul>"

# GET /relay/open - open the secret cabinet then return to /relay
# This ought to be a POST request in your implementation, and you probably
# want to add some security e.g. a 
@app.route("/relay/open")
def open():
  # Retract the lock:
  r2.off()
  sleep(0.5)
  # Fire the kicker twice:
  r1.on()
  sleep(0.25)
  r1.off()
  sleep(0.25)
  r1.on()
  sleep(0.25)
  r1.off()
  # Redirect back:
  return redirect(url_for('relay'))

@app.route("/relay/lock")
def lock():
  # Engage the lock:
  r2.on()
  return redirect(url_for('relay'))

Don’t use this code as-is on any kind of open network, obviously. Follow the comments for some tips on what you’ll need to change.

Once I had something I could trigger from a web browser or with curl, I could start experimenting with trigger mechanisms. I had a few ideas (and prototyped a couple of them), including:

A mercury tilt switch behind a different book, so you pull it to release the cabinet in the style of a classic movie secret door.
A microphone that listens for a specific pattern of knocks on a nearby surface.

I had far too much fun playing about with crappy prototypes.

An RFID reader mounted underneath another surface, and a tag on the underside of an ornament: moving the ornament to the “right” place on the surface triggers the cabinet (elsewhere in the room).
The current design, shown in the video above, where a code⁹ is transmitted to the cabinet for verification.

I think I’m happy with what I’ve got going on with it now. And it’s been a good opportunity to improve my carpentry, electronics, and Python.

Footnotes

¹ The two dormer windows, wouldn’t you guarantee it, were significantly different widths despite each housing a window of the same width. Such are the quirks of extending a building that the previous occupier had previously half-heartedly tried to extend, I guess.

² Why yes, I am a big fan of escape rooms. Why do you ask?

³ For one thing, I live with JTA, and I’m confident that he’d somehow be able to hear the silent screams of whatever trashy novels I opted to sacrifice for the good of the project.

⁴ As a bonus, my 10-year-old is a big fan of the book series that inspired the film (and a more-muted fan of the film itself) and she was ever-so excited at my project using real-life parts of the set of the movie… that she’s asked me to make a similar secret cabinet for her, when we get around to redecorating her room later in the year!

⁵ If I did it again, I might consider using a low-powered electromagnetic lock to hold the door shut. In this design, I used a permanent magnet and a pair of latch solenoids: one to operate a bolt, the second to “kick” the door open against the pull of the magnet, and… it feels a little clumsier than a magnetic lock might’ve.

⁶ That double-sided mounting tape really came in handy for this project!

⁷ Props to vlogger Technology Connections, one of whose excellent videos on the functionality of 1970s pinball tables – maybe this one? – taught me what a latch solenoid was in the first place, last year, which probably saved me from the embarrassment of trying to do this kind of thing with, I don’t know, a stepper motor or something.

⁸ I’m not a big fan of Python normally, but the people who made my relays had some up with a convenience library for them that was written in it, so I figured it would do.

⁹ Obviously the code isn’t A-B; I changed it temporarily for the video.

Nginx Caching for Passenger Applications

Suppose you’re running an application on a Passenger + Nginx powered server and you want to add caching.

Perhaps your application has a dynamic, public endpoint but the contents don’t change super-frequently or it isn’t critically-important that the user always gets up-to-the-second accuracy, and you’d like to improve performance with microcaching. How would you do that?

Where you’re at

Diagram showing the Internet connecting to an Nginx+Passenger webserver, connecting to an application written for Ruby, Python, or NodeJS. — Not pictured: the rest of the Internet.

Your configuration might look something like this:

server {
  # listen, server_name, ssl, logging etc. directives go here
  # ...

  root               /your/application;
  passenger_enabled  on;
}

What you’re looking for is proxy_cache and its sister directives, but you can’t just insert them here because while Passenger does act act like an upstream proxy (with parallels like e.g. passenger_pass_header which mirrors the behaviour of proxy_pass_header), it doesn’t provide any of the functions you need to implement proxy caching of non-static files.

Where you need to be

Instead, what you need to to is define a second server, mount Passenger in that, and then proxy to that second server. E.g.:

# Set up a cache
proxy_cache_path /tmp/cache/my-app-cache keys_zone=MyAppCache:10m levels=1:2 inactive=600s max_size=100m;

# Define the actual webserver that listens for Internet traffic:
server {
  # listen, server_name, ssl, logging etc. directives go here
  # ...

  # You can configure different rules by location etc., but here's a simple microcache:
  location / {
    proxy_pass http://127.0.0.1:4863; # Proxy all traffic to the application server defined below
    proxy_cache           MyAppCache; # Use the cache defined above
    proxy_cache_valid     200 3s;     # Treat HTTP 200 responses as valid; cache them for 3 seconds
    proxy_cache_use_stale updating;   # (Optional) send outdated response while background-updating cache
    proxy_cache_lock      on;         # (Optional) only allow one process to update cache at once
  }
}

# (Local-only) application server on an arbitrary port number to act as the upstream proxy:
server {
  listen 127.0.0.1:4863;

  root               /your/application;
  passenger_enabled  on;
}

The two key changes are:

Passenger moves to a second server block, localhost-only, on an arbitrary port number (doesn’t need HTTPS, of course, but if your application detects/”expects” HTTPS you might need to tweak your headers).
Your main server block proxies to the second as its upstream, and you can add whatever caching directives you like.

Obviously you’ll need to be smarter if you host a mixture of public and private content (e.g. send Vary: headers from your application) and if you want different cache durations on different addresses or types of content, but there are already great guides to help with that. I only wrote this post because I spent some time searching for (nonexistent!) passenger_cache_ etc. rules and wanted to save the next person from the same trouble!

OpenAI-powered Linux shell uses AI to Do What You Mean

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

It’s like Alexa/Siri/Cortana for your terminal!

This is a basic Python shell (really, it’s a fancy wrapper over the system shell) that takes a task and asks OpenAI for what Linux bash command to run based on your description. For safety reasons, you can look at the command and cancel before actually running it.

…

River

Of all the stupid uses of OpenAI’s GPT-3, this might be the most-amusing. It’s really interesting to see how close – sometimes spot-on – the algorithm comes to writing the right command when you “say what you mean”. Also, how terribly, terribly ill-advised it would be to actually use this for real.

Python Regular Expression HOWTO

This is a repost promoting content originally bookmarked via del.icio.us. See more del.icio.us imports or more things Dan's reposted.

Bookmarked via del.icio.us: Python Regular Expression HOWTO.

Dive Into Python

This is a repost promoting content originally bookmarked via del.icio.us. See more del.icio.us imports or more things Dan's reposted.

Bookmarked via del.icio.us: Dive Into Python.

Python from novice to pro

Amusing Line From The Python Manual

I just extracted the following line from the Python documentation (for those of you who don’t know, Python is a relatively-new and somewhat unusual programming language who’s name is derived not from the snake but from Monty Python):

Attempts to pickle unpicklable objects will raise the PicklingError exception; when this happens, an unspecified number of bytes may have already been written to the underlying file.

You have to love any manual with that line in it! It almost beats the famous Fortran line about defining pi as a constant in case it’s value changes.

Tag: python

I Wrote the Same Code Six Times!

Contents

The Test

The Theory

Simple case with O(n) complexity

Achieving O(1) time complexity

The Implementation

What did I learn?

Lesson #1: programming more languages can make you better at all of them

Lesson #2: even when I’m rusty, I can rely on my fundamentals

Lesson #3: structural rules are harder to gearshift than syntactic ones

Okay, time for a new job

Footnotes

Building a Secret Cabinet

Footnotes

Nginx Caching for Passenger Applications

Where you’re at

Where you need to be

OpenAI-powered Linux shell uses AI to Do What You Mean

Python Regular Expression HOWTO

Dive Into Python

Amusing Line From The Python Manual