ssh – Dan Q

Run your own WireGuard VPN

With the news that the British government are considering requiring identity checks for age verification before allowing people to use VPNs, it’s time for my periodic reminder that you don’t have to use a “VPN provider” to use a VPN¹.

As I’ll demonstrate, it’s surprisingly easy to spin up your own VPN provider on a virtual machine hosted by your choice of the cloud providers. You pay for the hours you need it², and then throw it away afterwards.

Today, I’ll be using Linode to host my “throwaway” VPN provider for a price of USD $0.0075 per hour ($5/month if I ran it full-time), using a Linode StackScript I created for this purpose.

If you’d prefer to use GCP, AWS Azure, or whomever else you like: all you need is a Debian 13 VM with a public IP address (the cheapest one available is usually plenty!) and this bash script.

Screenshot from Linode's Web interface, showing my StackScript and indicating the location of the Deploy New Linode button. — If you prefer the command-line, Linode’s got an API. But we’re going for ‘easy’ today, so it’ll all be clicking buttons and things.

First, spin up a VM and run my script³. If you’re using Linode, you can do this by going to my StackScript and clicking ‘Deploy New Linode’.

Linode configuration screenshot with the key options highlighted as described below. — You might see more configuration options than this, but you can ignore them.

Choose any region you like (I’m putting this one in Paris!), select the cheapest “Shared CPU” option – Nanode 1GB – and enter a (strong!) root password, then click Create Linode.

It’ll take a few seconds to come up. Watch until it’s running.

My script automatically generates configuration for your local system. Once it’s up and running you can use the machine’s IP address to download wireguard.conf locally. For example, if your machine has the IP address 172.239.9.151, you might type scp -o StrictHostKeyChecking=no root@172.239.9.151:wireguard.conf ./ – note that I disable StrictHostKeyChecking so that my computer doesn’t cache the server’s SSH key (which feels a bit pointless for a “throwaway” VM that I’ll never connect to a second time!).

If you’re on Windows and don’t have SSH/SCP, install one. PuTTY remains a solid choice.

File doesn’t exist? Give it a minute and try again; maybe my script didn’t finish running yet! Still nothing? SSH into your new VM and inspect stackscript.log for a complete log of all the output from my script to see what went wrong.

Open up WireGuard on your computer, click the “Import tunnel(s) from file” button, and give it the file you just downloaded.

You can optionally rename the new connection. Or just click “Activate” to connect to your VPN!

You can test your Internet connection is being correctly routed by your VPN by going to e.g. icanhazip.com or ipleak.net: you should see the IP address of your new virtual machine and/or geolocation data that indicates that you’re in your selected region.

When you’re done with your VPN, just delete the virtual machine. Many providers use per-minute or even per-second fractional billing, so you can easily end up spending only a handful of cents in order to use a VPN for a reasonable browsing session.

When you’re done, just disconnect and – if you’re not going to use it again immediately – delete the virtual machine so you don’t have to pay for it for a minute longer than you intend⁴.

I stopped actively paying for VPN subscriptions about a decade ago and, when I “need” the benefits of a VPN, I’ve just done things like what I’ve described above. Compared to a commercial VPN subscription it’s cheap, (potentially even-more) private, doesn’t readily get “detected” as a VPN by the rare folks who try to detect such things, and I can enjoy my choice of either reusable or throwaway IP addresses from wherever I like around the globe.

And if the government starts to try to age-gate commercial VPNs… well then that’s just one more thing going for my approach, isn’t it?

Footnotes

¹ If you’re a heavy, “always-on” VPN user, you might still be best-served by one of the big commercial providers, but if you’re “only” using a VPN for 18 hours a day or less then running your own on-demand is probably cheaper, and gives you some fascinating benefits.

² Many providers have coupons equivalent to hundreds of hours of free provision, so as long as you’re willing to shuffle between cloud providers you can probably have a great and safe VPN completely for free; just sayin’.

³ Obviously, you shouldn’t just run code that strangers give you on the Internet unless you understand it. I’ve tried to make my code self-explanatory and full of comments so you can understand what it does – or at least understand that it’s harmless! – but if you don’t know and trust me personally, you should probably use this as an excuse to learn what you’re doing. In fact, you should do that anyway. Learning is fun.

⁴ Although even if you forget and it runs for an entire month before your billing cycle comes up, you’re out, what… $5 USD? Plenty of commercial VPN providers would have charged you more than that!

Rebels in the Sky

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

It’s the year 2101. Corporations have taken over the world. The only way to be free is to join a pirate crew and start plundering the galaxy. The only means of survival is to play basketball.

Now it’s your turn to go out there and make a name for yourself. Create your crew and start wandering the galaxy in search of worthy basketball opponents.

The game is under heavy development and breaking changes are often introduced. If you can’t continue an old game because the save file is invalid, you probably need to start a new one or open an issue to check if the save file can be migrated.

…

Just try it out!

Connect via SSH to try the game.

ssh rebels.frittura.org -p 3788

Save files are deleted after 2 days of inactivity.

…

Inverness, in Rebels in the Sky

I feel like I’m reading a lot about SSH lately and how it can be used for exotic and unusual tasks. Tarpitting‘s fun, of course, but really what inspires me is all these dinky projects like ssh tiny.christmas that subvert the usual authentication-then-terminal flow that you expect when you connect to an SSH server.

These kinds of projects feel more like connecting to a BBS. And that’s pretty retro (and cool!).

Anyway: Rebels in the Sky is a networked multiplayer terminal-based game about exploring the galaxy with a team of basketball-loving space pirates. I met the main developer on a forum and they seem cool; I’m interested to see where this quirky little project ends up going!

(The pixel art planets, based on Deep-Fold’s work, are amazing too. Honestly impressed to see animations like these transmitted over a shell!)

Why does SSH send 100 packets per keystroke?

This is a repost promoting content originally published elsewhere. See more things Dan's reposted.

…

Further analysis on a smaller pcap pointed to these mysterious packets arriving ~20ms apart.

This was baffling to me (and to Claude Code). We kicked around several ideas like:

SSH flow control messages

PTY size polling or other status checks

Some quirk of bubbletea or wish

One thing stood out – these exchanges were initiated by my ssh client (stock ssh installed on MacOS) – not by my server.

…

In 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys.

That makes a lot of sense for regular ssh sessions, where privacy is critical. But it’s a lot of overhead for an open-to-the-whole-internet game where latency is critical.

…

Nolen Royalty, in Why does SSH send 100 packets per keystroke?

Keystroke timing obfuscation: I could’ve told you that! Although I wouldn’t necessarily have leapt to the possibility of mitigating it server-side by patching-out support for (or at least: the telegraphing of support for!) it; that’s pretty clever.

Altogether this is a wonderful piece demonstrating the whole “engineer mindset”. Detecting a problem, identifying it, understanding it, fixing it, all tied-up in an engaging narrative.

And after playing with his earlier work, ssh tiny.christmas – which itself inspired me to learn a little Bubble Tea/Wish (I’ve got Some Ideas™️) – I’m quite excited to see where this new ssh-based project of Royalty’s is headed!

Endless SSH Tarpit on Debian

Tarpitting SSH with Endlessh

I had a smug moment when I saw security researcher Rob Ricci and friends’ paper empirically analysing brute-force attacks against SSH “in the wild”.¹ It turns out that putting all your SSH servers on “weird” port numbers – which I’ve routinely done for over a decade – remains a pretty-effective way to stop all that unwanted traffic², whether or not you decide to enhance that with some fail2ban magic.

But then I saw a comment about Endlessh. Endlessh³ acts like an SSH server but then basically reverse-Slow-Loris’s the connecting client, very gradually feeding it an infinitely-long SSH banner and hanging it for… well, maybe 15 seconds or so but possibly up to a week.

Installing an Endlessh tarpit on Debian 12

I was just setting up a new Debian 12 server when I learned about this. I’d already moved the SSH server port away from the default 22⁴, so I figured I’d launch Endlessh on port 22 to slow down and annoy scanners.

Installation wasn’t as easy as I’d hoped considering there’s a package. Here’s what I needed to do:

Move any existing SSH server to a different port, if you haven’t already, e.g. as shown in the footnotes.
Install the package, e.g.: sudo apt update && sudo apt install -y endlessh
Permit Endlessh to run on port 22: sudo setcap 'cap_net_bind_service=+ep' /usr/bin/endlessh
Modify /etc/systemd/system/multi-user.target.wants/endlessh.service in the following ways:
1. uncomment AmbientCapabilities=CAP_NET_BIND_SERVICE
2. comment PrivateUsers=true
3. change InaccessiblePaths=/run /var into InaccessiblePaths=/var
Reload the modified service: sudo systemctl daemon-reload
Configure Endlessh to run on port 22 rather than its default of 2222: echo "Port 22" | sudo tee /etc/endlessh/config
Start Endlessh: sudo service endlessh start

To test if it’s working, connect to your SSH server on port 22 with your client in verbose mode, e.g. ssh -vp22 example.com and look for banner lines full of random garbage appearing at 10 second intervals.

It doesn’t provide a significant security, but you get to enjoy the self-satisfied feeling that you’re trolling dozens of opportunistic script kiddies a day.

Footnotes

¹ It’s a good paper in general, if that’s your jam.

² Obviously you gain very little security by moving to an unusual port number, given that you’re already running your servers in “keys-only” (PasswordAuthentication no) configuration mode already, right? Right!? But it’s nice to avoid all the unnecessary logging that wave after wave of brute-force attempts produce.

³ Which I can only assume is pronounced endle-S-S-H, but regardless of how it’s said out loud I appreciate the wordplay of its name.

⁴ To move your SSH port, you might run something like echo "Port 12345" | sudo tee /etc/ssh/sshd_config.d/unusual-port.conf and restart the service, of course.

How To Use SSH Tunnelling To Allow Services To Pass Through A Firewall

[this post has been partially damaged during a server failure on 11 July 2004; with the exception of the images, it was recovered on 13 October 2018]

Paul has been stuck with a problem of late – he’s now living in university accomodation, and he’s found that he can’t connect through the university firewall to his external mail server. I advised him that it’s possible to set up an ‘SSH Tunnel’ (through central.aber.ac.uk) to fix this problem, but he hasn’t met with much success (see his blog entry for more details). In any case, here’s my investigation (and solution) to the problem.

How To Use SSH Tunnelling To Allow Services To Pass Through A Firewall
In my example, I’m going to try the opposite to what Paul is trying to achieve. I’m going to try to allow my POP3 e-mail client to get access to the university e-mail server (pophost.aber.ac.uk). As things stand, this server is on the other side of the university firewall, and is inaccessible from outside. The server central.aber.ac.uk, however, is accessible from both sides of the firewall. So what I’ve got is this (yes, I know that this is a gross oversimplification):

As you can see, connecting from my home PC is futile:

C:\Documents and Settings\Dan>telnet pophost.aber.ac.uk 110 Connecting To pophost.aber.ac.uk...Could not open connection to the host, on por t 110: Connect failed

But if I SSH-in to central.aber.ac.uk…

central:~ $ telnet pophost.aber.ac.uk 110 Trying 144.124.16.40... Connected to pophost.aber.ac.uk. Escape character is '^]'. +OK mailsplit Oct 2000 ready

So, what I need to do is to tell my SSH client to connect to central.aber.ac.uk, and forward specific traffic through the firewall to the mail server. Here’s what I needed to know:

(a) A free TCP port number on my own computer from which I can virtually ‘pipe’ the connection. Most numbers over 1024 are fine. I chose ‘9110’.
(b) The name of the mail server – ‘pophost.aber.ac.uk’.
(c) The TCP port to which I wanted to connect – the standard port for a POP3 mail server is ‘110’.
(d) My user name on a server which: (1) I can connect to; (2) can connect to the server specified in (b). It happens to be ‘dlh9’.
(e) The name of the server specified in (d) (i.e. ‘central.aber.ac.uk’).
(f) My password on the server. Like I’m going to tell you that.

The syntax is:

ssh -L (a):(b):(c) (d)@(e)

I’m using the non-commercial version of SSH Secure Shell Client, so here’s what happens:

C:\Documents and Settings\Dan>"\Program Files\SSH Secure Shell\ssh2.exe" -L 9110 :pophost.aber.ac.uk:110 dlh9@central.aber.ac.uk dlh9's password: Authentication successful.

At this point, I’m ready to go. Look what happens when I connect to port 9110 on my own computer, now…

C:\Documents and Settings\Dan>telnet localhost 9110 +OK mailsplit Oct 2000 ready

I could simply point my e-mail program at the ‘mail server’ at localhost:9110, and I’d be able to collect my university e-mail (so long as my SSH connection remained open).

Hopefully this guide will help some folks out there who are struggling with this kind of thing, and in particular, help Paul.