I had a smug moment when I saw security researcher Rob Ricci and friends’ paper empirically analysing brute-force attacks against SSH “in the wild”.1 It turns out that putting all your SSH servers on “weird” port
numbers – which I’ve routinely done for over a decade – remains a pretty-effective way to stop all that unwanted traffic2,
whether or not you decide to enhance that with some fail2ban magic.
I was just setting up a new Debian 12 server when I learned about this. I’d already moved the SSH server port away from the default 224, so I figured
I’d launch Endlessh on port 22 to slow down and annoy scanners.
Installation wasn’t as easy as I’d hoped considering there’s a package. Here’s what I needed to do:
Move any existing SSH server to a different port, if you haven’t already, e.g. as shown in the footnotes.
change InaccessiblePaths=/run /var into InaccessiblePaths=/var
Reload the modified service: sudo systemctl daemon-reload
Configure Endlessh to run on port 22 rather than its default of 2222: echo "Port 22" | sudo tee /etc/endlessh/config
Start Endlessh: sudo service endlessh start
To test if it’s working, connect to your SSH server on port 22 with your client in verbose mode, e.g. ssh -vp22 example.com and look for banner lines full of random garbage
appearing at 10 second intervals.
It doesn’t provide a significant security, but you get to enjoy the self-satisfied feeling that you’re trolling dozens of opportunistic script kiddies a day.
Footnotes
1 It’s a good paper in general, if that’s your jam.
2 Obviously you gain very little security by moving to an unusual port number, given that
you’re already running your servers in “keys-only” (PasswordAuthentication no) configuration mode already, right? Right!? But it’s nice to avoid all the unnecessary
logging that wave after wave of brute-force attempts produce.
3 Which I can only assume is pronounced endle-S-S-H, but regardless of how it’s said out
loud I appreciate the wordplay of its name.
4 To move your SSH port, you might run something like echo "Port 12345" | sudo tee
/etc/ssh/sshd_config.d/unusual-port.conf and restart the service, of course.
Foundry is a wonderful virtual tabletop tool well-suited to playing tabletop roleplaying games with your friends, no
matter how far away they are. It compares very favourably to the market leader Roll20, once you
get past some of the initial set-up challenges and a moderate learning curve.
You can run it on your own computer and let your friends “connect in” to it, so long as you’re able to reconfigure your router a little, but you’ll be limited by the speed of your home
Internet connection and people won’t be able to drop in and e.g. tweak their character sheet except when you’ve specifically got the application running.
A generally better option is to host your Foundry server in the cloud. For most of its history, I’ve run mine on Fox, my NAS, but I’ve recently set one up on a more-conventional cloud virtual machine too. A couple of
friends have asked me about how to set up their own, so here’s a quick guide:
You will need…
A Foundry license ($50 USD / £48 GBP, one-off payment1)
A domain name for which you control the DNS records; you’ll need to point a domain, like “danq.me” (or a subdomain of it, e.g.
“vtt.danq.me”), at an IP address you’ll get later by creating an “A” record: your domain name registrar can probably help with this –
I mostly use Gandi and, ignoring my frustration with
recent changes to their email services, I think they’re great
An account with a cloud hosting provider: this example uses Linode but you can adapt for any of them
A basic level of comfort with the command-line
1. Spin up a server
Getting a virtual server is really easy nowadays.
You’ll need:
The operating system to be Debian 12 (or else you’ll need to adapt the instructions below)
The location to be somewhere convenient for your players: pick a server location that’s relatively-local to the majority of them to optimise for connection speeds
An absolute minimum of 1GB of storage space, I’d recommend plenty more: The Levellers’ campaign currently uses about 10GB for all of its various maps, art, videos,
and game data, so give yourself some breathing room (space is pretty cheap) – I’ve gone with 80GB for this example, because that’s what comes as standard with the 2
CPU/4GB RAM server that Linode offer
Choose a root password when you set up your server. If you’re a confident SSH user, add your public key so you can log in easily (and then
disable password authentication
entirely!).
For laziness, this guide has you run Foundry as root on your new server. Ensure you understand the implications of this.2
2. Point your (sub)domain at it
DNS propogation can be pretty fast, but… sometimes it isn’t. So get this step underway before you need it.
Your newly-created server will have an IP address, and you’ll be told what it is. Put that IP address into an A-record for your domain.
3. Configure your server
In my examples, my domain name is vtt.danq.me and my server is at 1.2.3.4. Yours will be different!
Connect to your new server using SSH. Your host might even provide a web interface if you don’t have an SSH client installed: e.g. Linode’s “Launch LISH Console” button will do pretty-much exactly that for you. Log in as root using the password you chose
when you set up the server (or your SSH private key, if that’s your preference). Then, run each of the commands below in order (the full script is available as a single file if you
prefer).
3.1. Install prerequisites
You’ll need unzip (to decompress Foundry), nodejs (to run Foundry), ufw (a firewall, to prevent unexpected surprises), nginx (a
webserver, to act as a reverse proxy to Foundry), certbot (to provide a free SSL certificate for Nginx),
nvm (to install pm2) and pm2 (to keep Foundry running in the background). You can install them all like this:
By default, Foundry runs on port 30000. If we don’t configure it carefully, it can be accessed directly, which isn’t what we intend: we want connections to go through the webserver
(over https, with http redirecting to https). So we configure our firewall to allow only these ports to be accessed. You’ll also want ssh enabled so we can remotely connect into the
server, unless you’re exclusively using an emergency console like LISH for this purpose:
Putting the domain name we’re using into a variable for the remainder of the instructions saves us from typing it out again and again. Make sure you type your domain name (that
you pointed to your server in step 2), not mine (vtt.danq.me):
DOMAIN=vtt.danq.me
3.4. Get an SSL certificate with automatic renewal
So long as the DNS change you made has propogated, this should Just Work. If it doesn’t, you might need to wait for a bit then try
again.
3.5. Configure Nginx to act as a reverse proxy for Foundry
You can, of course, manually write the Nginx configuration file: just remove the > /etc/nginx/sites-available/foundry from the end of the printf line to see
the configuration it would write and then use/adapt to your satisfaction.
set +H
printf "server {\n listen 80;\n listen [::]:80;\n server_name $DOMAIN;\n\n # Redirect everything except /.well-known/* (used for ACME) to HTTPS\n root /var/www/html/;\n if (\$request_uri !~ \"^/.well-known/\") {\n return 301 https://\$host\$request_uri;\n }\n}\n\nserver {\n listen 443 ssl http2;\n listen [::]:443 ssl http2;\n server_name $DOMAIN;\n\n ssl_certificate /etc/letsencrypt/live/$DOMAIN/fullchain.pem;\n ssl_certificate_key /etc/letsencrypt/live/$DOMAIN/privkey.pem;\n\n client_max_body_size 300M;\n\n location / {\n # Set proxy headers\n proxy_set_header Host \$host;\n proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;\n proxy_set_header X-Forwarded-Proto \$scheme;\n\n # These are important to support WebSockets\n proxy_set_header Upgrade \$http_upgrade;\n proxy_set_header Connection \"Upgrade\";\n\n proxy_pass http://127.0.0.1:30000/;\n }\n}\n" > /etc/nginx/sites-available/foundry
ln -sf /etc/nginx/sites-available/foundry /etc/nginx/sites-enabled/foundry
service nginx restart
3.6. Install Foundry
3.6.1. Create a place for Foundry to live
mkdir {vtt,data}
cd vtt
3.6.2. Download and decompress it
Substitute in your Timed URL in place of <url from website> (keep the quotation marks – " –
though!):
wget -O foundryvtt.zip "<url from website>"
unzip foundryvtt.zip
rm foundryvtt.zip
3.6.3. Configure PM2 to run Foundry and keep it running
Now you’re finally ready to launch Foundry! We’ll use PM2 to get it to run automatically in the background and keep running:
You can watch the logs for Foundry with PM2, too. It’s a good idea to take a quick peep at them to check it launched okay (press CTRL-C to exit):
pm2 logs 0
4. Start adventuring!
Provide your license key to get started, and then immediately change the default password: a new instance of Foundry has a blank default password, which means that
anybody on Earth can administer your server: get that changed to something secure!
Now you’re running on Foundry!
Footnotes
1Which currency you pay in, and therefore how much you pay, for a Foundry license depends on where in the world you are
where your VPN endpoint says you are. You might like to plan accordingly.
2 Running Foundry as root is dangerous, and you should consider the risks for yourself.
Adding a new user is relatively simple, but for a throwaway server used for a single game session and then destroyed, I wouldn’t bother. Specifically, the risk is that a vulnerability
in Foundry, if exploited, could allow an attacker to reconfigure any part of your new server, e.g. to host content of their choice or to relay spam emails. Running as a non-root user
means that an attacker who finds such a vulnerability can only trash your Foundry instance.
Earlier this year, for reasons of privacy/love of selfhosting, I moved the DanQ.me mailing list from Mailchimp to Listmonk (there’s a blog post about how I set it up), relaying
outbound messages via an SMTP server provided by my domain registrar, Gandi.
And because I learned a few things while doing so, I wrote this blog post so that next time I have to configure Postfix + DKIM, I’ll know where to find a guide. If it helps you in the meantime, that’s just a bonus.
Postfix
Running your own mailserver is a pain. I used to do it for all of my email, but – like many other nerds – when spam reached its peak and deliverability became an issue, I gave
up and oursourced it1.
Luckily, I don’t need it to do much. I just need a mail transfer agent with an (unauthenticated, but local-only) SMTP endpoint: something that Listmonk can dump emails into, which will then reach out to the mailservers representing each of the recipients and
relay them on. A default install of Postfix does all that out-of-the-box, so I ran sudo apt install postfix, accepted all the default
options, and put the details into Listmonk.
Next, I tweaked my DNS configuration to add an SPF record, and tested it.
This ought to have been enough to achieve approximate parity with what Gandi had been providing me with. Not bad.
I sent a test email to a Gmail account, where I noticed two problems:
The first problem was that Postfix on Debian isn’t configured by-default to use opportunistic TLS when talking to other
mailservers. That’s a bit weird, but I’m sure there’s a good reason for it. The solution was to add smtp_tls_security_level = may to my
/etc/postfix/main.cf.
The second problem was that without a valid DKIM signature on them, about half of my test emails were going straight to the
spam folder. Again, it seems that since the last time I seriously ran a mailserver 20 years ago, this has become something that isn’t strictly required… but your emails aren’t
going to get through if you don’t.
I’ve put it off this long, but I think it’s finally time for me to learn some practical DKIM.
Understanding DKIM
What’s DKIM, then?
A server that wants to send email from a domain generates a cryptographic keypair.
The public part of the key is published using DNS. The private part is kept securely on the server.
When the server relays mail on behalf of a user, it uses the private key to sign the message body and a stated subset of the headers3,
and attaches the signature as an email header.
When a receiving server (or, I suppose, a client) receives mail, it can check the signature by acquiring the public key via DNS and validating the signature.
In this way, a recipient can be sure that an email received from a domain was sent with the authorisation of the owner of that domain. Properly-implemented, this is a strong mitigation
against email spoofing.
OpenDKIM
To set up my new server to sign outgoing mail, I installed OpenDKIM and its keypair generator using sudo apt install opendkim
opendkim-tools. It’s configuration file at /etc/opendkim.conf needed the following lines added to it:
# set up a socket for Postfix to connect to:
Socket inet:12301@localhost
# set up a file to specify which IPs/hosts can send through us without authentication and get their messages signed:
ExternalIgnoreList refile:/etc/opendkim/TrustedHosts
InternalHosts refile:/etc/opendkim/TrustedHosts
# set up a file to specify which selector/domain are used to each incoming email address:
SigningTable refile:/etc/opendkim/SigningTable
# set up a file to specify which signing key to use for each selector/domain:
KeyTable refile:/etc/opendkim/KeyTable
Into /etc/opendkim/TrustedHosts I put a list of local IPs/domains that would have their emails signed by this server. Mine looks like this (in this example I’m using
example.com as my domain name, and default as the selector for it: the selector can be anything you like, it only matters if you’ve got multiple
mailservers signing mail for the same domain). Note that 192.168.0.0/16 is the internal subnet on which my sending VM will
run.
/etc/opendkim/SigningTable maps email addresses (I’m using a wildcard) to the subdomain whose TXT record will hold the public key for the signature. This also goes on to
inform the KeyTable which private key to use:
*@example.com default._domainkey.example.com
And then /etc/opendkim/KeyTable says where to find the private key for that:
The public key needs publishing via DNS. Conveniently, when you create a keypair using its tools, OpenDKIM provides a sample (in
BIND-style) for you to copy-paste from or adapt: look in /etc/opendkim/keys/example.com/default.txt!
Once we’ve restarted both services (sudo service postfix restart; sudo service opendkim restart), we can test it!
So I learned something new today.
If you, too, love to spend your Saturday mornings learning something new, have a look at those subscription
options to decide how you’d like to hear about whatever I get up to next.
Footnotes
1 I still outsource my personal email, and I sing the praises of the excellent folks
behind ProtonMail.
2 My desktop email client also doubles as my newsreader, because, yes, of course
you can still find me on USENET. Which, by the way, is undergoing a mini-revival…
3 Why doesn’t DKIM sign
all the headers in an email? Because intermediary servers and email clients will probably add their own headers, thereby invalidating the signature! DKIM gets used to sign the From: header, for obvious reasons, and ought to be used for other headers whose tampering could be
significant such as the Date: and Subject:, but it’s really up to the signing server to choose a subset.