This last month or so, my digital life has been dramatically improved by Syncthing. So much so that I want to tell you about it.
I started using it last month. Basically, what it does is keeps a pair of directories on remote systems “in sync” with one another. So far, it’s like your favourite cloud storage service, albeit self-hosted and much-more customisable. But it’s got a handful of killer features that make it nothing short of a dream to work with:
The unique identifier for a computer can be derived from its public key. Encryption comes free as part of the verification of a computer’s identity.
You can share any number of folders with any number of other computers, point-to-point or via an intermediate proxy, and it “just works”.
It’s super transparent: you can always see what it’s up to, you can tweak the configuration to match your priorities, and it’s open source so you can look at the engine if you like.
Here are some of the ways I’m using it:
Keeping my phone camera synced to my PC
I’ve tried a lot of different solutions for this over the years. Back in the way-back-when, like everybody else in those dark times, I used to plug my phone in using a cable to copy pictures off and sort them. Since then, I’ve tried cloud solutions from Google, Amazon, and Flickr and never found any that really “worked” for me. Their web interfaces and apps tend to be equally terrible for organising or downloading files, and I’m rarely able to simply drag-and-drop images from them into a blog post like I can from Explorer/Finder/etc.
At first, I set this up as a one-way sync, “pushing” photos and videos from my phone to my desktop PC whenever I was on an unmetered WiFi network. But then I switched it to a two-way sync, enabling me to more-easily tidy up my phone of old photos too, by just dragging them from the folder that’s synced with my phone to my regular picture storage.
Centralising my backups
Now I’ve got a fancy NAS device with tonnes of storage, it makes sense to use it as a central point for backups to run fom. Instead of having many separate backup processes running on different computers, I can just have each of them sync to the NAS, and the NAS can back everything up. Computers don’t need to be “on” at a particular time because the NAS runs all the time, so backups can use the Internet connection when it’s quietest. And in the event of a hardware failure, there’s an up-to-date on-site backup in the first instance: the cloud backup’s only needed in the event of accidental data deletion (which could be sync’ed already, of course!). Plus, integrating the sync with ownCloud running on the NAS gives easy access to my files wherever in the world I am without having to fire up a VPN or otherwise remote-in to my house.
Plus: because Syncthing can share a folder between any number of devices, the same sharing mechanism that puts my phone’s photos onto my main desktop can simultaneously be pushing them to the NAS, providing redundant connections. And it was a doddle to set up.
Maintaining my media centre’s screensaver
Since the NAS, running Jellyfin, took on most of the media management jobs previously shared between desktop computers and the media centre computer, the household media centre’s had less to do. But one thing that it does, and that gets neglected, is showing a screensaver of family photos (when it’s not being used for anything else). Historically, we’ve maintained the photos in that collection via a shared network folder, but then you’ve got credential management and firewall issues to deal with, not to mention different file naming conventions by different people (and their devices).
But simply sharing the screensaver’s photo folder with the computer of anybody who wants to contribute photos means that it’s as easy as copying the picture to a particular place. It works on whatever device they care to (computer, tablet, mobile) on any operating system, and it’s quick and seamless. I’m just using it myself, for now, but I’ll be offering it to the rest of the family soon. It’s a trivial use-case, but once you’ve got it installed it just makes sense.
In short: this month, I’m in love with Syncthing. And maybe you should be, too.
Last week I built Fox, the newest addition to our home network. Fox, whose specification called for not one, not two, not three but four 12 terabyte hard disk drives was built principally as a souped-up NAS device – a central place for us all to safely hold and control access to important files rather than having them spread across our various devices – but she’s got a lot more going on that that, too.
Enough hard drive space to give us 36TB of storage capacity plus 12TB of parity, allowing any one of the drives to fail without losing any data.
“Headroom” sufficient to double its capacity in the future without significant effort.
A mediumweight graphics card to assist with real-time transcoding, helping her to convert and stream audio and videos to our devices in whatever format they prefer.
A beefy processor and sufficient RAM to run a dozen virtual machines supporting a variety of functions like software development, media ripping and cataloguing, photo rescaling, reverse-proxying, and document scanning (a planned future purpose for Fox is to have a network-enabled scanner near our “in-trays” so that we can digitise and OCR all of our post and paperwork into a searchable, accessible, space-saving collection).
The last time I filmed myself building a PC was when I built Cosmo, a couple of desktops ago. He turned out to be a bit of a nightmare: he was my first fully-watercooled computer and he leaked everywhere: by the time I’d done all the fixing and re-fixing to make him behave nicely, I wasn’t happy with the video footage and I never uploaded it. I’d been wary, almost-superstitious, about filming a build since then, but I shot a timelapse of Fox’s construction and it turned out pretty well: you can watch it below or on YouTube or QTube.
The timelapse slows to real-time, about a minute in, to illustrate a point about the component test I did with only a CPU (and cooler), PSU, and RAM attached. Something I routinely do when building computers but which I only recently discovered isn’t commonly practised is shown: that the easiest way to power on a computer without attaching a power switch is just to bridge the power switch pins using your screwdriver!
Fox is running Unraid, an operating system basically designed for exactly these kinds of purposes. I’ve been super-impressed by the ease-of-use and versatility of Unraid and I’d recommend it if you’ve got a similar NAS project in your future! I’d also like to sing the praises of the Fractal Design Node 804 case: it’s not got quite as many bells-and-whistles as some cases, but its dual-chamber design is spot-on for a multipurpose NAS, giving ample room for both full-sized expansion cards and heatsinks and lots of hard drives in a relatively compact space.
Until the 17th century, to “fathom” something was to embrace it. Nowadays, it’s more likely to refer to your understanding of something in depth. The migration came via the similarly-named imperial unit of measurement, which was originally defined as the span of a man’s outstretched arms, so you can understand how we got from one to the other. But you know what I can’t fathom? Broadband.
Broadband Internet access has become almost ubiquitous over the last decade and a half, but ask people to define “broadband” and they have a very specific idea about what it means. It’s not the technical definition, and this re-invention of the word can cause problems.
What people think it means
High-speed, always-on Internet access.
What it originally meant
Communications channel capable of multiple different traffic types simultaneously.
Throughout the 19th century, optical (semaphore) telegraph networks gave way to the new-fangled electrical telegraph, which not only worked regardless of the weather but resulted in significantly faster transmission. “Faster” here means two distinct things: latency – how long it takes a message to reach its destination, and bandwidth – how much information can be transmitted at once. If you’re having difficulty understanding the difference, consider this: a man on a horse might be faster than a telegraph if the size of the message is big enough because a backpack full of scrolls has greater bandwidth than a Morse code pedal, but the latency of an electrical wire beats land transport every time. Or as Andrew S. Tanenbaum famously put it: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
Telegraph companies were keen to be able to increase their bandwidth – that is, to get more messages on the wire – and this was achieved by multiplexing. The simplest approach, time-division multiplexing, involves messages (or parts of messages) “taking turns”, and doesn’t actually increase bandwidth at all: although it does improve the perception of speed by giving recipients the start of their messages early on. A variety of other multiplexing techniques were (and continue to be) explored, but the one that’s most-interesting to us right now was called acoustic telegraphy: today, we’d call it frequency-division multiplexing.
What if, asked folks-you’ll-have-heard-of like Thomas Edison and Alexander Graham Bell, we were to send telegraph messages down the line at different frequencies. Some beeps and bips would be high tones, and some would be low tones, and a machine at the receiving end could separate them out again (so long as you chose your frequencies carefully, to avoid harmonic distortion). As might be clear from the names I dropped earlier, this approach – sending sound down a telegraph wire – ultimately led to the invention of the telephone. Hurrah, I’m sure they all immediately called one another to say, our efforts to create a higher-bandwidth medium for telegrams has accidentally resulted in a lower-bandwidth (but more-convenient!) way for people to communicate. Job’s a good ‘un.
Most electronic communications systems that have ever existed have been narrowband: they’ve been capable of only a single kind of transmission at a time. Even if you’re multiplexing a dozen different frequencies to carry a dozen different telegraph messages at once, you’re still only transmitting telegraph messages. For the most part, that’s fine: we’re pretty clever and we can find workarounds when we need them. For example, when we started wanting to be able to send data to one another (because computers are cool now) over telephone wires (which are conveniently everywhere), we did so by teaching our computers to make sounds and understand one another’s sounds. If you’re old enough to have heard a fax machine call a landline or, better yet used a dial-up modem, you know what I’m talking about.
As the Internet became more and more critical to business and home life, and the limitations (of bandwidth and convenience) of dial-up access became increasingly questionable, a better solution was needed. Bringing broadband to Internet access was necessary, but the technologies involved weren’t revolutionary: they were just the result of the application of a little imagination.
We’d seen this kind of imagination before. Consider teletext, for example (for those of you too young to remember teletext, it was a standard for browsing pages of text and simple graphics using an 70s-90s analogue television), which is – strictly speaking – a broadband technology. Teletext works by embedding pages of digital data, encoded in an analogue stream, in the otherwise-“wasted” space in-between frames of broadcast video. When you told your television to show you a particular page, either by entering its three-digit number or by following one of four colour-coded hyperlinks, your television would wait until the page you were looking for came around again in the broadcast stream, decode it, and show it to you.
Teletext was, fundamentally, broadband. In addition to carrying television pictures and audio, the same radio wave was being used to transmit text: not pictures of text, but encoded characters. Analogue subtitles (which used basically the same technology): also broadband. Broadband doesn’t have to mean “Internet access”, and indeed for much of its history, it hasn’t.
Here in the UK, ISDN (from 1988!) and later ADSL would be the first widespread technologies to provide broadband data connections over the copper wires simultaneously used to carry telephone calls. ADSL does this in basically the same way as Edison and Bell’s acoustic telegraphy: a portion of the available frequencies (usually the first 4MHz) is reserved for telephone calls, followed by a no-mans-land band, followed by two frequency bands of different sizes (hence the asymmetry: the A in ADSL) for up- and downstream data. This, at last, allowed true “broadband Internet”.
But was it fast? Well, relative to dial-up, certainly… but the essential nature of broadband technologies is that they share the bandwidth with other services. A connection that doesn’t have to share will always have more bandwidth, all other things being equal! Leased lines, despite technically being a narrowband technology, necessarily outperform broadband connections having the same total bandwidth because they don’t have to share it with other services. And don’t forget that not all speed is created equal: satellite Internet access is a narrowband technology with excellent bandwidth… but sometimes-problematic latency issues!
Equating the word “broadband” with speed is based on a consumer-centric misunderstanding about what broadband is, because it’s necessarily true that if your home “broadband” weren’t configured to be able to support old-fashioned telephone calls, it’d be (a) (slightly) faster, and (b) not-broadband.
But does the word that people use to refer to their high-speed Internet connection matter. More than you’d think: various countries around the world have begun to make legal definitions of the word “broadband” based not on the technical meaning but on the populist one, and it’s becoming a source of friction. In the USA, the FCC variously defines broadband as having a minimum download speed of 10Mbps or 25Mbps, among other characteristics (they seem to use the former when protecting consumer rights and the latter when reporting on penetration, and you can read into that what you will). In the UK, Ofcom‘s regulations differentiate between “decent” (yes, that’s really the word they use) and “superfast” broadband at 10Mbps and 24Mbps download speeds, respectively, while the Scottish and Welsh governments as well as the EU say it must be 30Mbps to be “superfast broadband”.
I’m all in favour of regulation that protects consumers and makes it easier for them to compare products. It’s a little messy that definitions vary so widely on what different speeds mean, but that’s not the biggest problem. I don’t even mind that these agencies have all given themselves very little breathing room for the future: where do you go after “superfast”? Ultrafast (actually, that’s exactly where we go)? Megafast? Ludicrous speed?
What I mind is the redefining of a useful term to differentiate whether a connection is shared with other services or not to be tied to a completely independent characteristic of that connection. It’d have been simple for the FCC, for example, to have defined e.g. “full-speed broadband” as providing a particular bandwidth.
Verdict: It’s not a big deal; I should just chill out. I’m probably going to have to throw in the towel anyway on this one and join the masses in calling all high-speed Internet connections “broadband” and not using that word for all slower and non-Internet connections, regardless of how they’re set up.
In the spring of that year, my travels brought me upon a previously undiscovered civilization. The people called themselves Ossians, and they lived in an isolated collection of villages in a remote part of South America.
Being remote as they were, their level of technology was understandably primitive. But I was surprised by the locals’ recent obsession with new forms of communication. It all started, they told me, when one of them discovered that by attaching a rope between two clay pots and stretching the rope taut, a voice uttered into one side could be heard on the other. (I neglected to tell them that even as a boy I had done this very thing with tin cans.)…
The other week I built Tiffany2, New Earth‘s new media centre computer. She’s well-established and being used to watch movies, surf the web, and whatnot, now, so I thought I’d better fulfil my promise of telling you about my other new smaller-than-average computer, Dana, whose existence was made possible by gifts from my family over Christmas and my birthday.
Dana‘s size and power-consumption is so small that it makes Tiffany2 look like a bloated monster. That’s because Dana is a DreamPlug, an open-architecture plug computer following in the footsteps of the coveted SheevaPlug and GuruPlug.
The entire computer including its detachable power supply is only a little larger than the mobile telephones of the mid-nineties, and the entire device can be plugged straight into the wall. With no hard disk (it uses SD cards) and no fans, the DreamPlug has no moving parts to wear out or make noise, and so it’s completely silent. It’s also incredibly low-power – mine idles at about 4 watts – that’s about the same as a radio alarm clock, and about a hundredth of what my desktop PCs Toni and Nena run at under a typical load.
I’ve fitted up mine with a Mimo Mini-Monster 10″: a dinky little self-powered USB-driven touchscreen monitor about the size of an iPad. Right now the whole assembly – about the size of a large picture frame – sits neatly in the corner of my desk and (thanks to the magic of Synergy) forms part of my extended multi-monitor desktop, as well as acting as a computer in her own right.
So on the surface, she’s a little bit like a wired tablet computer, which would seem a little silly (and indeed: at a glance you’d mistake her for a digital photo frame)! But because she’s a “real” computer underneath, with a 1.2GHz processor, 512MB RAM, USB, WiFi, and two Ethernet ports, there’s all kinds of fun things that can be done with her.
For a start, she provides an ultra low-power extension to my existing office development environment. I’ve experimented with “pushing” a few tasks over to her, like watching log file output, downloading torrents, running a web server, reading RSS feeds, and so on, but my favourite of her tasks is acting as a gateway between the rest of the world and my office.
While they’ve come a long way, modern ADSL routers are still woefully inadequate at providing genuine customisability and control over my home network. But a computer like this – small, silent, and cheap – makes it possible to use your favourite open-source tools (iptables, squid, sshd, etc.) as a firewall to segregate off a part of the network. And that’s exactly what I’ve done. My office – the pile of computers in the upper-right of the diagram, above – is regulated by Dana, whose low footprint means that I don’t feel bad about leaving her turned always-on.
That means that, from anywhere in the world (and even from my phone), I can now:
Connect into Dana using SSH.
Send magic packets to Toni, Nena, or Tiffany2 (all of which are on wired connections), causing them to turn themselves on.
Remotely control those computers to, for example, get access to my files from anywhere, set them off downloading something I’ll need later, or whatever else.
Turn them off when I’m done.
That’s kinda sexy. There’s nothing new about it – the technologies and standards involved are as old as the hills – but it’s nice to be able to do it using something that’s barely bigger than a postcard.
I have all kinds of ideas for future projects with Dana. It’s a bit like having a souped-up (and only a little bigger) Arduino to play with, and it’s brimming with potential. How about a webcam for my bird feeder? Or home-automation tools (y’know: so I can turn on my bedroom light without having to get out of bed)? Or a media and file server (if I attached a nice, large, external hard disk)? And then there’s the more far-fetched ideas: it’s easily low-power enough to run from a car battery – how about in-car entertainment? Or home-grown GPS guidance? What about a “delivered ready-to-use” intranet application, as I was discussing the other day with a colleague, that can be simply posted to a client, plugged in, and used? There’s all kinds of fun potential ideas for a box like this, and I’m just beginning to dig into them.
This weekend, I integrated two new computers into the home network on New Earth. The first of these is Tiffany2.
Tiffany2 replaces Tiffany, the media centre computer I built a little under four years ago. The original Tiffany was built on a shoestring budget of under £300, and provided the technical magic behind the last hundred or so Troma Nights, as well as countless other film and television nights, a means to watch (and record and pause) live TV, surf the web, and play a game once in a while.
The problem with Tiffany is that she was built dirt-cheap at a time when building a proper media centre PC was still quite expensive. So she wasn’t very good. Honestly, I’m amazed that she lasted as long as she did. And she’s still running: but she “feels” slow (and takes far too long to warm up) and she makes a noise like a jet engine… which isn’t what you want when you’re paying attention to the important dialogue of a quiet scene.
Tiffany2 is virtually silent and significantly more-powerful than her predecessor. She’s also a lot smaller – not much bigger than a DVD player – and generally more feature-rich.
This was the first time I’d built an ITX form-factor computer (Tiffany2 is Mini-ITX): I wanted to make her small, and it seemed like the best standard for the job. Assembling some of her components felt a little like playing with a doll’s house – she has a 2.5″ hard disk and a “slimline” optical drive: components that in the old days we used to call “laptop” parts, which see new life in small desktop computers.
In order to screw in some of the smaller components, I had to dig out my set of watchmaker’s screwdrivers. Everything packs very neatly into a very small space, and – building her – I found myself remembering my summer job long ago at DesignPlan Lighting, where I’d have to tuck dozens of little components, carefully wired-together, into the shell of what would eventually become a striplight in a tube train or a prison, or something.
She’s already deployed in our living room, and we’ve christened her with the latest Zero Punctuation, a few DVDs, some episodes of Xena: Warrior Princess, and an episode of Total Wipeout featuring JTA‘s old history teacher as a contestant. Looks like she’s made herself at home.
(for those who are sad enough to care, Tiffany2 is running an Intel Core i3-2100 processor, underclocked to 3GHz, on an mITX Gigabyte GA-H61N-USB3 motherboard with 4GB RAM, a 750GB hard disk, and DVD-rewriter, all wrapped up in an Antec ISK 300-150 case with a 150W power supply: easily enough for a media centre box plus some heavy lifting if I ever feel the need to give her any)
I keep getting caught up on small world coincidences, since I started working at the Bodleian Library last week. I know about selective biases, of course, and I’ve always said that coincidences happen nine times out of ten, but this is really starting to feel like some kind of amazing conspiracy that I’ve somehow wandered into.
The most recent chain of connected coincidences is also probably the most impressive. But to explain it, I’ll need to take you back in time by almost three years. Back in the summer of 2008, I went to BiCon for the second time, accompanied by Claire and Matt P. Among the various other things we got up to, we met a young lady called Ann (who, if I remember rightly, got along very well with Matt).
This morning I received an email from Ann. It turns out that she works in the Bodleian Libraries: she’s likely to be one of the very users who it’s now my job to provide training and technical support to! She saw my photograph in the newsletter I mentioned in my last blog post and looked me up: small world! I emailed back, suggesting that we get together for a drink after work, and she agreed: great! She also asked if she could bring a friend along, a colleague from the library. Sure, I said, sounds good.
This lunchtime I sorted out some of my holiday entitlement for the rest of this academic year. I booked off a few days for a Three Rings “code week” in the summer, and a couple of days around the time that I’ll be moving house next month. One of these days clashed with a meeting that I’d had planned with the Web/Digital Officer of one of the libraries (I’m doing a grand tour of many of the libraries that comprise the Bodleian, in order to meet all the relevant people), so I sent an email to this staff member to ask if we could reschedule our meeting to another time.
“Okay,” they said, “But I think I’m meeting you in the pub in 90 minutes anyway…”
It turns out that the person whose meeting I’ve asked to reschedule is the friend of the person who recognised me from the staff newsletter, having originally met me three years ago. Out of all of the people (I’m not sure how many exactly – it’s probably in the staff handbook I haven’t read yet – but I’ll bet it’s a lot) that are employed by this, the largest university library in the UK, what are the odds?
[this post has been partially damaged during a server failure on 11 July 2004; with the exception of the images, it was recovered on 13 October 2018]
Paul has been stuck with a problem of late – he’s now living in university accomodation, and he’s found that he can’t connect through the university firewall to his external mail server. I advised him that it’s possible to set up an ‘SSH Tunnel’ (through central.aber.ac.uk) to fix this problem, but he hasn’t met with much success (see his blog entry for more details). In any case, here’s my investigation (and solution) to the problem.
How To Use SSH Tunnelling To Allow Services To Pass Through A Firewall
In my example, I’m going to try the opposite to what Paul is trying to achieve. I’m going to try to allow my POP3 e-mail client to get access to the university e-mail server (pophost.aber.ac.uk). As things stand, this server is on the other side of the university firewall, and is inaccessible from outside. The server central.aber.ac.uk, however, is accessible from both sides of the firewall. So what I’ve got is this (yes, I know that this is a gross oversimplification):
As you can see, connecting from my home PC is futile:
C:\Documents and Settings\Dan>telnet pophost.aber.ac.uk 110
Connecting To pophost.aber.ac.uk...Could not open connection to the host, on por
t 110: Connect failed
But if I SSH-in to central.aber.ac.uk…
central:~ $ telnet pophost.aber.ac.uk 110
Connected to pophost.aber.ac.uk.
Escape character is '^]'.
+OK mailsplit Oct 2000 ready
So, what I need to do is to tell my SSH client to connect to central.aber.ac.uk, and forward specific traffic through the firewall to the mail server. Here’s what I needed to know:
(a) A free TCP port number on my own computer from which I can virtually ‘pipe’ the connection. Most numbers over 1024 are fine. I chose ‘9110’. (b) The name of the mail server – ‘pophost.aber.ac.uk’. (c) The TCP port to which I wanted to connect – the standard port for a POP3 mail server is ‘110’. (d) My user name on a server which: (1) I can connect to; (2) can connect to the server specified in (b). It happens to be ‘dlh9’. (e) The name of the server specified in (d) (i.e. ‘central.aber.ac.uk’). (f) My password on the server. Like I’m going to tell you that.