The boring operations post

Homelab content has a survivorship bias problem. The posts are about building things, migrating things, and heroically debugging things at two in the morning. Nobody writes about the weeks where nothing happened, because there is nothing to write.

But the weeks where nothing happens are the product. Excitement, in operations, is another word for failure. This post is about the unglamorous machinery that makes most weeks boring, which I have come to believe is the actual skill in self-hosting, and the least discussed.

§ The rule: twice by hand means never again by hand

Everything in that diagram earned its place the same way: it got done by hand exactly twice, once to learn it and once to resent it, and then it became a schedule. Nothing that matters here has a third manual run.

The rhythm today: every guest in the cluster is backed up nightly at half past two. A snapshot lands at three, pinning the fresh set. Hourly snapshots tick along all day on the data that matters. The pool gets scrubbed every Sunday at midnight, re-reading every byte and demanding zero errors. And this website checks for new commits every five minutes, all day, forever. The one red square in the diagram is the entire human contribution to a normal week: a git push. Everything downstream of it is machinery.

None of this is impressive. All of it is load-bearing. The test for whether something belongs on a schedule is embarrassingly simple: if it matters, it cannot depend on me remembering it, because I am the least reliable component in the building.

§ This website is the worked example

Publishing a post here is one git commit. The web container fetches on its timer, resets hard to whatever the repository says, and rebuilds the blog from markdown. There is no deploy button, no CI pipeline, nothing on the box beyond a read-only fetch key, and no step where I can forget something.

The design principle hiding in that: the server is a replica, not a pet. It holds no state worth preserving, so it can reset itself brutally and be correct by construction. Anything I edit directly on the container gets discarded within five minutes, which sounds like a bug and is actually the discipline: the repository is the truth, and the machine conforms to it.

That one property has quietly absorbed entire categories of operational work. There is no "what changed on the server" mystery, ever, because the answer is always: whatever is in git.

§ Documentation is an operational tool, not homework

There is a git repository describing the lab: what each machine is, how the network is laid out, what talks to what, and, crucially, what state everything was in when last verified, with dates.

Not automation. Not infrastructure-as-code. Prose and tables, updated when reality changes. I resisted this for years because it felt like homework, and I was wrong. The failure mode of a homelab is not usually a crash; it is the slow rot of the builder forgetting why. Six months later, a firewall rule or a strange mount option is a mystery, and the person who could explain it was me, and I am gone.

Two habits make it work. Findings get written down when they are confirmed, not batched for later, because later never comes. And the documentation records when something was last verified against reality, because a fact without a date quietly becomes a rumour. Twice now, a written-down "verified on this date" has settled a debugging session in minutes by proving the problem was newer than the config.

§ One war story: taming the fans

The NAS has an out-of-band management controller, the little always-on computer inside the server that works even when the server does not. Its firmware had one opinion about the fans: fast. All of them, always, at a volume unsuitable for a home.

The fix was community firmware that exposes actual fan control, and then, the part worth writing down, choosing the curve with measurements instead of vibes. The tempting mistake is to tune fans on an idle system, decide it is fine, and get surprised later. Disks heat up under sustained load, and the worst realistic case here is real: a full offsite sync reading the pool for hours while a weekly scrub does the same. So the cap was chosen by measuring disk temperatures during exactly that overlap, not during a quiet afternoon.

The general lesson outlives the fans: tune for the worst load you will actually experience, and go create that load on purpose rather than waiting to meet it by accident.

§ What breaks silently

The failures that cost me evenings were never loud. A tunnel client caching a DNS answer long after the address changed, so health checks failed while every backend answered perfectly. A DHCP lease quietly shifting under a service that someone, meaning me, had pointed at by IP. Firewall rules and port groups outliving the applications they were written for, confusing every later audit. A "working" page that was actually broken for everyone but me, because my browser carried a session nobody else had.

The common thread: each one looked healthy from where I was standing. The countermeasures are all boring, and they are the same four every time: give things names instead of addresses, restart resolvers after renumbering, test as a stranger in a private window, and delete configuration the moment its service dies rather than leaving it as an exhibit.

§ The 98%

The lab runs at roughly ninety-eight percent boring. Backups happen, snapshots rotate, the scrub reports zero, the site publishes itself, and the documentation remembers so I do not have to. The remaining two percent is the fun part, the building and breaking that the whole hobby is ostensibly about.

The ratio is the achievement. The two percent is only fun because the other ninety-eight is machinery, humming along whether anyone is watching or not.