jani@raatti:~ $ cat ~/blog/relrotate-logrotate-but-for-releases.md
---
title:

"relrotate: logrotate, but for releases"

date: 2026-11-13
author: jani
categories: [Linux, DevOps, Server]
reading_time: 9 min
cover: "relrotate-conveyor.png"
---
A night-time server room workbench: worn cardboard boxes ride a conveyor into a recycling bin while fresh boxes come out of a glowing server rack and a robot arm checks each one with a magnifying glass. A cat sleeps next to the monitors.

Your package manager is good at one thing. It keeps everything it installed patched, signed and boring. On a RHEL box dnf-automatic runs every few hours and I never think about it.

Then there is everything else. The Go binary in /usr/local/bin because upstream ships no RPM. The PHP app unpacked under /var/www because the distro package is two years behind. The container pinned to a version tag in a podman quadlet because latest is how you get surprised on a Sunday. None of that is in the package database, so none of it gets updated unless somebody remembers.

That somebody was me, and I had started remembering in shell scripts.

Five scripts walk into a server

Five identical grumpy robot technicians stand in a row in a server room, each holding a different tool: a wrench, a screwdriver, a clipboard, a flashlight and an oil can.

It started with one updater, for Roundcube. A daily cron job asked the GitHub API for the newest release, downloaded it, unpacked it next to the old one and flipped a symlink. One day upstream pushed an emergency release and the script installed it unattended. It worked, which surprised me more than it should have.

Then I looked closer. The script saved “this version is done” even when the update failed, so a broken release would have been skipped forever without a word. It had no idea what a major version was and would have happily jumped branches. It verified nothing.

That script was rewritten many times after that. One of the later rewrites, written by an AI agent, finally ran Roundcube’s own bin/update.sh against the new release before the symlink flipped, so the database schema moves together with the code. Then came the next updater, and the next. Forgejo got one. FreshRSS got one. Element Web got one. Headscale got one. Each was written for one app, and each one picked up the lessons of the one before: checksum verification, a symlinked release layout, a mail when something needs a human, and a systemd timer that takes patch releases on its own and mails about anything bigger.

By the fifth script the pattern was obvious. Same steps, different nouns. Five copies of the same idea is technical debt with a cron schedule.

What relrotate does

A server tower in the middle of a rotating ring of green puzzle pieces; one worn grey piece is being swapped for a bright new one while the ring keeps turning.

relrotate is that pattern, written once. One script, a few library files, one small recipe per app, and a systemd timer per app. Every run does the same thing:

  1. Ask the release API for recent releases. GitHub and Gitea/Forgejo both work, so Codeberg is covered too.
  2. Ignore drafts, pre-releases and tags that are not plain version numbers.
  3. Pick the newest version the policy allows. Anything newer than that gets one mail, not an install.
  4. Download the asset and verify it.
  5. Stage the new version next to the old one.
  6. Run the recipe’s hooks, then flip to the new version in one atomic step.
  7. Restart the service and run a health check.
  8. Healthy? Record the version, prune old releases, send a mail. Unhealthy? Flip back, restart, check again and send a different mail.

The version is recorded only after a successful health check. A failed update gets retried on the next run instead of being remembered as done. Getting that one rule wrong was the first thing I found in the Roundcube script.

A recipe

Recipes are plain shell. relrotate sources them, so there is no YAML to learn and nothing to parse:

# /etc/relrotate/apps.d/headscale.conf
SOURCE=github:juanfont/headscale
ASSET='headscale_${VERSION}_linux_${ARCH}'
VERIFY="api-digest checksums-file"
CHECKSUM_ASSET=checksums.txt
POLICY="patch"

LAYOUT=binary
TARGET=/usr/local/bin/headscale
SERVICE=headscale.service

pre_update() {
    sqlite3 /var/lib/headscale/db.sqlite \
        ".backup '/var/lib/headscale/backups/db-$OLD_VERSION.sqlite'"
}

VERIFY here means two independent checks have to agree: the SHA-256 digest GitHub publishes for every release asset, and upstream’s own checksums.txt. Other recipes use a .sha256 file next to the download or a GPG signature checked with gpgv. Opting out is possible, but you have to write VERIFY=none and relrotate will complain about it in the journal on every single run. That is on purpose.

Turning it on is the usual systemd ritual:

sudo relrotate check headscale
sudo relrotate update --dry-run headscale
sudo systemctl enable --now relrotate@headscale.timer

Three ways software lives on a server

Three museum display cases: a row of linked folders, a single polished metal cylinder, and a stack of shipping containers, standing for the symlink, binary and container layouts.

Almost everything I run outside the package manager falls into one of three shapes, so relrotate has three layouts.

Symlink. Web apps. Each version gets its own directory and the live path is a symlink. For Element Web, /var/www/element points at /var/www/element-releases/1.12.35. Config files get copied into each new release, data directories get linked in, and rolling back means pointing the symlink at the previous directory.

Binary. A single executable such as Headscale, Forgejo or restic. relrotate keeps old binaries in its state directory and swaps the live one with a rename. On the very first run it saves a copy of whatever binary you installed by hand, so even the first update can be rolled back.

Container. A podman quadlet with a pinned image tag. relrotate pulls the new image first, rewrites the Image= line, reloads systemd and restarts the unit. Rootless quadlets work too.

Some apps are several containers that must move together. Immich runs a server and a machine-learning container, and they have to be on the same version. A recipe can list both. relrotate pulls every image before touching any quadlet, so a failed pull changes nothing, and if the health check fails both containers roll back together.

Which updates go in on their own

The policy has three settings. patch stays within the same minor version, minor stays within the same major, major takes anything.

My own rule is that patch releases are treated like security fixes and go in automatically, and anything bigger waits for me to read the release notes. Roundcube taught me that one, since nearly every Roundcube release is a security fix. Waiting until I have time is how security fixes end up waiting for weeks.

Upstream projects do not always play along with version numbers, so some details took more care than expected:

  • Below 1.0, semver says a minor bump may break things. relrotate follows that, so for a 0.x project minor means patches only. restic 0.19 to 0.20 arrives as a mail.
  • Jellyfin uses two-part versions like 12.4. relrotate reads that as 12.4.0 and moves on.
  • Home Assistant uses dates. 2026.11.1 works fine as long as you read it as year, month, patch. patch then means fixes to this month’s release. The monthly releases are where the breaking changes live, so those arrive as a mail.
  • GitHub tags and image tags disagree. Paperless-ngx tags v3.4.2 on GitHub and publishes the image as 3.4.2. LLDAP keeps the v in both. Recipes pick ${TAG} or ${VERSION} to match.

Small things that bit

A few details deserve a mention, because each one would have bitten on a real server.

Mail from a hardened unit. I like NoNewPrivileges=yes in service units. Postfix’s postdrop is setgid, so with that setting the notification mail cannot be queued, postdrop retries forever, and the unit never finishes. relrotate’s unit leaves that setting out and says why in a comment, so nobody “fixes” it later.

set -e has holes. Bash suspends set -e inside a function called from an || chain. A flip function whose mv failed would still return success, because its last line was a log call. Every step that changes something now fails explicitly, and there is a test that makes the flip fail on purpose and checks nothing changed.

Bot shields. Some forges sit behind Anubis to keep AI scrapers out. It challenges browser-like clients and lets plain API clients through. relrotate identifies itself as relrotate/<version>, which keeps it on the right side of that line.

Why bash

There is prior art. distronomicon does atomic GitHub release updates with systemd timers, and it is a nice piece of work. It is also Rust, and the features I care about most, like version policies and signature checks, were still on its roadmap.

I wanted something a sysadmin can read at three in the morning without learning a new language first. relrotate needs bash, curl, jq and coreutils, all in the base repositories of every mainstream distro. It runs on RHEL and its clones, and on Debian and Ubuntu. On SELinux systems it runs restorecon on every file it puts down.

Every event is one key=value line in the journal. Loki parses that with | logfmt, so “alert me when an update fails” is one query.

It comes with a test suite that fakes releases with local files and stubs out systemctl and podman, so it runs anywhere without root. There are example recipes for twenty apps, Hugo, rclone, Vaultwarden, Home Assistant and Immich among them. Each one has been checked against the project’s real release files and image tags.

Who wrote it

AI coding agents wrote a lot of the code, in the five scripts and in relrotate. What they could not do from a chat window was sit on the server when an update broke. Roundcube was the clearest case. Some of those problems took plain manual sysadmin debugging before anyone knew what the script should do differently. The scripts grew out of that: my experience and refinements, plus a lot of agent-written bash. relrotate is those refinements written down once, with tests this time.

What it is not

relrotate is not a package manager. It does not resolve dependencies, it does not install anything your distro already packages, and it should not. If dnf or apt can give you a maintained package, use that. relrotate is for the stuff that falls through the gap.

That gap is also why I built it. Once a security patch is out, you have a window of a few hours before mass exploitation starts. The package manager already watches the packaged half of a server. relrotate watches the software outside it, so I don’t have to.

My five scripts are still running. Each one gets switched off after relrotate has handled a real update of that app, and not before.

The code is at github.com/raatti/relrotate. The example recipes live in examples/apps.d, one file per app, and they are the easiest place to start. If your favourite app is missing, or one of my recipes is wrong for your setup, pull requests are welcome. A recipe is a dozen lines of shell, so the bar is pleasantly low.

jani@raatti:~ $ git commit # leave a comment