Backups & restore

hopboxd's state is small, but one piece of it is irreplaceable: the SSH host key is the front door's identity. Lose it and every user who has ever connected gets a host-key mismatch — there is no "regenerate" that gets you back to the key they already trust. The SQLite databases (box ownership, accounts, share links) are the other half — recoverable in principle from nothing, but only by rebuilding every box, account, and secret by hand.

hopbox-backup — installed by provision.sh — snapshots both, proves the snapshot is intact, and ships it off-host on a daily timer. This page is the setup, what's in an archive, and — the part that actually matters — how to restore.

The tool travels with the release: every hopbox-host update installs the hopbox-backup that the deployed release shipped with (it rides inside the bundle), and a rollback restores the previous one. So it stays in step with the daemon even on a host with no source checkout.

Set it up

provision.sh installs the tool, the daily timer, and a template config, but can't pick a destination for you. Point it at off-host storage:

$EDITOR /etc/hopbox/backup.env       # HOPBOX_BACKUP_DEST=<rclone remote | off-host path>
hopbox-backup run                    # take one now
hopbox-backup verify                 # fetch it back and prove it restores

Until HOPBOX_BACKUP_DEST is set, every hopbox-backup command exits 78 with an explicit error — the timer fails loudly instead of silently backing up nothing, and provision.sh prints a warning at the end of the install until it's configured.

The daily timer

provision.sh installs hopbox-backup.timer (OnCalendar=daily, a randomized delay so a fleet of hosts doesn't all hit the destination at once) driving hopbox-backup.service — a Type=oneshot unit with UMask=0077 that runs hopbox-backup run:

systemctl list-timers hopbox-backup.timer     # next run
systemctl status hopbox-backup.service        # last run's result
journalctl -u hopbox-backup.service           # its output

The backup runs against the live daemon — no downtime, no maintenance window. It reads the DBs while hopboxd keeps writing to them (see how consistency is guaranteed).

What's in an archive

Each run produces hopbox-backup-<name>-<UTC-timestamp>.tar.gz (<name> defaults to hostname -s) plus a .sha256 sidecar, both mode 0600 — the archive holds a private key:

File	What
hopboxd.db	Box ownership, state, durable-home mapping.
hopboxd-accounts.db	Accounts, verification, quotas.
hopboxd-shares.db	Share links.
ssh-host-key (+ .pub)	The front door's identity — the one thing that can't be recreated.
hopboxd.yaml	The daemon config (optional — HOPBOX_BACKUP_INCLUDE_CONFIG, default on). May contain secrets.
MANIFEST	Per-file sha256 + size + integrity_check result, the host-key fingerprint, and what is deliberately absent.

The three DB names are derived from the daemon's --db path exactly the way hopboxd derives them (<db>-accounts.db, <db>-shares.db) — see --db.

Consistency

A raw file copy of a live SQLite database can capture a half-written page. Instead each DB is snapshotted with SQLite's VACUUM INTO — an online, consistent copy — under a PRAGMA busy_timeout (default 30s) with up to 3 retries if the daemon holds the write lock at that instant, falling back to the .backup API on SQLite older than 3.27. A snapshot only counts as a backup once it has proved itself: PRAGMA integrity_check must return ok, and its sqlite_master object count must match the source — a truncated copy can still pass integrity_check alone.

This is also why hopboxd's own stores now open with busy_timeout(5000): the backup's read and a daemon write can briefly collide on these rollback-journal databases, and without a busy timeout that collision used to fail the daemon's write with SQLITE_BUSY. Now it just waits a few milliseconds. A backup running should never be something you can observe as a write failure.

Destination

HOPBOX_BACKUP_DEST has no default and is required. Two kinds of value work:

• An rclone remote — remote:path, any rclone backend (S3, MinIO, B2, R2, Wasabi, SFTP, WebDAV, …):

  rclone config create s3backup s3 provider=Other \
    endpoint=https://<offsite-s3-host> access_key_id=… secret_access_key=…
  HOPBOX_BACKUP_DEST=s3backup:hopbox-backups/ks

  The archive contains a private key, so layer an rclone crypt remote over it for at-rest encryption at the destination:

  rclone config create s3crypt crypt remote=s3backup:hopbox-backups/ks password=…
  HOPBOX_BACKUP_DEST=s3crypt:

• A filesystem path on off-host storage (an NFS or sshfs mount) — no rclone needed, just a path.

hopbox-backup warns on the classic mistake: a "backup" that dies with the host. It flags an rclone remote whose config points at 127.0.0.1/localhost, and a filesystem path that resolves to the same device as the live state (df comparison). The host's own MinIO, or another directory on the same disk, is not a backup — RAID is not a backup either; it covers a dead disk and nothing else, not an rm, not a bad migration.

Retention

Setting	Default
HOPBOX_BACKUP_KEEP_DAYS	30	Prune archives at the destination older than this.
HOPBOX_BACKUP_KEEP_MIN	7	…but never below this many, however old — an outage at the destination can't expire the last copy.
HOPBOX_BACKUP_KEEP_LOCAL	3	Local staging copies kept in /var/backups/hopbox (mode 0700) — a convenience, not the backup itself.

Config reference

/etc/hopbox/backup.env (written by provision.sh from deploy/backup.example.env, mode 0600). Every key:

Key	Default
HOPBOX_BACKUP_DEST	(none — required)	Off-host destination: an rclone remote:path or a filesystem path. See destination.
HOPBOX_BACKUP_NAME	`hostname -s`	Name tag in the archive filename.
HOPBOX_BACKUP_RCLONE_CONFIG	(rclone's own config)	Path to an rclone config file, if not the default.
HOPBOX_BACKUP_KEEP_DAYS	30	Prune archives older than this at the destination.
HOPBOX_BACKUP_KEEP_MIN	7	Never prune below this many archives.
HOPBOX_BACKUP_KEEP_LOCAL	3	Local staging copies kept.
HOPBOX_BACKUP_LOCAL_DIR	/var/backups/hopbox	Staging + restore scratch directory (mode 0700).
HOPBOX_BACKUP_INCLUDE_CONFIG	true	Include hopboxd.yaml in the archive (it may hold secrets).
HOPBOX_BACKUP_SERVICE	hopboxd	systemd unit that restore --apply stops/starts. Empty = no service manager involved; you stop/start the daemon yourself.
HOPBOX_BACKUP_BUSY_MS	30000	How long a snapshot waits for a busy DB before retrying.
HOPBOX_BACKUP_TRIES	3	Snapshot attempts before run fails.
HOPBOX_BACKUP_CONFIG	/etc/hopbox/hopboxd.yaml	Override where db: / host-key: are read from.
HOPBOX_BACKUP_DB	(read from config)	Override the box database path directly.
HOPBOX_BACKUP_HOST_KEY	(read from config)	Override the SSH host key path directly.
HOPBOX_BACKUP_ENV	/etc/hopbox/backup.env	Which config file hopbox-backup itself sources.
HOPBOX_BACKUP_ALLOW_RUNNING	(unset)	One-off guard for restore --apply — see restoring.

Paths are normally discovered automatically from /etc/hopbox/hopboxd.yaml's db: and host-key: keys (or /etc/hopbox/hopboxd.env) — the same config file the daemon itself reads. Only set HOPBOX_BACKUP_CONFIG / HOPBOX_BACKUP_DB / HOPBOX_BACKUP_HOST_KEY if your install keeps them somewhere unusual.

Commands

Command	Does
hopbox-backup run	Snapshot + verify + ship + prune. What the timer runs. Default with no subcommand.
hopbox-backup list	Archives at the destination, oldest → newest.
hopbox-backup verify [latest|ARCHIVE]	Fetch an archive and prove it restores — see rehearsing a restore.
hopbox-backup restore [latest|ARCHIVE] [--into DIR] [--apply]	Fetch, verify, and (with --apply) install it over the live state. See restoring below.

Restoring

Without --apply, restore only fetches and verifies — it changes nothing on the host, and prints the by-hand commands to finish the job:

hopbox-backup restore latest
#   checksum: ok
#   hopboxd.db: integrity_check=ok
#   hopboxd-accounts.db: integrity_check=ok
#   hopboxd-shares.db: integrity_check=ok
#   ssh-host-key: SHA256:…
# staged and verified: /var/backups/hopbox/restore-20260801T…/hopbox-backup-ks-…
#
# Nothing has been changed on this host. To install this snapshot over the live state:
#
#   hopbox-backup restore hopbox-backup-ks-… --apply

The guided path

hopbox-backup restore latest --apply

This:

1. Stops hopboxd (HOPBOX_BACKUP_SERVICE, default hopboxd).
2. Copies the current live DBs and host key aside to /var/backups/hopbox/pre-restore-<timestamp>/ — nothing is deleted, so a bad restore is itself recoverable.
3. Installs the archive's DBs and host key at mode 0600, and removes any stale -wal/-shm files next to them (a leftover WAL from the old database must never be replayed onto the restored one).
4. Starts hopboxd again and prints the restored front-door fingerprint.

The by-hand path

Equivalent to the guided path, if you'd rather drive it yourself (or HOPBOX_BACKUP_SERVICE is empty):

systemctl stop hopboxd
install -m600 <dir>/hopboxd.db          /var/lib/hopbox/hopboxd.db
install -m600 <dir>/hopboxd-accounts.db /var/lib/hopbox/hopboxd-accounts.db
install -m600 <dir>/hopboxd-shares.db   /var/lib/hopbox/hopboxd-shares.db
rm -f /var/lib/hopbox/hopboxd*.db-wal /var/lib/hopbox/hopboxd*.db-shm
install -m600 <dir>/ssh-host-key        /var/lib/hopbox/hopboxd-ssh-host-key
systemctl start hopboxd

(hopbox-backup restore <archive> without --apply prints these commands with your host's actual paths already filled in.)

What connected users experience

Stopping the daemon drops every open SSH session — clients see the connection close and reconnect once hopboxd is back. Because the host key comes back with the state, there is no host-key warning on reconnect: as far as a client's known_hosts is concerned, nothing changed. Boxes created (or state changed) after the snapshot was taken are not in it — a restore rolls the fleet back to that point in time, it doesn't merge.

The running-daemon guard

restore --apply refuses to run while a process named hopboxd is alive — writing under a live daemon would corrupt the DBs it has open. If that process is provably a different daemon than the one that owns these files (a second install on the same host, say), override the check for that one invocation:

HOPBOX_BACKUP_ALLOW_RUNNING=1 hopbox-backup restore latest --apply

Rehearse restores

A backup nobody has restored from is a hypothesis. hopbox-backup verify proves one is real without touching the host: it fetches the archive, checks the .sha256 sidecar, runs PRAGMA integrity_check on every DB inside it, confirms the host key parses, prints the MANIFEST, and throws the fetched copy away.

hopbox-backup verify           # the newest archive
hopbox-backup verify <name>    # a specific one, e.g. before trusting an old snapshot

Run this on a schedule (a second timer, or alongside your other health checks) — it's the only way to know the backup you're depending on would actually restore, rather than assuming it because run didn't error.

What is not backed up

The shared workspace (/wrk) is not covered

The shared workspace is not in these archives. Its data lives in the MinIO hopbox-wrk bucket and its metadata in Postgres (JuiceFS) — two systems that must stay consistent with each other, and restoring one without a matching copy of the other is not a restore, it's corruption. Rather than ship a backup that's half-covered, /wrk is out of scope entirely: treat it as scratch space shared across your boxes, not storage of record.

Also not backed up: box disks, microVM snapshots, and persistent homes. If a box's own files need to survive its host, that's a separate concern from the daemon's own state — this tool only covers what cannot be regenerated at all: the front door's identity and the control-plane databases.

See also

• Deploy a server — provision.sh installs the timer as its last step.
• hopboxd config — --db and --host-key, the paths this tool backs up.
• Shared workspace (/wrk) — why it's separate, and not covered here.