hopbox-host reference

hopbox-host is the resilient deploy/update tool for a hopbox host (see Updating — hopbox-host for the walkthrough). Run hopbox-host --help for the authoritative usage banner.

hopbox-host update --from <dir|tar.gz> [--releases N]  # deploy PREBUILT artifacts — the production path
hopbox-host update [--ref main|<sha>] [--releases N]   # build on this host instead (needs a checkout + Go)
hopbox-host status                                     # channel + version, health, service, releases, image drift
hopbox-host rollback                                   # flip to the previous release (health-gated)
hopbox-host catalog [build <image>…]                   # rebuild base images (decoupled from update)
hopbox-host check [--accept]                           # health + drift since the deploy (run by a 5-min timer)
hopbox-host bootstrap --from <dir|tar.gz>              # config + unit + deploy + catalog + backup timer
hopbox-host clean [--apply]                            # collect the legacy pre-releases/ layout

Every deploy — built or prebuilt — goes through the same pipeline: stage into a fresh /opt/hopbox/releases/<id> dir → hopboxd --check → atomic symlink swap → restart → health gate → rollback to the previous release if the new one isn't healthy. The gate is two probes: the cheap --health poll until the daemon is back up, then one --health-full deep probe at the commit point, which proves the release can actually spawn a box before its id is recorded. Only how the release dir gets filled (Stage) differs between the two update forms — a prebuilt bundle gets the same deep gate as a host-built one.

update

Flag	Default	Description
--from	(empty)	Deploy prebuilt artifacts: a directory (make dist output) or a .tar.gz bundle (a GoReleaser release asset). No Go toolchain or checkout needed on the host — the path for a host serving users.
--ref	main	Git ref to build on this host and deploy (branch, tag, or sha). Needs a Go toolchain and a checkout at /opt/hopbox-src; suited to a development host, not a production one.
--channel	(empty)	trunk or release: the channel this deploy may leave the host on. Required only when that differs from the channel the host is on — see Changing channel.
--releases	3	Number of releases to keep on disk for rollback; older ones are GC'd after a healthy deploy. The currently-live release is always kept even if it falls outside this window.

--from and --ref are alternatives. Passing an explicit --ref together with --from is refused (--from and --ref are alternatives … pass one); --ref left at its default main is fine, since it just means "no ref was asked for." Leaving both off builds main — the same as before --from existed.

Re-running update when already deployed at the target release and healthy is a no-op (up to date at …); a same-release-but-unhealthy daemon is redeployed. That no-op check uses the cheap probe only — a drift-aware re-run must not spawn a box on a host that is already serving. Switching channel at the same commit (trunk → release or back) always redeploys — see Channel & version — including the first update on a host that predates channel tracking, which costs one extra redeploy to start recording it.

Changing channel

A deploy that would move the host between channels — release → trunk or back — is refused unless --channel names the destination:

$ sudo hopbox-host update --ref main
hopbox-host: refusing to move this host from the release channel to trunk — that changes what this host IS, and nothing asked for it
  deployed:  v0.6.1 (release) — artifacts /tmp/hopbox_linux_amd64.tar.gz
  target:    9c1f0b1a2b3c (trunk) — source build of main in /opt/hopbox-src
  trunk is a build only this host has ever seen; release is the artifact users install
  keep it on release:  hopbox-host update --from <hopbox_linux_amd64.tar.gz>  (https://hopbox.dev/dl/latest/)
  really move it to trunk:  re-run the same command with --channel trunk

The refusal happens before any host effect — nothing is fetched, staged or swapped.

Situation	--channel needed?
Deploying within a channel (release → release, trunk → trunk)	no
First deploy to a host with no recorded channel (fresh, or predating channel tracking)	no — it adopts a channel rather than changing one
release → trunk (e.g. --ref, or --from a locally built bundle)	yes: --channel trunk
trunk → release (deploying a tagged release bundle)	yes: --channel release

--channel asserts what the deploy is, so it can never be a blanket "yes": passing --channel release for a source build is refused (a build on this host is trunk whatever ref it names). A change that does go through is logged and named in the result line:

hopbox-host: deployed 9c1f0b1a2b3c (trunk) — channel changed release -> trunk

Why this exists: a routine update --ref main on a host running v0.6.1 (release) used to deploy cleanly, health-gate correctly, and silently leave the host on an untagged trunk build — no warning, no confirmation, nothing in the output — and a later --from of a release tarball moved it back the same way. A host's channel is a property of the host, not a side effect of which flag was typed.

bootstrap takes the same --channel flag and applies the same rule, so a provision.sh re-run (always a source build) cannot be a second door onto the same silent demotion — see HOPBOX_CHANNEL there.

--from layouts

Both artifact layouts hopbox produces are accepted; the binary matching the host's own architecture is picked automatically:

Source	Binary names	Produced by
A directory	hopboxd-linux-amd64, hopboxd-linux-arm64, … (both arches side by side)	make dist
A .tar.gz	hopboxd, box-guest, … (bare names, one arch per archive)	A GoReleaser release asset

# the tagged release users install
curl -fsSLO https://hopbox.dev/dl/latest/hopbox_linux_amd64.tar.gz
sudo hopbox-host update --from hopbox_linux_amd64.tar.gz

# an unreleased change, built where the code is
make deploy HOST=<host>          # cross-compile → scp → gated deploy, in one step

make deploy is the scripted form of make dist + scp + update --from, for one architecture (ARCH=amd64 by default). It also installs the bundle's hopbox-host before deploying, since update never replaces that binary itself. Pass CHANNEL=trunk the first time you send a local build to a host that runs a release.

Validation

A bundle handed to --from is not trusted — everything below is checked before anything is copied into a release dir, so a bad bundle fails with the live daemon untouched rather than half-populating a release:

• It must contain a hopbox-release.json manifest with a version or a sha — a bundle that can't say what it is is refused. (make dist and every GoReleaser release asset write this; see build/release-manifest.sh.)
• Each of hopboxd, box-guest, hopbox-mcp, hopbox-agent must be present, executable, and an ELF binary for this host's architecture — an arm64 tarball unpacked on an amd64 host is refused by name, not discovered later by --check trying to run it.
• A stage that fails removes its half-populated release dir (unless that dir is the live one, i.e. a same-release redeploy) — it can never become a rollback target.

A bundle also carries backup.sh, which the deploy installs as hopbox-backup. It is optional — a bundle built before it rode along still deploys, and the deploy falls back to /opt/hopbox-src/deploy/backup.sh — but it is how an artifact-fed host with no checkout keeps its backup/restore tool in step with its daemon instead of frozen at provisioning time. It lands in the release dir, so a rollback restores the script that release shipped with. Every bundle producer writes it (make dist, make deploy, GoReleaser, provision.sh's staging bundle).

hopbox-host itself is not replaced by update — it never was, built or prebuilt. The bundle does contain a hopbox-host binary; installing a new one is a manual step:

sudo install -m755 hopbox-host-linux-amd64 /usr/local/bin/hopbox-host   # from a dist/ dir
sudo install -m755 hopbox-host /usr/local/bin/hopbox-host               # from a tar.gz bundle

status

channel:   release            # or: trunk  (an unreleased build — NOT the artifact users install)
                              # or: (unknown — deployed before channels were recorded)
version:   v0.6.1             # or (untagged)
deployed:  9c1f0b1a2b3c
source:    artifacts /root/hopbox_linux_amd64.tar.gz     # omitted when unknown
health:    healthy
service:   active
releases:  N (newest first) …
images:    in sync
legacy:    3 pre-releases/ path(s) under /opt/hopbox, 900.5 MB — run `hopbox-host clean`

A trunk host says so in words rather than leaving a bare channel name that reads the same on a dev box and on one serving users. It is a note, not an alert: check never fires on it, because reaching trunk takes an explicit --channel.

The legacy: line appears only when a pre-releases/ layout is present; a host without one stays silent. See clean. status always uses the cheap --health probe — it is a report, not a deploy gate, and must not spawn a box on a host that is serving.

images: — are the images stale?

in sync, or DRIFT naming what went stale:

images:    DRIFT: box-guest — run `hopbox-host catalog build` so new boxes get what is deployed

A content-sha comparison over every artifact catalog build bakes into an image — not just the agent:

Artifact	Deployed copy compared
hopbox-agent	/var/lib/hopbox/hopbox-agent-linux-amd64
box-guest	/usr/local/bin/box-guest
hopbox-init	providers/compute/microvm/assets/hopbox-init in /opt/hopbox-src (a checked-in script, not a build output)

catalog build records one <name> <sha> line per artifact in /opt/hopbox-microvm/images/.baked-shas, and status compares that baseline to what is on disk now. DRIFT means a box spawned right now would not get the named artifact — existing boxes keep theirs until they respawn either way. This line is the only place drift is reported; check, the timer, watches health and the release fingerprints instead.

Unknown never warns, from either side. An artifact missing from the baseline (images built before it was tracked) and one that can't be read on disk are both "can't tell" — a host deployed from a prebuilt bundle has no /opt/hopbox-src, so hopbox-init is legitimately absent there and stays silent rather than warning forever. A host that has not rebuilt its catalog since .baked-shas existed still has the old agent-only .agent-sha, which is read as the agent's entry until the next rebuild retires it.

A box-guest change used to be invisible

Only the agent was fingerprinted, so a deploy that changed box-guest — baked into every image, claude-code included — reported in sync while every image kept the old one until someone happened to rebuild. Both the catalog build and this check now read one list, so they cannot come apart again.

The comparison is only meaningful if a binary is a function of its source, so the two baked binaries — hopbox-agent and box-guest — are built with -buildvcs=false on every path that produces them: this tool's own source build, provision.sh, the Makefile, the release pipeline, and the image builder's fallback. Go otherwise stamps the commit (vcs.revision, vcs.time, vcs.modified) into every binary it links, which gave the agent a new content sha on every commit — so a daemon-only deploy reported DRIFT with the agent's source and dependency closure untouched, and a genuine change was indistinguishable from that noise. Unstamped, the sha moves only when the artifact does.

Dropping the stamp loses nothing you can't get elsewhere: what a host runs is recorded per release in deployed-release.json and in each release dir, which is where the deployed:, channel: and version: lines above come from. Nothing has ever read a commit back out of a binary.

One drift, once, on the upgrade

The first deploy that crosses this change swaps a stamped agent for an unstamped one, so status says DRIFT — correctly, because the images really do hold a different binary. Run hopbox-host catalog build once to record the new baseline; after that the line stays in sync until a baked artifact itself changes.

Channel & version

Channel	Meaning
trunk	Compiled on this host from a git ref — including a tag: building a tag from source is still not the artifact users install.
release	A prebuilt bundle that declares itself one (a GoReleaser tag build).

A bundle with no declared channel is treated as trunk, never silently promoted to release. Moving a host from one channel to the other takes an explicit --channel. The deploy record lives in /var/lib/hopbox/deployed-release.json (channel/version/sha/source) alongside the existing /var/lib/hopbox/deployed-sha (unchanged — still just the release id, for anything that greps it). A copy of the record is left in every release dir, so rollback restores the channel/version bookkeeping too, not just the binaries.

rollback

Flips to the previous release (by mtime) behind the same health gate update uses. A successful rollback also restores that release's recorded channel/version. Fails with no previous release to roll back to if there isn't one.

Rollback gates on the cheap probe only. Its job is "the daemon is not left down", and a host-level fault — an unreachable object store, a wiped image catalog — would fail the deep probe on every release, so a deep-gated rollback would blame the release it just restored instead of reporting the honest outcome.

catalog

hopbox-host catalog build              # rebuild every base image in the curated catalog
hopbox-host catalog build claude-code  # rebuild just one image

Decoupled from update on purpose — a code deploy never triggers a slow, destructive image rebuild. Existing boxes only pick up a new hopbox-agent or box-guest on their next respawn; status's images: DRIFT line is the reminder, and it names every baked artifact that genuinely changed. On success it records the new baseline in .baked-shas — which is why rebuilding is what makes the line go quiet. See the image catalog.

check — between deploys

hopbox-host check            # what the hopbox-check.timer runs, every 5 minutes
hopbox-host check --accept   # re-baseline the fingerprints against what is on disk now

update health-gates and rolls back, and status is accurate — but only when a human runs it. check is the part that watches a host between deploys: a daemon that goes unhealthy an hour after a good deploy, or a binary replaced by hand, used to be invisible. Exits non-zero when anything alerting is wrong, so systemctl status hopbox-check tells the story even with no webhook configured.

Check	Alerts?	Why
health	yes	The daemon is not answering its front door. Uses the cheap --health probe — the deep one spawns a box and has no business running every five minutes on a host that is serving.
service	yes	systemd reports the unit as something other than active.
binary	yes	A file in the deployed release dir no longer matches what that deploy staged. A release is meant to be an immutable rollback target; this is the only thing that verifies it.
config	no — reported	hopboxd.yaml was edited since the deploy. Usually deliberate, so it never pages; check --accept re-baselines it.
baseline	no — reported	No fingerprints recorded yet. Every host deployed before this existed starts here; the next deploy records a baseline.

Fingerprints are written by a healthy deploy (alongside deployed-sha and deployed-release.json) into /var/lib/hopbox/deployed-fingerprints.json: a sha256 per release binary plus one for the config.

Alerting

Set a webhook in /etc/hopbox/alert.env:

HOPBOX_ALERT_WEBHOOK=https://hooks.example.com/…

The check POSTs the JSON CheckResult (host, release, findings, timestamp) on a state change only — when it breaks, when it recovers, and when it breaks in a new way. It stays silent while a known problem persists. A monitor that posts every five minutes during an outage is one people mute, and a muted monitor is the same as no monitor. State lives in /var/lib/hopbox/check-state.json.

Unconfigured, the check still runs and still fails visibly in systemd — it just cannot page anyone.

bootstrap

hopbox-host bootstrap --from <dir|tar.gz>

Brings a host from "the prerequisites are installed" to "deployed, running, and backed up". provision.sh calls it as its last step; you rarely run it by hand.

Flag	Default	Description
--from	(required)	The binaries to deploy — the same bundle shape update --from takes. Refuses without it rather than half-configuring a host.
--channel	(empty)	Passed straight to the deploy, with the same rule as update --channel. provision.sh forwards HOPBOX_CHANNEL into it: the provisioner builds from source, so re-running it on a host that runs a release artifact would move that host to trunk, and is refused unless asked for. Without this, a provisioner re-run would be a second door onto the demotion update now refuses.

In order:

1. Config — writes /etc/hopbox/hopboxd.yaml (0600) only when absent; an existing one is never touched, because that is where your real settings live. On a first install it picks the front-door port: :22 if free (the cutover target), else 2222/2200/8022, so it can never displace the host's own sshd and lock you out. It also prunes accumulated <file>.bak* siblings of the config and the Caddyfile, keeping the 3 newest.
2. Unit — rewrites hopboxd.service every run, and drops stale hopboxd.service.d drop-ins (a capability-capping one breaks bridge setup). This is how NotifyAccess=main and TimeoutStopSec=300 reach a host that has only ever been updated — update ships binaries, not units, so a host installed before the drain budget landed keeps the old unit until a bootstrap runs.
3. Deploy — on a host already under release management, hands the bundle to the same gated pipeline as update --from. On a fresh host, installs directly; the first update migrates that into /opt/hopbox/releases.
4. Catalog — runs catalog build after the deploy, so the images bake the agent that was actually installed, and so the drift fingerprint is recorded (only hopbox-host writes it — see agent: in status).
5. Backups — installs hopbox-backup + the daily timer, and the example backup.env when absent. It does not pick a destination: until HOPBOX_BACKUP_DEST is set the timer fails loudly (exit 78), and bootstrap says so.
6. Monitoring — installs hopbox-check.timer (every 5 minutes) and an example alert.env. See check.

It finishes by reporting the front door read back from the config, not the port it would have picked — on a host with an existing config those differ, and printing the guess is how a provisioner run ends by naming a port nothing is serving.

clean — collecting the legacy layout

Hosts provisioned before the <base>/releases/<sha> layout landed still carry their old v<MAJOR>.<MINOR>.<PATCH> install directories (and a vestigial current symlink) as direct children of /opt/hopbox. update --releases N's GC only ever looks inside /opt/hopbox/releases, so it can never see this older layout — those directories just accumulate (commonly several hundred MB to ~1GB). clean collects them.

It previews by default and only deletes under --apply:

$ hopbox-host clean
  current                         0 B
  v0.6.2                     412.0 MB
  v0.7.13                    488.5 MB
3 legacy path(s) under /opt/hopbox, 900.5 MB
dry run — nothing removed; re-run as `hopbox-host clean --apply` to collect

$ hopbox-host clean --apply
  current                         0 B
  v0.6.2                     412.0 MB
  v0.7.13                    488.5 MB
3 legacy path(s) under /opt/hopbox, 900.5 MB
clean: collected 3 legacy path(s), freed 900.5 MB

On a host with no legacy layout, clean is a no-op (nothing to collect).

Safety guards:

• It only ever considers direct children of /opt/hopbox matching the strict legacy shape — a vMAJOR.MINOR.PATCH directory name or the current symlink — never something nested deeper or a hand-made directory that happens to live there.
• It never removes /opt/hopbox/releases (the current layout).
• It never removes a directory the live /usr/local/bin/{hopboxd,box-guest,hopbox-mcp} symlinks resolve into, checking both the one-hop and the fully-resolved target so an intermediate symlink can't hide the live release from the guard.
• If there is something to collect but clean cannot resolve which release is live (no $PREFIX symlink yet — i.e. before the very first update), it refuses rather than guesses.

clean is deliberately a separate subcommand, not a step inside update — a one-off delete of this size does not belong inside an unattended deploy window. status reports what's there (legacy: …) so you know to run it; it stays silent when there's nothing to collect.

Host layout hygiene

What needs /opt/hopbox-src

Two things, and nothing else:

• update --ref — the build-on-host deploy path;
• catalog build — the microVM image catalog is defined by build/microvm/catalog.sh and the guest assets in the source tree.

A host deployed only with --from, and whose images are built elsewhere (or which runs the docker backend), needs no checkout and no Go toolchain at all — that is the point of the artifact path. Everything a release needs travels in the bundle, including the hopbox-backup script.

/opt/hopbox-src is root-owned, so git's dubious-ownership guard makes git -C /opt/hopbox-src log refuse to run for a non-root login — the normal way a human checks what's actually deployed. Both hopbox-host update (on every build-on-host run) and provision.sh (on provisioning and every re-run) mark it a git safe.directory at system scope (/etc/gitconfig, so it covers every account on the host, not just root's). This is idempotent and best-effort — it never fails a deploy. A --from deploy skips it: there may be no checkout to mark.

build/release-manifest.sh

Both artifact producers (make dist, GoReleaser) write hopbox-release.json via this script, which is what makes their output a bundle --from can deploy:

build/release-manifest.sh <out.json> [version] [sha] [channel]

With no arguments it describes the current checkout — an exact tag is the release channel, anything else is trunk. make dist calls it that way; GoReleaser passes the tag it is building (and release) explicitly. provision.sh passes trunk explicitly, because a source build of a tag is still not the artifact users install.

Note that a make dist of a checkout sitting exactly on a tag therefore declares itself release. That is deliberate — those are the same bits CI would publish — but it means such a bundle can move a host onto the release channel, which --channel makes you say out loud.

See also

• Updating — hopbox-host — the guide walkthrough: when and why to run each command, plus the zero-touch provisioner.
• hopboxd reference — the daemon hopbox-host deploys, and the --check / --health / --health-full probes it runs on every deploy.