Run an active and a standby instance

Goal

Run two henceforth serve processes against one database, with exactly one active and the other waiting as a standby, and have your supervisor or orchestrator restart them so that a stop, a crash, or a database failover ends with one active instance again.

Prerequisites

A PostgreSQL database reached directly, or through a session-mode pooler. PgBouncer in transaction mode breaks the instance lease; do not put it between the server and the database.
One spare database connection per process, standbys included.
A DATABASE_URL role that owns the database. At every start the server revokes CONNECT on the database from PUBLIC, which only the owner can do, and refuses to start otherwise. Both processes should use the same role; a second role needs GRANT CONNECT ON DATABASE "<db>" TO "<role>";.
The henceforth binary (or container image) and the flags you already use to start a single instance, including --database-url (or DATABASE_URL), --metadata-path, --bind and --public-origin for production. See Serve an instance on a public URL.
A way to give a standby the same metadata as the active instance: the same --metadata-path on a shared volume, or a clone of the metadata repository at the same git ref. The metadata git repository is not stored in PostgreSQL.

Steps

1. Start both processes with identical configuration

Start the second process like the first, against the same --database-url, on another host or on another port (two processes cannot bind one port). It needs no extra flag: it detects that the database is held, logs the holder, and waits.

INFO standby: another Henceforth instance holds this database's lease (host=web-1 pid=4242 since=2026-10-11T08:00:00+00:00 from=10.0.0.5 role=same); waiting

2. Probe liveness and readiness separately

Endpoint	Active	Standby
`GET /health`	`200`	`200`
`GET /ready`	`200`	`503`

Point liveness probes at /health and readiness probes at /ready. A standby answers 503 INSTANCE_STANDBY on every other route, so a load balancer that checks /ready sends traffic only to the active instance.

Scripts that wait for a server to become usable must wait on /ready. /health answers as soon as the process listens, which is before it holds the lease and finishes startup.

3. Under systemd, restart on every exit

[Service]
ExecStart=/usr/local/bin/henceforth serve --bind 127.0.0.1 --public-origin https://hf.example.com
Restart=always
RestartSec=1

A process whose lease connection drops, errors or stops answering exits with status 75 and logs instance lease lost. Restart=always (or Restart=on-failure, which counts 75 as a failure) restarts it, and it comes back as a standby. systemctl stop sends SIGTERM: the process stops accepting new connections (/ready answers 503 only on connections that are already open), finishes in-flight requests for at most 20 seconds, and releases the lease so a waiting standby takes over. Requests still open after 20 seconds are cut off when the process exits: an uncommitted write rolls back, and a committed write may go unacknowledged, so clients should treat a reset connection on a write as an unknown outcome. A second signal exits at once.

4. On Kubernetes, avoid a rollout that needs two ready pods

spec:
  strategy:
    type: Recreate
  template:
    spec:
      containers:
        - name: henceforth
          livenessProbe:
            httpGet: { path: /health, port: 8080 }
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }

Only one pod is ever ready, so a rolling update that waits for a new pod to become ready before it stops the old active never finishes: the new pod stays unready while the old one holds the lease. With two replicas the standby already counts as unavailable, so even maxUnavailable: 1 deadlocks once the old standby is replaced. Use Recreate. A RollingUpdate works only with maxUnavailable equal to the replica count (100%), which behaves like Recreate.

Kubernetes restarts a container that exits 75, so a lease loss ends with a standby pod, not a failed one.

5. Upgrade by cutting over

Start a standby on the new binary, then stop the old active instance. The standby takes the lease and, because it took it from a live holder, waits about seven seconds before it starts, so the old process’s lease check has fired. An uncontested start has no wait.

Verify

Check readiness on both instances:

curl -i http://active-host:8080/ready
curl -i http://standby-host:8080/ready

The active instance answers HTTP/1.1 200 OK with body OK. The standby answers HTTP/1.1 503 with body not ready. Stop the active instance, then repeat: the standby’s /ready turns 200 within about ten seconds plus its startup time.

Troubleshooting

A pod or process stays unready forever: another instance holds the lease. Find it in the standby’s log (host= and pid=), or list lease holders with

SELECT l.pid, a.application_name, a.usename
FROM pg_locks l JOIN pg_stat_activity a USING (pid)
WHERE l.locktype = 'advisory' AND l.granted
  AND l.classid = 1212566597 AND l.objid = 1095976192
  AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database());

(standbys also have a lease connection, but only the holder has the lock).

The active instance keeps exiting with status 75: its database connection is being dropped. Check for pg_terminate_backend, a failover, a proxy idle timeout, or a transaction-mode pooler in front of PostgreSQL.
The standby log warns that the holder “does not look like a Henceforth lease connection of this role”. The server restricts CONNECT, so a foreign holder is a superuser, the database owner or a member of it, a role granted CONNECT explicitly, or a session that was open before CONNECT was restricted. Find the session with the query above (it lists whoever holds the lease key in this database, whatever it calls itself). End it with SELECT pg_terminate_backend(<pid>);. To keep an explicitly granted role out, REVOKE CONNECT ON DATABASE "<db>" FROM "<role>"; (this does nothing while the role holds CONNECT through membership in another role).
The server exits at startup with PUBLIC still holds CONNECT ... through a grant by "<role>": that role granted CONNECT to PUBLIC with GRANT OPTION, and the owner cannot revoke it. Run one of the remedies in the message. Either SET ROLE "<grantor>"; REVOKE CONNECT ON DATABASE "<db>" FROM PUBLIC; (a plain REVOKE as superuser only removes the owner’s own grant), or, as the owner, REVOKE GRANT OPTION FOR CONNECT ON DATABASE "<db>" FROM "<role>" CASCADE; for EVERY role the message lists (each statement cuts one delegation path, and CASCADE also removes the grants made on that role’s grant option). When the message lists only SET ROLE, no delegation chain back to the owner is visible.
The server exits at startup with database "<db>" is owned by ...: the DATABASE_URL role does not own the database. Run the ALTER DATABASE "<db>" OWNER TO "<role>"; from the message as a superuser, then the GRANT CONNECT it names for the old owner if that role still needs access (ownership changes move the old owner’s privileges to the new owner).
The server exits at startup with could not take the CONNECT ACL lock: a session holds the lock that serializes changes to the database’s CONNECT list. The message names it; end it with pg_terminate_backend. If it says the holder had released the lock by the time it was looked up, start the server again.
The server exits at startup with could not lock the catalog row of database ...: an open transaction holds a catalog object the step changes (the database row, or a role it grants to), typically a GRANT or ALTER DATABASE that was never committed. Find the idle in transaction session in pg_stat_activity and end it.
A backup or monitoring job fails with permission denied for database: the server revokes CONNECT from PUBLIC. Run GRANT CONNECT ON DATABASE "<db>" TO "<role>"; for that role. Superusers and the database owner are not affected.
hf dev fails with INSTANCE_LEASE_HELD: another instance holds the database hf dev uses. Run hf dev stop in the other project, or point DATABASE_URL at another database. hf dev refuses instead of waiting.