Runbook: backup and restore#

Audience: site operator / MSP admin — you host Marula yourself and own its deployment.

This is the first authored page of the deploy/runbooks/ series. The scaffold and future runbook index live at deploy/runbooks/README.md.

Everything in Marula lives in the Postgres database, and once S3 object storage is enabled also the object-storage (Garage/S3) bucket. Back both up or back neither up alone: document attachments (content-addressed uploads) and versioned documents reference object storage, while everything else — tickets, time, invoices, FX rates, the provenance and audit ledges, AI decision answers, customization rows — is rows in Postgres.

Backup#

Three tiers — pick by how much you can lose. The compose backup profile (runner at deploy/backup/) automates every tier below:

docker compose -f deploy/docker-compose.yml exec postgres \
  pg_dump -U marula -d marula -Fc -f /tmp/marula.dump
docker cp marula-postgres-1:/tmp/marula.dump marula-$(date +%F).dump

Notes the runbook enforces:

• Dump as the table owner (marula), not marula_app — the RLS-constrained app role must never be able to read across tenants, which also means it cannot produce a complete dump.
• Dump with a client that matches the server's major version. The backup runner runs inside the postgres image on purpose; a diverged local pg_dump refuses the archive outright.
• If you run OBJECT_STORE=s3 with Garage, the restic leg snapshots the garage data/meta volumes in the same window so attachments and rows stay as consistent as two separate stores allow — this is not atomic: a few seconds of drift between the WAL cutoff and the object snapshot is possible. Both directions of that drift have a repair path now: the worker's object integrity pass (MARULA_OBJECT_GC=on) reclaims objects no row references and marks rows whose bytes are gone, and re-uploading the same file clears the mark, because the object key is that file's digest. See data lifecycle §3.
• The same pass snapshots the local object store, which lives in the objectdata volume now — it used to be a directory inside a read-only container, which could not survive a restart.
• Take a dump before every upgrade step (see upgrade step 3). The upgrade preflight runs against a database copy — supply it with a fresh one.

WAL-G specifics (PITR)#

1. Copy deploy/backup/wal-g.env.example → wal-g.env (compose: same for restic), fill in the Garage/S3 coordinates, and build with WAL-G: docker compose --profile backup build --build-arg INSTALL_WAL_G=1 backup. The template also takes the database connection — PGHOST, PGPORT, PGUSER, PGDATABASE, and a PGPASSWORD or PGPASSFILE. Those are libpq's own names, so pg_dump, psql and wal-g all read the same values.

2. The PGUSER role needs REPLICATION in addition to read access. A backup-push with no data directory streams the base backup over the BASE_BACKUP protocol, which Postgres accepts only from a role holding that attribute, and the replication connection also needs a pg_hba.conf line — initdb writes host replication for loopback only, so a backup container on another address is refused until the rule exists:

   Copy

   CREATE ROLE marula_backup LOGIN PASSWORD '<generated>' REPLICATION;
   GRANT CONNECT ON DATABASE marula TO marula_backup;
   GRANT pg_read_all_data TO marula_backup;

   Copy

   # /var/lib/postgresql/data/pg_hba.conf, then SELECT pg_reload_conf();
   host replication marula_backup all scram-sha-256

   The compose marula role is the cluster superuser and holds both, so a single-host install that uses it needs no extra grant. The restore drill is a different role: restore.sh creates and drops databases and roles, and its PGUSER needs CREATEDB and CREATEROLE for that.

3. Enable WAL archiving by starting postgres with -c archive_command='wal-g wal-push %p' — compose-override file documented in deploy/backup/compose.walg.example.yml.

4. wal-g backup-push runs on the scheduled nightly pass. Retention via WALG_RETAIN_FULL: after each push the runner calls wal-g delete retain FULL <n> --confirm. The flag matters — without it wal-g delete only lists what it would remove, so a destination fills up while every night's log says retention ran. To rehearse before letting it delete, set WALG_RETENTION_DRY_RUN=1: the runner then calls the same command without --confirm and logs retention DRY RUN.

5. Alert on archive_command silence. A broken archive pipe fails silently and quietly degrades you from "any second" back to "last night." If WAL shipping shows no progress for >24h (check the archiver's pg_stat_archiver view), page yourself.

Restore#

1. Stop the services (docker compose … stop core-api worker).

2. For the last-night tier: plain pg_restore (below). For PITR: follow wal-g restore — stop Postgres, wal-g backup-fetch, replay WAL to the recovery target, then start. Stop at a point after the damage, not before it (recovery_target_time defaults to "latest consistent state").

3. Rehearse before you need it:

   Copy

   sh deploy/backup/restore.sh /tmp/marula.dump

   It restores into a scratch DB and asserts the restore is real: the schema lands (with RLS enabled on every org-scoped table), and when the source stack is reachable the row counts of every public table match, which catches a dump that loses rows in transit. Set PGDATABASE=marula (plus PGSOURCE_HOST/PGSOURCE_PORT against a different server) to turn the parity check on. Without it the leg stops after the schema checks. It drops the scratch DB when done — a backup that has never restored is not a backup. The backup profile already runs this drill on a cadence (DRILL_EVERY_DAYS, default 7) and records each run, with its duration, in ops.backup_drill_runs; RPO and RTO explains what that duration is for and what the drill cannot prove.

4. Restore the garage data/meta volumes or bucket contents if you use object storage.

5. Start the services. Do not re-run migrations between restore and start unless the restored dump predates a schema change you have since applied — migrations are forward-only and append-only, history is never edited, so any deployed database replays them identically.

Verify a restore#

Boot a scratch stack pointed at the restored volume and log in. Confirm: an open ticket renders in the workbench, today's time entries exist, and /audit shows the append-only ledger intact. The upgrade-fixture CI restores are production-like snapshots — treat your restore rehearsal the same way, not as a theoretical step.