Take a backup by hand
For a one off archive, before an upgrade for instance, the block below writes a compressed custom format archive, fails on any error, and verifies that PostgreSQL can read its table of contents before declaring success. For anything repeating, use the script in Automate it rather than pasting this into cron.umask 077 gives the archive restrictive permissions, which matters because the dump contains everything.
Both commands run inside the postgres container, so the client always matches the server and there is no version to keep in step. Running pg_dump from the host instead needs a client at least as new as the server, which is PostgreSQL 18. An older host client stops with server version mismatch and writes nothing.
Run it from the repository root with the same Compose project name the deployment uses. The default project name is the directory name. For a deployment started with docker compose -p production, add -p production to every command on this page.
Where does the data live?
In the named volumepgdata, mounted at /var/lib/postgresql in the postgres container. Compose prefixes it with the project name, so it appears as production_pgdata under a project called production.
Copying the volume directory while PostgreSQL is running is not a backup. Use pg_dump, which produces a consistent snapshot.
How long it takes
Measured on a running stack, restoring an archive the scheduled script produced rather than one taken by hand for the drill.
That instance holds 2 apps and 30 tracked keyword markets with 42 applied migrations. Both durations scale with row count rather than with database size, and the archive compresses well because the largest columns are repeated JSON payloads. A year of daily rankings for 20 apps at 200 keywords is roughly 1.5 million rows, so plan for minutes rather than seconds at that scale and measure your own instance before you rely on a maintenance window.
The restore was verified with the full checklist in Restore PostgreSQL: the archive’s table of contents read before anything was dropped, row counts matched the live database, all 42 migrations came back finished, no constraint was left unvalidated, both workspace scoped unique indexes were present, and an insert succeeded afterwards. A restore can report success and still refuse a write, which is why the write is on the list.
How long are archives kept?
The window is a decision, not a detail, because a deleted workspace is not deleted until it has aged out of the last archive that holds it. Write yours down before you schedule anything. The shipped script keeps two generations.
That makes 56 days the longest an archive can hold a row, and therefore the real deletion horizon for the deployment. Erasure removes data from the live database at once and from the last archive up to 56 days later. See Data export, deletion and retention.
Both live in
$HOME/asobeast-backups, daily/ and weekly/, on the same host by default.
Change the window by editing BACKUP_KEEP_DAILY and BACKUP_KEEP_WEEKLY at the top of the script, and change the deletion horizon you publish along with it. A shorter live retention promise than the backup window is not a promise you keep.
The offsite copies below carry their own window, BACKUP_REMOTE_KEEP_DAILY and BACKUP_REMOTE_KEEP_WEEKLY, which default to the local numbers so the published horizon does not move on its own. Lengthening the remote window lengthens the deletion horizon, and the horizon you publish has to be the longest one you actually keep.
Copy the archive off the host
A backup on the same disk as the database survives a mistake but not a disk. On a single machine the daily archives and thepgdata volume are the same SSD, so one failure loses the database and every copy of it in the same second. Theft, a wiped filesystem and ransomware have the same shape.
Offsite means somewhere the loss of this machine cannot reach: object storage, a different machine on a different power supply, or an external disk that is not mounted between runs. It does not mean a second directory or a second partition.
Set two variables and the shipped script does the rest.
Encryption is not optional offsite
pg_dump output contains every password hash, every session row, every billing identifier and every review the instance has collected. That is acceptable at chmod 700 on a machine you control and unacceptable in somebody else’s storage, so the script refuses to push anything unless BACKUP_AGE_RECIPIENT names a recipient.
Generate the key pair somewhere that is not the machine being backed up, and put only the public half on the host.
BACKUP_AGE_RECIPIENT. Keep the identity file off the host and off the remote: a compromised mini PC can then write its offsite history and never read it back. Losing that file loses every offsite archive, so store it the way you store the recovery codes for your accounts.
What the failure path does
A failed upload is a failed run. The exit code is what the scheduler reports, and the completion timestamp the API watches is written only after the upload lands. See Hosted observability.
The schedule is watched
A backup that silently stops running is discovered during a restore, which is the one moment it cannot be fixed. The exit code alone does not protect you, because it goes to a timer nobody reads. Every successful run therefore reports its completion time to Redis, and the API reads it back on every/metrics scrape. Set BACKUP_MAX_AGE_HOURS in the API environment to the age you consider stale, and backup.stale fires when the last run is older than that, escalating from investigate to page at twice the window. A run that never reported at all pages, rather than reading as fresh.
Leave BACKUP_MAX_AGE_HOURS at 0 if you back up some other way. Nothing is watched and nothing alerts, which is better than an alert you learn to ignore.
Reporting is best effort. A run that finished but could not reach Redis says so on stderr and still exits zero, because the archive is good; the instance then reports the schedule as stale until the next run gets through, which is the conservative direction to be wrong in.
Restore from the offsite copy
The offsite copy is only a backup once you have restored one. See Restore PostgreSQL, which restores an archive fetched from the remote with the host powered off, and records the recovery objectives.Automate it
The repository ships the script, atscripts/asobeast-backup.sh. Use it rather than a copy of the block above, because a copy on the host and a block in a document drift apart and nobody notices until a restore.
It dumps, verifies the archive with pg_restore --list, promotes the Sunday archive to the weekly generation, copies an encrypted copy offsite when one is configured, prunes every generation to its window, and exits non zero on any failure so the scheduler reports it. CI runs this same script from a directory that is not the repository, restores the archive it produces and compares it to the source, and then fetches the encrypted object back out of object storage and restores that too, so the scheduled path is the tested path. It writes to a .partial file and renames only after verification, so an interrupted run never leaves a truncated archive that pruning would later mistake for a good one.
Install it and schedule it.
COMPOSE_FILE is the one variable you have to set, and it must be absolute. Compose locates a deployment through its file, not through its project name, so a run that starts anywhere but the repository has nothing to talk to. The script refuses to start without it rather than dumping whichever stack happens to be under the working directory.
Cron runs with a minimal environment and from a working directory that is probably not the repository. Give cron a
HOME if BACKUP_ROOT depends on it, and set COMPOSE_PROJECT only if the deployment named its project, because the shipped Compose file already names it asobeast.
Check the failure path before you trust it
A schedule that fails silently is worse than no schedule, because it reports success by saying nothing. Verify the failure path once, on purpose, rather than discovering it during a restore. Stop PostgreSQL and run the script.service "postgres" is not running and exit=1, with no file written under daily/, because the dump goes to a .partial path that is removed on any failure and renamed only after pg_restore --list reads it. A failed run also stops before pruning, so it can never delete a good archive on its way out.
Then start PostgreSQL again and confirm the next run writes an archive.
When to run it
Back up shortly before the daily pipeline, not after it.CRON_DAILY defaults to 0 3 * * *, so the hour before that is the quietest point in the day: the previous run finished long ago, the next has not started, and no store jobs are in flight. The schedule above uses 02:00 for that reason.
Backing up during the run is not dangerous, because pg_dump takes a consistent snapshot, but it captures a day that is half collected. Restoring it later leaves a partial day that the next run will not revisit, since a store SERP is a point in time observation.
If you change CRON_DAILY, move the backup with it.
Keep tested generations off host. See Copy the archive off the host.
Verify a backup
A backup that has never been restored is not a backup. Schedule a real restore into a disposable stack on a regular cadence and confirm the app and keyword counts match. See Restore PostgreSQL.Related
Restore PostgreSQL
The other half of the drill.
Upgrade and roll back
Why the backup has to precede the upgrade.