Backups for a database running at the edge

High Level Setup

Spilo runs PostgreSQL with Patroni, which manages the cluster’s leader and replicas. The leader accepts writes. Replicas follow it by replaying PostgreSQL’s write-ahead log (WAL), and can usually serve read-only queries. If the leader fails, Patroni can promote a healthy replica so the database can continue operating.

The important question is where those instances run. Two database pods on one machine provide little protection from a machine failure. Placing them on separate Kubernetes nodes protects against that failure; placing them in separate availability zones can protect against a wider outage. Replication improves availability, but it is not a backup: an accidental deletion or bad migration is replicated too.

An edge setup is simply one where the database runs close to the application or users, perhaps at a site with an unreliable connection to a central cloud. Spilo works there, but it is equally useful in a conventional data center or cloud cluster. At an edge site, it is especially important to ask which resources share a failure domain: the database, its replicas, the Kubernetes control plane, and the backup storage.

Kubernetes can include nodes in different zones or regions. For example, a K3s worker can be joined with the following command, after providing the server address and join token:

curl -sfL https://get.k3s.io | \
  K3S_URL=https://SERVER_ADDRESS:6443 \
  K3S_TOKEN=JOIN_TOKEN sh -

Joining a node does not automatically place a replica there. Node labels and pod placement rules determine where database instances run, while the storage and network must support that placement. Distance also adds latency, which matters particularly for replication and failover. Kubernetes describes scheduling across failure zones.

Backups

Spilo can use WAL-G to take periodic base backups and archive completed WAL segments to object storage. The base backup provides a starting copy of the database; the WAL records subsequent changes. A restore needs both a usable base backup and the WAL sequence leading to the desired recovery point. PostgreSQL calls this point-in-time recovery.

The backup schedule and destination are configuration choices. One design sends backups to in-cluster object storage such as MinIO, then copies them to another region or cloud provider. That can be useful when an edge site temporarily loses its internet connection. Another design sends WAL-G backups directly to remote object storage.

If local object storage is used, it should at least be on a different node from the database. A copy in another site or region protects against losing the entire local cluster. The copies also need appropriate retention and regular restore tests; seeing backup files in a bucket does not establish that they can be restored.

Recovery

For a failed database node, the quickest path is often promotion of a replica. Applications reconnect to the new leader, and the failed instance can later be rebuilt as a replica. With asynchronous replication, the newest transactions might not have reached the promoted replica. Patroni explains its replication modes and failover behavior.

Backups handle a different class of problem: lost storage, replicated mistakes, or an outage that takes down the whole site. Recovery involves starting a new cluster from a base backup and replaying archived WAL to a chosen point. If backups were first stored locally and then copied off-site, a disaster recovery procedure could copy the archive into replacement storage before restoring it. The exact path matters less than proving that the full procedure works.

Replicas in Another Region

Spilo can support regional redundancy. One option is to run replicas in a Kubernetes cluster whose nodes span regions. This requires suitable pod placement, network connectivity, and storage in each region. It also ties those nodes to the same Kubernetes control plane, so a remote replica alone does not make the whole system region-independent.

Another option is a separate standby cluster in the second region. Patroni supports a standby leader that follows a remote primary, with further replicas following that standby leader. The Zalando Postgres Operator can configure this using a connection to the primary or an S3/GCS WAL archive. If the first region is lost, operators can deliberately promote the standby and redirect applications to it. That decision must account for replication lag and ensure the old primary cannot also accept writes. Patroni and the Postgres Operator document this pattern.

Drawbacks

• Every replica needs storage for another copy of the database.
• Cross-region replication uses bandwidth and introduces delay. Requiring a remote replica to confirm every write can increase application latency.
• Failover can lose recent transactions if replication is asynchronous.
• A local object store may fail alongside the database if both depend on the same site.
• Remote failover needs a plan for promotion, application routing, and preventing two writable leaders.

Benefits

• Replicas on separate nodes can shorten recovery from a node failure.
• Placement across zones reduces reliance on one data center.
• A standby cluster can keep a recent database copy ready in another region.
• Off-site base backups and WAL allow recovery to an earlier point after replicated mistakes.
• Kubernetes provides a way to add nodes and express placement rules as the system grows.