A look into database replication at the edge. With Spilo

April 20, 2025

An overview on how, and why the postgres database cluster works, why to choose it, and how it might save you money.

How Spilo and Patroni Configure PostgreSQL and WAL-G

Last repository review: 2026-10-04.

Scope

This article describes an edge deployment where Spilo archives PostgreSQL WAL and base backups to a local S3-compatible MinIO service. Keeping the first copy local allows backups to continue when the connection to a cloud provider is unreliable. The local archive can then be copied or replicated to durable off-site storage when connectivity is available.

Local MinIO is not automatically an off-site backup. If PostgreSQL and MinIO share a machine, disk, site, power source, or administrator, one incident can destroy both the database and its backup. So keep in mind, local duplication and multiple local nodes is very important.

Spilo packages PostgreSQL, Patroni, and backup/restore tooling such as WAL-G (WAL-E is now legacy) . Exact paths and generated commands vary by Spilo version, so inspect the running pod before relying on any example here.

Glossary

PostgreSQL is the database that stores the application’s durable data. It writes every change to its Write-Ahead Log before updating the main database files.
Spilo is a container image that packages PostgreSQL with high-availability and backup tooling. It converts environment variables into the runtime configuration used by PostgreSQL, Patroni, and WAL-G.
Patroni manages PostgreSQL availability. It decides which instance is the primary, maintains replicas, and coordinates controlled failover when the primary becomes unavailable.
WAL (Write-Ahead Log) is PostgreSQL’s stream of database changes. Archived WAL files can be replayed after a base backup to restore the database to a particular point in time.
WAL-G uploads PostgreSQL base backups and WAL files to object storage and retrieves them during recovery. In this edge design, it writes to an S3-compatible MinIO service.
MinIO provides S3-compatible object storage close to the edge database. It allows backups to continue during cloud connectivity problems, though those backups should later be copied to another machine, site, or cloud bucket.
Zalando Postgres Operator converts a Kubernetes PostgreSQL resource into StatefulSets, Services, Secrets, and Spilo configuration. It also continually reconciles the running cluster with the declared configuration.
Kubernetes runs and supervises the database containers. It provides storage, networking, Secrets, configuration sources, and replacement of failed pods.
A base backup is a physical snapshot of the PostgreSQL data directory. Recovery begins with a base backup and then replays archived WAL to reach the desired recovery time.
Point-in-time recovery (PITR) combines a base backup with an uninterrupted WAL archive. It allows PostgreSQL to be restored to a selected moment rather than only to the time of the last base backup.

Configuration flow

In a Zalando Postgres Operator deployment, configuration begins before the Spilo container starts:

Postgresql custom resource
        |
        v
Zalando Postgres Operator
  - reads structured fields such as clone: and postgresql.parameters
  - combines operator defaults and explicitly supplied pod environment
  - renders a StatefulSet, Services and credential Secrets
        |
        v
Kubernetes starts the Spilo container with its final environment
        |
        v
Spilo initialization (/launch.sh and configure_spilo.py)
  |-- generates Patroni configuration (/run/etc/patroni.yml)
  |-- generates WAL-G envdir files (/run/etc/wal-e.d/env/)
  |-- configures PostgreSQL archiving and recovery
  `-- installs the scheduled base-backup cron job
        |
        v
Patroni starts and manages PostgreSQL
        |
        v
WAL-G archives completed WAL and creates scheduled base backups

The container environment may be assembled from the PostgreSQL custom resource, operator-wide configuration, ConfigMaps, Secrets, and Kubernetes metadata. Spilo does not retain the origin of each value. At startup it sees only the final process environment.

What happens at startup

1. The operator renders the pod template

The operator converts structured custom-resource fields into a StatefulSet. For example, it can translate a clone: block into CLONE_* environment variables expected by Spilo. Values in spec.env are already low-level Spilo inputs and are copied into the container environment.

This distinction establishes ownership:

use structured operator fields where the custom resource provides them;
use spec.env for non-overlapping Spilo settings;
use Secrets for credentials and mount or reference them deliberately;
do not manually recreate environment variables that the operator generates.

There is no useful general rule that "the CR always wins" or "the ConfigMap is read later." Kubernetes constructs one environment before starting the container. Explicit env entries override values imported through envFrom when names overlap, but duplicate names in an env list remain ambiguous and must not be used as a configuration strategy. Controllers may also compare the ordered list while reconciling the StatefulSet.

2. Spilo generates runtime configuration

Spilo's startup scripts consume the recognized environment variables, validate the configuration, and generate files used by Patroni, PostgreSQL, and WAL-G. A simplified Patroni example looks like this:

# Conceptual example only; inspect the running Spilo version for exact output.
scope: acid-minimal-cluster
name: pod-name

postgresql:
  listen: 0.0.0.0:5432
  connect_address: pod-ip:5432
  data_dir: /home/postgres/pgdata/pgroot/data

  parameters:
    archive_mode: "on"
    archive_timeout: 60s
    wal_level: replica
    archive_command: "<Spilo-generated WAL-G command>"

  recovery_conf:
    restore_command: "<Spilo-generated WAL-G command>"

  create_replica_methods:
    - basebackup_fast_xlog

  basebackup_fast_xlog:
    command: /scripts/basebackup.sh
    retries: 2

The base-backup schedule is not a Patroni backup schedule. Spilo installs it in the pod's crontab.

3. Spilo generates the WAL-G envdir

For the edge MinIO deployment, /run/etc/wal-e.d/env/ can contain files such as:

/run/etc/wal-e.d/env/
|-- AWS_ACCESS_KEY_ID
|-- AWS_SECRET_ACCESS_KEY
|-- AWS_ENDPOINT
|-- AWS_REGION
|-- AWS_S3_FORCE_PATH_STYLE
|-- WALG_DISABLE_S3_SSE
|-- WALG_S3_PREFIX
`-- WALE_S3_PREFIX

The directory name and WALE_* compatibility variables are historical. Their presence does not by itself prove that the WAL-E binary is handling backups.

WAL_S3_BUCKET is an input consumed by Spilo. Spilo combines it with the cluster scope and other settings to generate a final prefix such as WALG_S3_PREFIX, which WAL-G consumes. Credentials should come from a Secret; do not put their values in documentation or print them during routine checks.

4. PostgreSQL archives completed WAL

With WAL-G backup support enabled, Spilo generates an archive_command that ultimately invokes WAL-G to upload a completed WAL segment. A simplified form is:

archive_command = 'envdir /run/etc/wal-e.d/env wal-g wal-push %p'

envdir exposes the generated WAL-G configuration.
wal-g wal-push uploads the completed segment.
PostgreSQL substitutes %p with the segment's local path.

The installed Spilo release may use a wrapper around this command. Treat the result of SHOW archive_command as runtime truth.

If backup support is disabled, Spilo may generate a no-op archive command such as /bin/true, or archiving may be disabled another way. Check the live PostgreSQL settings rather than assuming a particular fallback.

5. Cron starts base backups

When backups and BACKUP_SCHEDULE are enabled, Spilo installs a cron entry. A typical entry uses Spilo's wrapper:

15 2 * * * envdir "/run/etc/wal-e.d/env" /scripts/postgres_backup.sh "/home/postgres/pgdata/pgroot/data"

The wrapper invokes WAL-G to create a physical base backup and applies the retention behavior implemented by that Spilo version. Inspect crontab -l and the wrapper before assuming that BACKUP_NUM_TO_RETAIN means a count; some versions interpret it as an age in days.

Base backups and WAL archiving serve different purposes:

a base backup provides the starting filesystem image;
archived WAL supplies the changes after that image and enables point-in-time recovery;
a recoverable chain requires both a usable base backup and continuous WAL.

6. Patroni manages PostgreSQL

Patroni reads the generated configuration, starts PostgreSQL, manages leader election and replication, and applies supported dynamic configuration changes. It is not responsible for the object-store durability of WAL-G backups.

Why `WAL_S3_BUCKET` can appear not to work

Spilo cannot use a ConfigMap that was never injected into its pod, and it does not know that an environment value originally came from a ConfigMap. If WAL_S3_BUCKET appears ineffective, check the entire path:

Is the ConfigMap actually referenced by the rendered StatefulSet?
Does the running container contain the expected variable name?
Is another explicit env entry using the same name?
Is USE_WALG_BACKUP enabled?
Did the pod restart after its environment-bearing pod template changed?
Did Spilo generate the expected WALG_S3_PREFIX?
What do SHOW archive_command and crontab -l report?

Defining the bucket directly in the PostgreSQL custom resource is explicit and keeps the setting close to the cluster it controls:

apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  name: acid-minimal-cluster
spec:
  env:
    - name: USE_WALG_BACKUP
      value: "true"
    - name: WAL_S3_BUCKET
      value: postgresql

This does not mean every value must live in spec.env. Credentials belong in a Secret, and values generated from structured fields such as clone: should remain owned by the operator.

Safe runtime verification

Confirm the namespace and cluster before running commands. Avoid printing credential files or the full process environment.

# Generated Patroni configuration. Review before sharing because deployments
# can embed endpoints or other environment-specific details.
kubectl exec -n default "$POD" -- sed -n '1,240p' /run/etc/patroni.yml

# Show filenames only; do not cat credential files.
kubectl exec -n default "$POD" -- ls -la /run/etc/wal-e.d/env/

# Verify selected non-secret settings.
kubectl exec -n default "$POD" -- sh -c \
  'printf "WALG_S3_PREFIX="; cat /run/etc/wal-e.d/env/WALG_S3_PREFIX; \
   printf "AWS_ENDPOINT="; cat /run/etc/wal-e.d/env/AWS_ENDPOINT'

# PostgreSQL runtime truth.
kubectl exec -n default "$POD" -- su - postgres -c \
  'psql -c "SHOW archive_mode;" -c "SHOW archive_command;" -c "SHOW archive_timeout;"'

# Scheduled base backups.
kubectl exec -n default "$POD" -- crontab -l

Object names or live WAL-G configuration must be inspected before claiming a compression format. Historical .lzo objects may have been written by WAL-E and do not prove that current WAL-G uploads use LZO. If documenting a measured format, state when and where it was observed.

Applying configuration changes

Container environment variables are fixed for the lifetime of a container. Updating a ConfigMap does not change the environment of an already running process. A changed operator-managed pod template normally results in a controlled rollout after the PostgreSQL custom resource or operator configuration is updated.

Some PostgreSQL and Patroni settings can be reloaded dynamically; others need a PostgreSQL restart or pod replacement. Do not use pod deletion as a generic configuration mechanism. Preview the operator-owned change, apply it deliberately, and observe Patroni health throughout the rollout.

Case study: duplicate clone variables caused a reconciliation loop

The later GCS-backed Horse deployment exposed an important consequence of the configuration layers described above. Its PostgreSQL custom resource contained a structured clone: block, while spec.env also manually declared variables that the Postgres Operator derived from that block.

The manually duplicated variables were:

CLONE_METHOD
CLONE_WAL_GS_BUCKET
CLONE_GOOGLE_APPLICATION_CREDENTIALS

The operator also generated clone method, bucket, credential path, archive scope, source UID, and recovery target settings. Some Spilo clone method names retain WALE for historical compatibility even when WAL-G performs the actual transfer, so a name such as CLONE_WITH_WALE does not by itself mean that the legacy WAL-E binary is in use.

How the effective configuration is prioritized

The important priority is ownership, not just which duplicate happens to be last:

clone: structured configuration
        |
        `-- Postgres Operator generates CLONE_* environment entries

spec.env manual configuration
        |
        `-- Operator also places these entries in the pod template

rendered StatefulSet
        |
        `-- now contains overlapping or differently rendered entries

Spilo process environment
        |
        `-- one effective value is observed at startup

generated Patroni/WAL-G configuration
        |
        `-- controls clone/restore behavior in PostgreSQL

Even if the effective process value happens to be correct, the controller must still reconcile the declarative pod template. Duplicate names, different ordering, or conflicting representations can make its desired environment differ from the environment stored in the existing StatefulSet. Relying on "last value wins" therefore does not make the resource stable.

What happened

The Postgres Operator resynchronized the cluster every 30 minutes. On each pass, it reported:

new statefulset containers's postgres (index 0) environment does not match the current one

It treated the StatefulSet as outdated and initiated another rolling update. Patroni kept the database available by moving roles as necessary, but the pod replacement and PostgreSQL lifecycle activity were followed by another base backup. The bucket consequently showed backups about every 30 minutes even though the configured backup cron ran only every six hours.

An observed sequence on 2026-10-04 was:

06:23 UTC  base backup after reconciliation activity
06:53 UTC  another base backup
07:23 UTC  another base backup
07:53 UTC  another base backup
08:23 UTC  another base backup from the old loop

08:29 UTC  duplicate manual CLONE_* entries removed and corrected CR applied
08:34 UTC  expected final backup associated with the intentional rollout
08:48 UTC  next periodic operator resync started
08:49 UTC  resync completed without mismatch, rollout, failover, or backup

The backup cron was not creating the 30-minute cadence. The repeated reconciliation and rolling updates were the trigger.

The fix and operational rule

The manual CLONE_METHOD, CLONE_WAL_GS_BUCKET, and CLONE_GOOGLE_APPLICATION_CREDENTIALS entries were removed. The operator now derives them solely from clone:. Non-overlapping Spilo controls such as USE_WALG_RESTORE and CLONE_USE_WALG_RESTORE remain explicit where needed.

The general rule is:

Give every generated environment variable one owner. Prefer structured operator fields, use explicit environment variables only for settings the operator does not generate, and verify both the rendered StatefulSet and the resulting PostgreSQL/WAL-G runtime configuration.

This incident also shows why an unexpected backup frequency should be correlated with operator logs, StatefulSet revisions, pod creation times, and Patroni role changes before changing the backup schedule.

Summary

The Postgres Operator renders the Kubernetes resources and final pod environment from several configuration sources.
Spilo consumes that environment at container startup and generates Patroni, PostgreSQL, WAL-G, and cron configuration.
PostgreSQL archives completed WAL continuously; cron starts periodic base backups.
Local MinIO supports an edge deployment during cloud outages, but a durable copy must exist in another failure domain.
Environment changes require pod replacement, while some Patroni/PostgreSQL settings can be changed dynamically.
Operator-generated variables must not also be declared manually. Duplicate ownership can cause perpetual drift, rolling updates, and expensive secondary effects even when the effective value looks correct inside a pod.

← Back to posts