Backups & Recovery
Backups & Disaster Recovery
Every data store has a defined backup mechanism, retention window, and restore path. This is a HIPAA contingency-plan requirement (§164.308(a)(7)) — designed in, not bolted on.
What backs up what
| Store | Mechanism | Retention | Restore |
|---|---|---|---|
| Neon (OLTP · source of truth) | Point-in-time restore (PITR) — continuous WAL, restore to any timestamp; plus named branch snapshots | History window set on the Scale plan (confirm current window; typically 7–30 days) | Create a branch at a timestamp, or restore the primary (see runbook) |
| BigQuery (OLAP) | Time travel + table snapshots + scheduled exports to GCS | 7-day time travel; snapshot / export retention per lifecycle | FOR SYSTEM_TIME AS OF, snapshot restore, or reload from GCS |
| GCS (objects / uploads) | Object versioning + lifecycle rules | Non-current versions kept per lifecycle policy | Restore a prior object version |
| Config / IaC | Pulumi state on Pulumi Cloud (versioned) + Git history | Full history | pulumi up from a prior state / commit |
Neon is the source of truth. BigQuery and GCS are recoverable, but the authoritative recovery target is always Neon — rebuilding BigQuery is a re-run of the extract-load + dbt (see Data Pipeline), not a restore.
RTO / RPO targets
| Store | RPO (max data loss) | RTO (max downtime) |
|---|---|---|
| Neon (OLTP) | ≈ 0 — continuous WAL / PITR | minutes — branch-at-timestamp |
| BigQuery (OLAP) | ≤ 24 h — nightly rebuild from Neon | hours — rebuild via pipeline |
| GCS (objects) | ≈ 0 — versioned | minutes — restore version |
These are targets to design and test against, not guarantees. Confirm the Neon retention window on the current plan, and treat the RPO/RTO here as the SLOs the backup config must satisfy.
Restore runbook — Neon
- Identify the target time — the last-known-good timestamp, before the incident.
- Branch, don't overwrite. Create a Neon branch at that timestamp (
neondatabase/create-branch-action/ Neon API) and validate the data there first — never restore in place blind. - Cut over — repoint the app's
DATABASE_URL(via GCP Secret Manager) to the restored branch, or promote it to primary. - Rebuild downstream — trigger the extract-load + dbt to rebuild BigQuery from the restored Neon state.
- Record the event (what, when, blast radius, root cause) for the compliance / audit log.
Test restores on a schedule. A backup that has never been restored is only a hypothesis. Run a periodic restore drill (e.g., quarterly) into a throwaway Neon branch and confirm the RTO/RPO above actually hold.
Managed in IaC
Retention windows, GCS object versioning + lifecycle, and BigQuery snapshot schedules are declared in Pulumi (see Infrastructure) — reproducible and code-reviewed, never clicked in by hand.